8.1.1 raises exception if boto is imported in environment

[effigies]

Pytest 8.1.1 on Ubuntu 22.04.

❯ pipenv run pytest ./tests --pdb
============================================= test session starts =============================================
platform linux -- Python 3.10.10, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/chris/Projects/openneuro/openneuro/services/datalad
configfile: pytest.ini
plugins: xdist-3.5.0, cov-4.1.0
collecting ... 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stderr >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[WARNING] Requested extension 'next' is not available 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/runner.py:340: in from_call
    result: Optional[TResult] = func()
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/runner.py:388: in collect
    return list(collector.collect())
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:576: in collect
    self._register_setup_module_fixture()
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:589: in _register_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:315: in obj
    self._obj = obj = self._getobj()
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:573: in _getobj
    return importtestmodule(self.path, self.config)
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:520: in importtestmodule
    mod = import_path(
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/pathlib.py:546: in import_path
    mod = _import_module_using_spec(
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/pathlib.py:630: in _import_module_using_spec
    spec = meta_importer.find_spec(module_name, [str(module_location)])
E   AttributeError: '_SixMetaPathImporter' object has no attribute 'find_spec'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/pathlib.py(630)_import_module_using_spec()
-> spec = meta_importer.find_spec(module_name, [str(module_location)])
(Pdb) meta_importer
<boto.vendored.six._SixMetaPathImporter object at 0x76dad9949d20>
(Pdb) l
625  	        "src_tests_test_foo".
626  	    """
627  	    # Checking with sys.meta_path first in case one of its hooks can import this module,
628  	    # such as our own assertion-rewrite hook.
629  	    for meta_importer in sys.meta_path:
630  ->	        spec = meta_importer.find_spec(module_name, [str(module_location)])
631  	        if spec is not None:
632  	            break
633  	    else:
634  	        spec = importlib.util.spec_from_file_location(module_name, str(module_path))
635  	    if spec is not None:

Observed in OpenNeuroOrg/openneuro#3020, which attempted to auto-update dependencies.

$ pip list
Package            Version
------------------ ---------------------
annexremote        1.6.4
async-timeout      4.0.3
boto               2.49.0
boto3              1.34.75
botocore           1.34.75
certifi            2024.2.2
cffi               1.16.0
chardet            5.2.0
charset-normalizer 3.3.2
coverage           7.4.4
cryptography       42.0.5
datalad            0.19.6
Deprecated         1.2.14
distro             1.9.0
dnspython          2.6.1
ecs-logging        2.1.0
elastic-apm        6.21.4.post8347027212
elastic-transport  8.13.0
elasticsearch      8.13.0
exceptiongroup     1.2.0
execnet            2.0.2
falcon             3.1.3
falcon-elastic-apm 0.1.1
fasteners          0.19
gevent             24.2.1
gitdb              4.0.11
GitPython          3.1.43
greenlet           3.0.3
gunicorn           21.2.0
humanize           4.9.0
idna               3.6
importlib_metadata 7.1.0
iniconfig          2.0.0
iso8601            2.1.0
jaraco.classes     3.4.0
jaraco.context     4.3.0
jaraco.functools   4.0.0
jeepney            0.8.0
jmespath           1.0.1
keyring            25.0.0
keyrings.alt       5.0.1
looseversion       1.3.0
mock               5.1.0
more-itertools     10.2.0
msgpack            1.0.8
packaging          24.0
patool             2.2.0
pip                24.0
platformdirs       4.2.0
pluggy             1.4.0
pycparser          2.22
pygit2             1.14.1
PyGithub           2.2.0
PyJWT              2.8.0
PyNaCl             1.5.0
pytest             8.1.1
pytest-cov         4.1.0
pytest-xdist       3.5.0
python-dateutil    2.9.0.post0
python-gitlab      4.4.0
redis              5.0.3
requests           2.31.0
requests-toolbelt  1.0.0
s3transfer         0.10.1
SecretStorage      3.3.3
setuptools         69.2.0
six                1.16.0
smmap              5.0.1
tomli              2.0.1
tqdm               4.66.2
typing_extensions  4.10.0
urllib3            2.2.1
wheel              0.42.0
wrapt              1.14.1
zipp               3.18.1
zope.event         5.0
zope.interface     6.2

[The-Compiler]

Anything that's on the sys.meta_path can/should have a find_spec ever since Python 3.4 (2013). It not being there is deprecated since Python 3.10 (2021), and indeed six added this back in 2021.

boto is using a vendored six version from 2018, so you most likely want to switch to boto3 instead which is actively maintained (as also recommended by boto, which hasn't been touched in the past 2 years).

Not sure if pytest should try to paint over this. If I'm not mistaken, this code was around since pytest 6 in 2020 (ab6dacf), and I don't think the recent move (4dea183) changed anything?

[effigies]

I don't disagree, but I have very limited control over when a dependency upgrades from boto to boto3. I am using this to encourage them to prioritize it. I haven't looked into what change in 8.1 caused this; possibly you removed a suppression of this error, but it does seem like a good idea to guard against a bad importer that some other library stuck into sys.meta_path.

[Zac-HD]

Happy to say that we're not going to "fix" this, and your other dependency should.