Import error when running pytest with poetry

[cswinter]

On my local checkout, I can't seem to be able to run pytest because of an import error.

⋊> ~/s/enn-incubator on main ◦ poetry run pytest                                                                                                                    (base) 21:50:44
=============================================================================== test session starts ===============================================================================
platform linux -- Python 3.7.3, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/clemens/src/enn-incubator
collected 3 items / 2 errors / 1 selected

===================================================================================== ERRORS ======================================================================================
____________________________________________________________ ERROR collecting enn_ppo/enn_ppo/test_sample_recorder.py _____________________________________________________________
ImportError while importing test module '/home/clemens/src/enn-incubator/enn_ppo/enn_ppo/test_sample_recorder.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../miniconda3/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
enn_ppo/enn_ppo/test_sample_recorder.py:3: in <module>
    from enn_ppo.sample_recorder import SampleRecorder, Sample, Trace
enn_ppo/enn_ppo/sample_recorder.py:4: in <module>
    from entity_gym.environment import ActionSpace, ObsSpace
entity_gym/entity_gym/environment.py:6: in <module>
    from ragged_buffer import RaggedBufferF32, RaggedBufferI64
../../.cache/pypoetry/virtualenvs/incubator-Qe8CM38i-py3.7/lib/python3.7/site-packages/ragged_buffer/__init__.py:1: in <module>
    from typing import Any, Generic, Protocol, Type, TypeVar, Union, cast, overload
E   ImportError: cannot import name 'Protocol' from 'typing' (/home/clemens/miniconda3/lib/python3.7/typing.py)
________________________________________________________________ ERROR collecting enn_ppo/enn_ppo/test_training.py ________________________________________________________________
ImportError while importing test module '/home/clemens/src/enn-incubator/enn_ppo/enn_ppo/test_training.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../miniconda3/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
enn_ppo/enn_ppo/test_training.py:1: in <module>
    from enn_ppo import train
enn_ppo/enn_ppo/train.py:21: in <module>
    from entity_gym.environment import (
entity_gym/entity_gym/environment.py:6: in <module>
    from ragged_buffer import RaggedBufferF32, RaggedBufferI64
../../.cache/pypoetry/virtualenvs/incubator-Qe8CM38i-py3.7/lib/python3.7/site-packages/ragged_buffer/__init__.py:1: in <module>
    from typing import Any, Generic, Protocol, Type, TypeVar, Union, cast, overload
E   ImportError: cannot import name 'Protocol' from 'typing' (/home/clemens/miniconda3/lib/python3.7/typing.py)
============================================================================= short test summary info =============================================================================
ERROR enn_ppo/enn_ppo/test_sample_recorder.py
ERROR enn_ppo/enn_ppo/test_training.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================================ 2 errors in 1.16s ================================================================================

[cswinter]

cc @vwxyzjn

[cswinter]

Not sure why this is happening, CI seems to be working. Maybe something is weird about my system, though you would think poetry would protect against that.

[cswinter]

This mypy command also gives an error, the other two work though 🤔

⋊> ~/s/enn-incubator on main ◦ poetry run mypy rogue_net                                                                                                            (base) 21:55:19
rogue_net/rogue_net/backbone_creator.py:1: error: Module "typing" has no attribute "Literal"  [attr-defined]
Found 1 error in 1 file (checked 7 source files)

[cswinter]

Yeah it doesn't run:

⋊> ~/s/enn-incubator on main ◦ poetry run python enn_ppo/enn_ppo/train.py                                                                                           (base) 21:57:15
Traceback (most recent call last):
  File "enn_ppo/enn_ppo/train.py", line 21, in <module>
    from entity_gym.environment import (
  File "/home/clemens/src/enn-incubator/entity_gym/entity_gym/environment.py", line 6, in <module>
    from ragged_buffer import RaggedBufferF32, RaggedBufferI64
  File "/home/clemens/.cache/pypoetry/virtualenvs/incubator-Qe8CM38i-py3.7/lib/python3.7/site-packages/ragged_buffer/__init__.py", line 1, in <module>
    from typing import Any, Generic, Protocol, Type, TypeVar, Union, cast, overload
ImportError: cannot import name 'Protocol' from 'typing' (/home/clemens/miniconda3/lib/python3.7/typing.py)

[cswinter]

Maybe it's related to the Python version?

[cswinter]

Long issue thread that seem related, only skimmed so not sure if there's a good resolution: python-poetry/poetry#522

[cswinter]

There is this part in the pyproject.toml:

[tool.poetry.dependencies]
python = ">=3.7.1,<3.10"

Changing it to python 3.9.7, removing the previous env, and running poetry install again runs into a new issue:

⋊> ~/s/enn-incubator on main ⨯ poetry install                                                                                                                       (base) 22:16:20
The currently activated Python version 3.7.3 is not supported by the project (>=3.9.7,<3.10).
Trying to find and use a compatible version.

  NoCompatiblePythonVersionFound

[vwxyzjn]

@cswinter did you run pytest under the poetry virtual environment? So like

poetry shell
pytest

or just

poetry run pytest

The latter should always work.

[Bam4d]

You also have to do the same with black

poetry shell for some reason is really finicky with me and doesn't actually put me in the virtual environment.
I have to manually jump into the environment:

source /home/bam4d/.cache/pypoetry/virtualenvs/incubator-rocsl_Dg-py3.8/bin/activate

[vwxyzjn]

The last listed error seems to suggest poetry did not recognize your 3.9.3. Consider explicitly specify it (https://python-poetry.org/docs/managing-environments/), like

poetry env use /full/path/to/python

[vwxyzjn]

@Bam4d try running poetry run black .

[Bam4d]

@Bam4d try running poetry run black .

Yeah thats what I meant when I said "you have to do the same with black"

[cswinter]

I was doing poetry run pytest, see my first comment. I tried poetry shell as well which didn't help.
I do still think it's most likely related to Python version.

[cswinter]

OK, so it is at least partially related to existing conda install, fully deactivating conda give me a slightly different error.

[cswinter]

Hmm well now the torch-scatter build is failing:

    csrc/segment_coo.cpp:1:10: fatal error: Python.h: No such file or directory
        1 | #include <Python.h>
          |          ^~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
    ERROR: Failed building wheel for torch-scatter
    Running setup.py clean for torch-scatter
  Failed to build torch-scatter
  Installing collected packages: torch-scatter
      Running setup.py install for torch-scatter: started
      Running setup.py install for torch-scatter: finished with status 'error'
      ERROR: Command errored out with exit status 1:
       command: /home/clemens/.cache/pypoetry/virtualenvs/incubator-Qe8CM38i-py3.8/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-y92zdx4r/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-y92zdx4r/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-yfkwkr8k/install-record.txt --single-version-externally-managed --compile --install-headers /home/clemens/.cache/pypoetry/virtualenvs/incubator-Qe8CM38i-py3.8/include/site/python3.8/torch-scatter
           cwd: /tmp/pip-req-build-y92zdx4r/

[vwxyzjn]

Oh I had this errror before. Try apt install python3-dev

[cswinter]

kk, we should add that to our setup docs

[cswinter]

This is working now, keeping this issue open to track documenting the following gotchas:

• sudo apt install python3-dev
• making sure no conda environment is active