When using virtualenv, mypy has to be installed in virtualenv before project is installed

[nickwilliams-eventbrite]

• Are you reporting a bug, or opening a feature request?
  It's not a feature request. I'm pretty sure it's a bug, but it might be a bug in the form of incomplete documentation (i.e., this is an undocumented feature), or it might be part code bug part documentation bug.
• Please insert below the code you are checking with mypy, or a mock-up repro if the source is private. We would appreciate if you try to simplify your case to a minimal repro.
  Please see the open source project https://github.com/eventbrite/conformity. It has a py.typed file and MyPy does successfully use that in other projects, but only in certain circumstances.
• What is the actual behavior/output?
  • Have mypy already installed in the system Python 3.7. mkvirtualenv for a new project with Conformity in its dependencies and using Conformity types. pip install mypy in the virtualenv, then install the project dependencies. mypy . in the project works without any errors.
  • Have mypy already installed in the system Python 3.7. mkvirtualenv for the same project. Install the project dependencies, then pip install mypy in the virtualenv, too. mypy . in the project results in import errors about Conformity.
  • Have mypy already installed in the system Python 3.7. mkvirtualenv for the same project. Install the project dependencies but do not install mypy in the virtualenv. mypy . in the project results in import errors about Conformity.
• What is the behavior/output you expect?
  • The first scenario is what I expect. No errors.
  • The second scenario should work like the first. No errors. It shouldn't matter whether you install mypy before or after you install the project dependencies (namely Conformity). It appears that, in this scenario, MyPy is using the system Python path instead of the virtualenv Python path, even though which mypy is correct (see below).
  • I would hope that the third scenario could also be made to work, but I understand if it can't (different paths for the mypy command shebang vs. the virtualenv python). However, this isn't documented anywhere, so it should be.
• What are the versions of mypy and Python you are using? Do you see the same issue after installing mypy from Git master?
  I have not tried installing Git from master because the problem is not version-dependent AFAICT. It is install-order dependent. I did, however, look for any other similar issues that had been addressed, or documentation that could help here, and found none.

  • $ which python
    /Users/nwilliams/.virtualenvs/ebmetrics3/bin/python
    $ python --version
    Python 3.7.4
    $ which mypy
    /Users/nwilliams/.virtualenvs/ebmetrics3/bin/mypy
    $ mypy --version
    mypy 0.720
    

  • $ which python
    /Users/nwilliams/.virtualenvs/ebmetrics3/bin/python
    $ python --version
    Python 3.7.4
    $ which mypy
    /Users/nwilliams/.virtualenvs/ebmetrics3/bin/mypy
    $ mypy --version
    mypy 0.720
    

  • $ which python3
    /usr/local/bin/python3
    $ python3 --version
    Python 3.7.4
    $ which mypy
    /usr/local/bin/mypy
    $ mypy --version
    mypy 0.701
    

• What are the mypy flags you are using? (For example --strict-optional)

  [mypy]
  python_version = 3.7
  check_untyped_defs = True
  [mypy-setup]
  ignore_errors = True
  [mypy-tasks]
  ignore_errors = True
  [mypy-django.*]
  ignore_missing_imports = True
  [mypy-pytest.*]
  ignore_missing_imports = True
  

• If mypy crashed with a traceback, please paste the full traceback below.
  No crash.

[nickwilliams-eventbrite]

Actually, I need to amend this after a lot more experimentation. I need to replace scenario 2 above (install project dependencies and THEN MyPy) with the following:

• Have mypy already installed in the system Python 3.7. mkvirtualenv for the same project. Install the project dependencies, then pip install mypy in the virtualenv, too. mypy . in the project works without any errors.
• Have mypy already installed in the system Python 3.7. mkvirtualenv for the same project. Put mypy in the project dependencies. Install the project dependencies. mypy . in the project works without any errors.
• Have mypy already installed in the system Python 3.7. mkvirtualenv for the same project. Install the project dependencies, but do not install mypy in the virtualenv yet. Run mypy . in the project, which results in import errors about Conformity. Now, after running the system mypy, do pip install mypy in the virtualenv. mypy . in the project still results in import errors about Conformity, but it should not. It should work.

Note that the only thing I'm changing between each scenario is the virtualenv. I'm not refreshing the project clone or clearing cache directories. But, importantly, I can move from a failing scenario to a successful scenario without deleting any cache directories, so I believe this rules out any local (in the project directory) cache directories being the culprit.

[JukkaL]

@ethanhs Do you have time to look at this issue? It seems related to PEP 561 support.

[emmatyping]

Sure, I can take a look at it a bit later.

[emmatyping]

I'm afraid I am unable to reproduce this, though perhaps this is an issue with mkvirtualenv (I am not familiar with how it works). Here is what I tried:

# install in system Python
$ sudo -H pip3 install mypy
# create then source venv
$ virtualenv -p python3 ~/.env
$ . ~/.env/bin/activate
(.env) $ git clone git@github.com:eventbrite/conformity.git
(.env) $ cd conformity
(.env) $ mypy .  # no error
(.env) $ pip install .
(.env) $ mypy .  # no error
(.env) $ pip install mypy
(.env) $ mypy .  # no error
(.env) $ pip uninstall conformity -y
(.env) $ mypy .  # no error

Do you have a sequence of commands I could run to reproduce this?

[nickwilliams-eventbrite]

You got it backwards. The problem isn't seen in Conformity. It's seen in projects that use Conformity. These exact steps should replicate the problem for you. Includes steps that don't work, steps that do work, and a conclusion.

Create a new project directory with these three files:

setup.cfg:

[wheel]
universal=1

[mypy]
python_version = 3.7
check_untyped_defs = True
[mypy-setup]
ignore_errors = True
[mypy-tasks]
ignore_errors = True
[mypy-django.*]
ignore_missing_imports = True
[mypy-pytest.*]
ignore_missing_imports = True

setup.py:

from setuptools import setup

setup(
    name='test_typing',
    version='1.0.0',
    description='Python library to test typing',
    author='Nick Williams',
    packages=[
        str('test_typing'),
    ],
    install_requires=['conformity~=1.25'],
    license='Apache 2.0',
)

test_typing/__init__.py:

from conformity import fields

field: fields.Base = fields.Dictionary({}, allow_extra_keys=True)

Prerequisite

• pip install virtualenvwrapper (this augments mkvirtualenv so that you don't have to manually source the virtualenv ... it is sourced immediately on creation and can be sourced again later with workon [virtualenv name])

Steps that Don't Work

• cd into the project directory
• mkvirtualenv test-typing-1 (this will also activate the virtualenv)
• pip install -e . (this will also install Conformity)
• Try mypy . (results are as expected):

  $ mypy .
  test_typing/__init__.py:1: error: Cannot find module named 'conformity'
  test_typing/__init__.py:1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
  

• pip install mypy
• Try mypy . again ... still does not work, but it should, because which mypy is correct:

   $ which mypy
  /Users/nwilliams/.virtualenvs/test-typing-1/bin/mypy
  

Steps that Do Work

• cd into the same project directory if not there already (do not create a new project directory or delete any Python or MyPy caches ... this proves that it's not a project-local cache causing the problem)
• deactivate virtualenv if still using it
• mkvirtualenv test-typing-2 (this will also activate the virtualenv)
• pip install -e . (this will also install Conformity)
• pip install mypy (this is the only difference ... running this before calling mypy)
• mypy . ... no errors this time

Conclusion
With a system mypy installed, running mypy with the virtualenv activated before installing mypy in the virtualenv breaks the virtualenv so that mypy never works in it, even after installing it in the virtualenv.

Note
It would really be nice if the system MyPy could use the dependencies installed in the virtualenv, but I get it if it can't.

[emmatyping]

So I think I found the issue. It appears mypy is discovering the system Python installation because we try to execute pythonX.Y when given a python_version=X.Y, however this is the wrong thing to do in a virtualenv. (it doesn't symlink correctly). What we likely should do is:

Try pythonX if we are in a virtualenv, otherwise use pythonX.Y outside, or maybe fall back to pythonX if that doesn't wok outside.

For now you can add --python-executable=python in your virtualenv, but I will come up with a patch.

[nickwilliams-eventbrite]

I don't think that's it. As you'll see in the Steps that Do Work section in my previous comment, it does work in a virtualenv as long as you install mypy in the virtualenv before calling it. (And, also, all my virtualenvs are already made with --python=python, because I have the two aliases that I actually use:

alias mkvirtualenv2='mkvirtualenv --python=/usr/local/bin/python2'
alias mkvirtualenv3='mkvirtualenv --python=/usr/local/bin/python3'

)

[emmatyping]

Ok, coming back to this with a fresh set of eyes, the issue is that the global mypy is conflicting with the virtualenv mypy. (i.e. when you run mypy ., it is using the module finder in the global mypy).

I am going to close this, but I filed #17214 to make it clearer when this issue happens.