mypy not finding modules located above the current working directory

[ada-waffles]

I have a project with a structure similar to:

• project
  • mypy.ini
  • src
    • main.py
    • pkg_a
      • __init__.py
      • mod_a.py
    • pkg_b
      • __init__.py
      • mod_b.py

Where main.py has from pkg_a.mod_a import ThingFromA and mod_a has from pkg_b.mod_b import ThingFromB

If I run MyPy from project or from src, either pointing it at main.py or at pkg_a/mod_a.py, it works fine. If, however, I cd into pkg_a and run it on ./mod_a.py, it fails to resolve the import to package B.

The MyPy docs state that the behavior of absolute import resolution is to walk up the file tree until the first folder that does not have an __init__.py and treat that as the root of the project by adding it to the search path. It seems that there is an additional restriction here that MyPy will only walk as far up as the cwd it is called from. The docs make no mention of this that I can find. Is this an intentional limitation? I can reproduce the problem on a colleague's computer, but hours of googling have not managed to find anyone else complaining about this so I feel like I'm missing something obvious here.

The reason this is an issue is because, while I use VSCode, the team coming onto this project uses PyCharm on Windows, and the MyPy plugin we found for it seems to be calling MyPy from the directory of the file it is checking, meaning PyCharm users are currently stuck without working editor integration (we don't want to use PyCharm's own type checker as we want the type checking on our development machines to behave the same as the type checking in CI (which will be MyPy)).

I am able to reproduce this with the latest MyPy 0.7. The project is Python 3.6.

[ilevkivskyi]

Is it also a problem with the official mypy PyCharm plugin?

[ada-waffles]

My understanding was that that plugin doesn't work on Windows (based on this open issue). I can give it a try anyway...

[ilevkivskyi]

Ah, yes sorry, it might not work on Windows indeed.

[ada-waffles]

Yeah we tried that plugin and it does not, in fact, work on Windows (it's looking for /bin/bash).

Do you know if the behavior I describe with module resolution is correct? Could there be something wrong with how we're configuring MyPy or structuring our project?

[gvanrossum]

What if you used an absolute path? I.e. (assuming you're inside pkg_a) mypy C:\path\to\project\src\pkg_a\mod_a.py? Does that work or not?

[ada-waffles]

That does work! So it looks like it's not the cwd that's the issue, but the way the file is specified.

[ada-waffles]

Took my best stab at a bit of code spelunking now that I was armed with that extra bit of information.

It looks to my completely-unfamiliar-with-your-codebase eye that the crawling is being done using string manipulation on the path, and that the behavior I'm seeing is being caused by the use of os.path.normpath at https://github.com/python/mypy/blob/master/mypy/find_sources.py#L36, rather than abspath. I'm going to guess that this was an intentional choice. Might I ask the reasoning?

[ada-waffles]

Looks like I've found two separate issues here, after all.

Turns out that PyCharm plugin is passing --follow-imports skip for some reason.

I still find the behavior described in this issue odd, so I'll leave it open until someone at least confirms the behavior is intentional.

[gvanrossum]

Good find. I don't actually know why it is done that way, but I can reproduce that if we replace that with abspath() it does find the sibling module. However I note that asking Python to execute that module (from inside the package) doesn't find the sibling module either. And mypy in general tries to emulate the import behavior of Python (with some exceptions). So this feels like it's intentional.

[ada-waffles]

That is true. I guess the conflict of assumptions here then would be that I was assuming that calling MyPy on a file doesn't necessarily imply that that file is intended as an entry point, just that I want to type-check that file. I could totally understand the entry point assumption, though. I would, however, think that the documentation should be modified to reflect it if that assumption is taken as the canonical one. Right now the import discovery process is not described as having any such restriction.

[gvanrossum]

You're right that the docs probably don't explain subtleties like this very well. Do you feel up to making a PR for the docs so they document what you were missing?

[ada-waffles]

I would be happy to. I'd want guidance on how exactly to write the clarification, though.

Would it be sufficient to adjust https://mypy.readthedocs.io/en/latest/running_mypy.html#how-imports-are-found by following the sentence

For sources given on the command line, the path is adjusted by crawling up from the given file or package to the nearest directory that does not contain an __init__.py or __init__.pyi file.

with

If the given path is relative, it will only crawl as far as the current working directory.

?

I confirmed the above description by calling MyPy on "main.py" (no .\) from src. It was still able to resolve the imports from main, so it does consider the cwd even if it's not explicitly a segment of the given path.

[gvanrossum]

Yes, that sounds right.