Add partial support for pkgutil.extend_path()

[ipkiss42]

Is your feature request related to a problem? Please describe.
Yes. This feature enables packages with a __init__.py (which predate the implicit namespace packages from PEP 420) to successfully "collaborate" with implicit namespace packages.

Unfortunately, this situation is still fairly common in my work place, due to how different teams are all contributing to common packages "independently".

To reproduce the issue on a simple example, you can use the following script (very similar to the example in #4230):

 #!/bin/sh

# Create a fake repository
rm -rf repo && mkdir repo
echo "from foo.bar import Bar" > repo/bug.py
cat > repo/pyrightconfig.json << EOF
{
    "executionEnvironments": [
        {
            "root": "."
        }
    ],
    "venv": "venv",
    "venvPath": ".."
}
EOF

# Create a venv
rm -rf venv && python -m venv venv

# Go to the site-packages directory
cd venv/lib/python*/site-packages

# Create fake "foo.bar" dependencies (via .pth files)
mkdir -p Bar1/foo/bar
cat > Bar1/foo/bar/__init__.py << EOF
class Bar:
    pass
EOF
echo "Bar1" > Bar1.pth
mkdir -p Bar2/foo
echo "Bar2" > Bar2.pth
cat > Bar2/foo/__init__.py << EOF
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
EOF

Describe the solution you'd like

Support a limited version of pkgutil.extend_path(). I understand that this is getting close to runtime behavior (and thus arguably not in scope for Pyright), so my proposal is to find a tradeoff that limits the complexity, while still helping with most common cases.

Concretely, I suggest the following behavior:

1. Only support __path__, __name__ arguments to extend_path. As mentioned in the documentation, this is the intended use.
2. When an __init__.py file only contains these two lines (other than comments), treat it as if the __init__.py file was missing (I am not an expert in PEP 420, but my understanding is that it would have a similar effect).
3. Ignore all the other cases as unsupported (note: the behavior from 2. could be extended to __init__.py files which start with these two lines, and have additional ones, but I don't know how much complexity this would incur. It would be a nice to have, but not useful for most cases).

I am curious to hear your thoughts on this. Thanks!

[erictraut]

This is dynamic behavior, and we don't support it in a static type checker. Even the "limited form" that you're suggesting here would be somewhere between "extremely complex" and "impossible" to implement given pyright's lazy (just-in-time) analysis design. We need to know about potential import paths much earlier in the analysis process, but your proposal requires that we parse and analyze the file to determine that it is effectively modifying the import resolution paths dynamically. That's not really feasible.

You should switch to implicit namespace packages if you want to work with static type analysis.

[ipkiss42]

Fair enough. Thank you for the explanation!