Jedi only recognizes first of intersecting implicit namespace packages on PYTHONPATH

[andrew-christianson]

If multiple implicit namespaces packages share a common path, but are located in two different directories on disk, recent versions of Jedi return completions for only the first package found in in the search path. For example, given two PEP420 packages with a common prefix, but different locations like:

pkg1
└── aa
    └── bb
        └── cc
            └── one.py
pkg2
└── aa
    └── bb
        └── dd
            └── two.py

then completions are only produced for the first package found, Goto only works for that package, etc e.g.

$ PYTHONPATH=pkg1:pkg2 ./venv/bin/python -c 'import jedi;print(jedi.Script("import aa.bb.").complete())'
[<Completion: cc>]
$ PYTHONPATH=pkg1:pkg2 ./venv/bin/python -c 'import jedi;print(jedi.Script("from aa.bb.dd.two import b;b.").complete())'
[]

While the example is contrived, this situation comes up in real usage when multiple PEP420 packages are installed as editable.

This appears to mirror same issue for old-style namespace packages (#1105) - hopefully the fix is as simple?

A complete reproducible example:

mkdir -p pkg1/aa/bb/cc
mkdir -p pkg2/aa/bb/dd

echo 'a=1' > pkg1/aa/bb/cc/one.py
echo 'b=2' > pkg2/aa/bb/dd/two.py

python3 -m venv venv 
./venv/bin/python -m pip install jedi

$ PYTHONPATH=pkg1:pkg2 ./venv/bin/python -c 'import jedi;print(jedi.Script("import aa.bb.").complete())'
[<Completion: cc>]
$ PYTHONPATH=pkg2:pkg1 ./venv/bin/python -c 'import jedi;print(jedi.Script("import aa.bb.").complete())'
[<Completion: dd>]
$ PYTHONPATH=pkg1:pkg2 ./venv/bin/python -c 'import jedi;print(jedi.Script("from aa.bb.dd.two import b;b.").complete())'
[]
$ PYTHONPATH=pkg2:pkg1 ./venv/bin/python -c 'import jedi;print(jedi.Script("from aa.bb.dd.two import b;b.").complete())'
[<Completion: as_integer_ratio>, <...]

Versions:
MacOS 11.2.3
Python 3.9.2
Jedi 0.18.0, current master

[davidhalter]

I'm pretty sure this is just a bug. Thanks for the investigation. I think with the amount of investigation you should probably also be able to fix it yourself :).

The import code is a bit complicated, because I'm using the importlib stuff (which is pretty annoying to work with IMO). You can search for ImplicitNSInfo and you will find the file jedi/inference/compiled/subprocess/functions.py which you would need to patch for it. I'm pretty sure that Jedi's internals already support multiple paths, since it's used for explicit namespace packages. So the only thing you probably need to do is to change the path on this line:

implicit_ns_info = ImplicitNSInfo(full_name, spec.submodule_search_locations._path)

Since debugging of this can be a bit hard, because this runs in a subprocess, I would recommend you use jedi.Script(..., environment=jedi.InterpreterEnvironment()) to run tests, so you can easily print debug this.

[andrew-christianson]

Thanks for the pointer! I have a quick patch on my fork that ... does not locate the changes in the place you mentioned. I'll take a closer look and see if I can fix from within ImplicitNSInfo

[bbugyi200]

The following PR fixes this bug: #1784

[davidhalter]

The bug was fixed in #1805.

However #1784 remains open, because of the annotation work, but this should not be an issue anymore.