Autocompletion not working for numpy >= 1.22

[Wellidontcare]

Jedi v0.18.1 autocompletion seems to be broken for the numpy versions since Python 3.10 support.

Minimal reproducible example with numpy==1.23.0:

Python 3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 07:04:59) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.version.full_version
'1.23.0'
>>> import jedi
>>> s = jedi.Script("import numpy as np\n np.")
>>> s.complete()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/api/helpers.py", line 487, in wrapper
    return func(self, line, column, *args, **kwargs)
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/api/__init__.py", line 214, in complete
    return completion.complete()
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/api/completion.py", line 170, in complete
    cached_name, completion_names = self._complete_python(leaf)
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/api/completion.py", line 284, in _complete_python
    cached_name, n = self._complete_trailer(dot.get_previous_leaf())
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/api/completion.py", line 399, in _complete_trailer
    return cached_name, self._complete_trailer_for_values(values)
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/api/completion.py", line 404, in _complete_trailer_for_values
    return complete_trailer(user_context, values)
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/api/completion.py", line 550, in complete_trailer
    completion_names += filter.values()
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/inference/filters.py", line 117, in values
    return self._convert_names(
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/inference/filters.py", line 114, in _convert_names
    return [self.name_class(self.parent_context, name) for name in names]
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/inference/filters.py", line 114, in <listcomp>
    return [self.name_class(self.parent_context, name) for name in names]
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/inference/filters.py", line 120, in <genexpr>
    for name in self._filter(
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/inference/filters.py", line 144, in _filter
    names = [n for n in names if self._is_name_reachable(n)]
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/inference/filters.py", line 144, in <listcomp>
    names = [n for n in names if self._is_name_reachable(n)]
  File "/home/joris/.local/lib/python3.10/site-packages/jedi/inference/gradual/stub_value.py", line 89, in _is_name_reachable
    if definition.type in ('import_from', 'import_name'):
AttributeError: 'NoneType' object has no attribute 'type'

[Wellidontcare]

The issue seems to be with numpy's new typing feature. For np.typing.NDArray jedi crashes in

jedi/jedi/inference/gradual/stub_value.py

Lines 88 to 89 in 7ff0d2d

	definition = name.get_definition()
	if definition.type in ('import_from', 'import_name'):

because name.get_definition() returns None.

For me, a simple guard like this

if definition is not None:
  if definition.type in ('import_from', 'import_name'): 
  ...

fixes the issue. As I am not very much familiar with the codebase, I do not know if NDArray should even end up down this path, maybe this is just the symptom of another underlying issue.

I do have 2 failed tests with this applied, but they are the same failed tests as without the fix.

[davidhalter]

How is np.typing.NDArray typed?

[Wellidontcare]

https://github.com/numpy/numpy/blob/25bcff17a58e942fbeaaba65902157b0f070376a/numpy/_typing/_generic_alias.py#L240-L245

This is the definition from the current numpy master branch, which should be numpy version 1.23.1.

[yut23]

The direct cause for this is the last line of numpy/__init__.pyi (https://github.com/numpy/numpy/blob/25bcff17a58e942fbeaaba65902157b0f070376a/numpy/__init__.pyi#L4408), which uses features from python 3.8 (namely, positional-only parameters):

def from_dlpack(obj: _SupportsDLPack[None], /) -> NDArray[Any]: ...

Type stubs are currently parsed using python 3.7 grammar:

jedi/jedi/api/__init__.py

Lines 123 to 130 in 6c9cab2

	self._module_node, code = self._inference_state.parse_and_get_code(
	code=code,
	path=self.path,
	use_latest_grammar=path and path.suffix == '.pyi',
	cache=False, # No disk cache, because the current script often changes.
	diff_cache=settings.fast_parser,
	cache_path=settings.cache_directory,
	)

jedi/jedi/inference/__init__.py

Lines 194 to 195 in 6c9cab2

	grammar = self.latest_grammar if use_latest_grammar else self.grammar
	return grammar.parse(code=code, path=path, file_io=file_io, **kwargs), code

jedi/jedi/inference/__init__.py

Line 93 in 6c9cab2

self.latest_grammar = parso.load_grammar(version='3.7')

jedi ignores the syntax errors, and thinks NDArray[Any]: ... is some sort of setitem definition, but StubFilter._is_name_reachable() calls get_definition() without include_setitem=True and it returns None. Bumping the grammar version up to 3.8 fixes the issue.

[davidhalter]

Feel free to PR this. Thanks a lot for the investigation. I feel like we should just use the 3.10 or 3.11 grammar there instead of 3.8 though. Ideally you would add tests, but I would even merge without that.

[davidhalter]

I just increased the stub parsing version. So this should now be fixed. But please test this, I was not able to reproduce the issue with Numpy 1.23.

[yut23]

Yes, applying that fix to v0.18.1 works. I was only able to reproduce it on master if I reverted #1870.
Simple repro example:

$ cat module.pyi
def foo(a: int, /) -> list[int]: ...

$ python -c 'import jedi; script = jedi.Script("import module\nmodule."); script.complete(2, 7)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/build/jedi/jedi/api/helpers.py", line 487, in wrapper
    return func(self, line, column, *args, **kwargs)
  ...
  File "/tmp/build/jedi/jedi/inference/gradual/stub_value.py", line 89, in _is_name_reachable
    if definition.type in ('import_from', 'import_name'):
AttributeError: 'NoneType' object has no attribute 'type'