Stub files were "randomly" picked and ignored

[jjyyxx]

Environment data

• Language Server version: 0.5.45
• OS and version: Linux Ubuntu 18.04.3 LTS in Docker
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.6.8

Expected behaviour

I know there are quite a number of issues about stub files, but I'm not sure if it's a duplicate.

I'm using protobuf in my project. I tried mypy-protobuf to generate stub files for protos. These .pyi files are put alongside their corresponding .py files. I expect proper autocomplete (at least consistent result). Btw, the Jedi backend did this rather good.

Actual behaviour

Even under the same folder,

├── __init__.py
├── chassis_pb2.py
├── chassis_pb2.pyi
├── control_pb2.py
└── control_pb2.pyi

Class Control_command in control_pb2 got autocomplete, but class Chassis in chassis_pb2 do not.

I tried to:

• copy all control_pb2 content (both py and pyi) to chassis_pb2
• then reload vscode
  And it provides no autocompletion when importing Control_command via chassis_pb2.

Furthermore, after some modification, the autocompletion may go wild, which may provide all global variables as complete suggestions for this simple snippet:

a = Chassis()
a. # after type this dot, all global variables show up

Logs

Part of trace, generated with grep -B 3 -A 3 'control_pb2\|chassis_pb2'

[Info  - 8:49:06 AM] Initializing for /usr/bin/python3
[Info  - 8:49:06 AM] Analysis caching mode: None.
Opening document file:///apollo/modules/exercises/exercise1-basic/exercise1.2.todo.py
Analysis of exercise1.2.todo (User) queued. Dependencies: time, sys, termios, fcntl, os, modules.control.proto.control_pb2, modules.control.proto.chassis_pb2
Create built-in compiled (scraped) module:  time /usr/bin/python3 
Create built-in compiled (scraped) module:  sys /usr/bin/python3 
Create compiled (scraped):  termios /usr/lib/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so /usr/lib/python3.6/lib-dynload 
Create built-in compiled (scraped) module:  fcntl /usr/bin/python3 
Import:  os /usr/lib/python3.6/os.py 
Import:  modules.control.proto.chassis_pb2 /apollo/py_proto/modules/control/proto/chassis_pb2.py 
Analysis version 1 of 1 entries has started.
Analysis of exercise1.2.todo (User) on depth 0 completed in 18.32 ms.
Missing keys: time(), sys(), termios(/usr/lib/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so), fcntl(), os(/usr/lib/python3.6/os.py), modules.control.proto.control_pb2(/apollo/py_proto/modules/control/proto/control_pb2.pyi), modules.control.proto.chassis_pb2(/apollo/py_proto/modules/control/proto/chassis_pb2.py)
Analysis version 1 of 1 entries has been completed in 34.74 ms.
Analysis of modules.control.proto.chassis_pb2 (User) queued. Dependencies: modules.control.proto.chassis_pb2, sys, google.protobuf, google.protobuf.descriptor, google.protobuf.message, google.protobuf.reflection, google.protobuf.symbol_database, google.protobuf.descriptor_pb2
Import:  google.protobuf /home/USER/.local/lib/python3.6/site-packages/google/protobuf/__init__.py 
Import:  google.protobuf.descriptor /home/USER/.local/lib/python3.6/site-packages/google/protobuf/descriptor.py 
Import:  google.protobuf.message /home/USER/.local/lib/python3.6/site-packages/google/protobuf/message.py 
--
Import:  google.protobuf.descriptor_pb2 /home/USER/.local/lib/python3.6/site-packages/google/protobuf/descriptor_pb2.py 
Analysis version 9 of 8 entries has started.
Analysis of os(Library) canceled (no AST yet).
Analysis of modules.control.proto.control_pb2(Stub) canceled (no AST yet).
Analysis of fcntl(CompiledBuiltin) canceled (no AST yet).
Analysis of termios(Compiled) canceled (no AST yet).
Analysis of sys(CompiledBuiltin) canceled (no AST yet).
Analysis of modules.control.proto.chassis_pb2 (User) on depth 0 completed in 7.21 ms.
Analysis of time(CompiledBuiltin) canceled (no AST yet).
Analysis version 9 has been completed in 11.96 ms with 7 entries analyzed and 1 entries skipped.
Analysis of google.protobuf.descriptor_pb2 (Library) queued. Dependencies: google.protobuf.descriptor_pb2 (stub), google.protobuf, google.protobuf.descriptor, google.protobuf.message, google.protobuf.reflection, google.protobuf.symbol_database
Analysis version 16 of 14 entries has started.
Analysis of os(Library) canceled (no AST yet).
Analysis of modules.control.proto.control_pb2(Stub) canceled (no AST yet).
Analysis of google.protobuf(Library) canceled (no AST yet).
Analysis of google.protobuf.descriptor(Library) canceled (no AST yet).
Analysis of google.protobuf.message(Library) canceled (no AST yet).
--
Analysis of google.protobuf.descriptor_pb2 (Stub) queued. Dependencies: google.protobuf.internal.containers (stub), google.protobuf.message (stub)
Analysis version 17 of 15 entries has started.
Analysis of os(Library) canceled (no AST yet).
Analysis of modules.control.proto.control_pb2(Stub) canceled (no AST yet).
Analysis of google.protobuf(Library) canceled (no AST yet).
Analysis of google.protobuf.descriptor(Library) canceled (no AST yet).
Analysis of google.protobuf.message(Library) canceled (no AST yet).
--
Analysis of google.protobuf.internal.containers (Stub) queued. Dependencies: google.protobuf.descriptor (stub), google.protobuf.internal.message_listener (stub), google.protobuf.message (stub)
Analysis version 19 of 16 entries has started.
Analysis of os(Library) canceled (no AST yet).
Analysis of modules.control.proto.control_pb2(Stub) canceled (no AST yet).
Analysis of google.protobuf(Library) canceled (no AST yet).
Analysis of google.protobuf.descriptor(Library) canceled (no AST yet).
Analysis of google.protobuf.message(Library) canceled (no AST yet).
--
Analysis of google.protobuf.internal.message_listener (Stub) queued. Dependencies: 
Analysis version 21 of 17 entries has started.
Analysis of os(Library) canceled (no AST yet).
Analysis of modules.control.proto.control_pb2(Stub) canceled (no AST yet).
Analysis of google.protobuf(Library) canceled (no AST yet).
Analysis of google.protobuf.descriptor(Library) canceled (no AST yet).
Analysis of google.protobuf.message(Library) canceled (no AST yet).
--
Import:  google.protobuf.message_factory /home/USER/.local/lib/python3.6/site-packages/google/protobuf/message_factory.py 
Analysis version 22 of 17 entries has started.
Analysis of os(Library) canceled (no AST yet).
Analysis of modules.control.proto.control_pb2(Stub) canceled (no AST yet).
Analysis of google.protobuf(Library) canceled (no AST yet).
Analysis of google.protobuf.descriptor(Library) canceled (no AST yet).
Analysis of google.protobuf.message(Library) canceled (no AST yet).
--
Import:  google.protobuf.internal.python_message /home/USER/.local/lib/python3.6/site-packages/google/protobuf/internal/python_message.py 
Analysis version 27 of 21 entries has started.
Analysis of os(Library) canceled (no AST yet).
Analysis of modules.control.proto.control_pb2(Stub) canceled (no AST yet).
Analysis of google.protobuf(Library) canceled (no AST yet).
Analysis of google.protobuf.descriptor(Library) canceled (no AST yet).
Analysis of google.protobuf.message(Library) canceled (no AST yet).
--
Import:  google.protobuf.text_format /home/USER/.local/lib/python3.6/site-packages/google/protobuf/text_format.py 
Analysis version 31 of 24 entries has started.
Analysis of os(Library) canceled (no AST yet).
Analysis of modules.control.proto.control_pb2(Stub) canceled (no AST yet).
Analysis of google.protobuf(Library) canceled (no AST yet).
Analysis of google.protobuf.descriptor(Library) canceled (no AST yet).
Analysis of google.protobuf.message(Library) canceled (no AST yet).
--
Analysis of termios (Compiled) queued. Dependencies: termios (stub)
Analysis of fcntl (CompiledBuiltin) queued. Dependencies: fcntl (stub)
Analysis of fcntl (Stub) queued. Dependencies: io (stub)
Analysis of modules.control.proto.control_pb2 (Stub) queued. Dependencies: sys, google.protobuf.descriptor, google.protobuf.message
Analysis of modules.control.proto.chassis_pb2 (Stub) queued. Dependencies: sys, google.protobuf.descriptor, google.protobuf.internal.containers, google.protobuf.message
Analysis of google.protobuf (Stub) queued. Dependencies: 
Analysis of google.protobuf (Library) queued. Dependencies: google.protobuf (stub)
Analysis of os (Library) queued. Dependencies: os (stub), abc, sys, errno, stat, posix, posixpath, ntpath, warnings, _collections_abc, subprocess, io
--
    google.protobuf.descriptor_pb2(Library)
    google.protobuf.message_factory(Library)
    google.protobuf.internal.python_message(Library)
Analysis of modules.control.proto.chassis_pb2 (Stub) on depth 1 completed in 10.05 ms.
Analysis of typing_extensions (Stub) queued. Dependencies: sys (stub)
Analysis of modules.control.proto.control_pb2 (Stub) on depth 1 completed in 3.58 ms.
Analysis version 135 has been completed in 699.56 ms with 93 entries analyzed and 5 entries skipped.
Analysis version 137 of 96 entries has started.
Analysis of google.protobuf.descriptor_pool(Library) canceled (no AST yet).
--
    google.protobuf.internal.python_message(Library)
    google.protobuf.internal.decoder(Library)
    google.protobuf.text_format(Library)
Analysis of modules.control.proto.control_pb2 (Stub) on depth 1 completed for library in 10.9 ms.
Analysis of modules.control.proto.chassis_pb2 (Stub) on depth 1 completed in 13.19 ms.
Analysis of modules.control.proto.chassis_pb2 (User) on depth 0 completed in 2.76 ms.
Analysis of exercise1.2.todo (User) on depth 0 completed in 7.14 ms.
Analysis version 796 of 434 entries has been completed in 4826.82 ms.
Hover in file:///apollo/modules/exercises/exercise1-basic/exercise1.2.todo.py at (24, 32)
Hover in file:///apollo/modules/exercises/exercise1-basic/exercise1.2.todo.py at (23, 33)
Hover in file:///apollo/modules/exercises/exercise1-basic/exercise1.2.todo.py at (15, 31)
Code Action in file:///apollo/modules/exercises/exercise1-basic/exercise1.2.todo.py at (40, 36) - (40, 36)
Analysis of exercise1.2.todo (User) queued. Dependencies: time, sys, termios, fcntl, os, modules.control.proto.control_pb2, modules.control.proto.chassis_pb2
Analysis version 797 of 1 entries has started.
Analysis of exercise1.2.todo (User) on depth 0 completed in 16.27 ms.
Analysis version 797 of 1 entries has been completed in 16.92 ms.
Analysis of exercise1.2.todo (User) queued. Dependencies: time, sys, termios, fcntl, os, modules.control.proto.control_pb2, modules.control.proto.chassis_pb2
Analysis version 798 of 1 entries has started.
Analysis of exercise1.2.todo (User) on depth 0 completed in 10.35 ms.
Analysis version 798 of 1 entries has been completed in 11.17 ms.

Code Snippet / Additional lnformation

I could provide chassis_pb2.pyi and control_pb2.pyi stub files if necessary, though I don't think there are any fundamental differences between these two files.

[MikhailArkhipov]

#1126 most probably

[MikhailArkhipov]

Any particular reason to use stubs vs adding typing to the primary code?

[jakebailey]

This is the way it's done for pytorch, and common for those who are still stuck in Python 2.

[jjyyxx]

@MikhailArkhipov
Python code generated by protoc can't get proper type inference with snippets like

_CONTROL_COMMAND = _descriptor.Descriptor(
  name='Control_Command',
  full_name='apollo.control.Control_Command',
  filename=None,
  file=DESCRIPTOR,
  containing_type=None,
  fields=[
    _descriptor.FieldDescriptor(
      name='steer_angle', full_name='apollo.control.Control_Command.steer_angle', index=0,
      number=1, type=2, cpp_type=6, label=1,
      has_default_value=False, default_value=float(0),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      options=None),
    _descriptor.FieldDescriptor(
      name='throttle', full_name='apollo.control.Control_Command.throttle', index=1,
      number=2, type=2, cpp_type=6, label=1,
      has_default_value=False, default_value=float(0),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      options=None),
  ],
  extensions=[
  ],
  nested_types=[],
  enum_types=[
  ],
  options=None,
  is_extendable=False,
  syntax='proto2',
  extension_ranges=[],
  oneofs=[
  ],
  serialized_start=55,
  serialized_end=111,
)

Control_Command = _reflection.GeneratedProtocolMessageType('Control_Command', (_message.Message,), dict(
  DESCRIPTOR = _CONTROL_COMMAND,
  __module__ = 'modules.control.proto.control_pb2'
  # @@protoc_insertion_point(class_scope:apollo.control.Control_Command)
  ))
_sym_db.RegisterMessage(Control_Command)

The tool mypy-protobuf provided by dropbox addressed this issue by providing in a stub file

class Control_Command(google___protobuf___message___Message):
    DESCRIPTOR: google___protobuf___descriptor___Descriptor = ...
    steer_angle = ... # type: builtin___float
    throttle = ... # type: builtin___float

    def __init__(self,
        *,
        steer_angle : typing___Optional[builtin___float] = None,
        throttle : typing___Optional[builtin___float] = None,
        ) -> None: ...
    if sys.version_info >= (3,):
        @classmethod
        def FromString(cls, s: builtin___bytes) -> Control_Command: ...
    else:
        @classmethod
        def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> Control_Command: ...
    def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ...
    def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ...
    def HasField(self, field_name: typing_extensions___Literal[u"steer_angle",b"steer_angle",u"throttle",b"throttle"]) -> builtin___bool: ...
    def ClearField(self, field_name: typing_extensions___Literal[u"steer_angle",b"steer_angle",u"throttle",b"throttle"]) -> None: ...
global___Control_Command = Control_Command

I believe a tool provided by dropbox should make some sense.

P.S. I write TypeScript more often than Python. I was expecting that .pyi files are to some extent similar to .d.ts files in TypeScript.

[MikhailArkhipov]

Sounds like duplicate of #1126.