Missing Autocomplete in VSCode / PyCharm

[wlwwt]

Bug Description

Hello together,

I use bytewax within a virtual environment. Environment was created under Python 3.10.9 using the command
python -m venv .venv
and then I activated the environment. I installed bytewax via pip and can create imports.

from bytewax.dataflow import Dataflow

The problem is that "Dataflow" is not found within the IDE. This means that autocomplete, etc. do not work.

However, I can execute and use Dataflow without any problems. It is only annoying within the IDEs.

Python version (python -V)

Python 3.10.9

Bytewax version (pip list | grep bytewax)

0.14.0

Relevant log output

Cannot find reference 'Dataflow' in 'dataflow.py'

Steps to Reproduce

1. Create new project in PyCharm / VSCode (under MacOS 13.1)
2. Create and activate venv
3. pip install bytewax
4. create new .py file with
   from bytewax.dataflow import Dataflow

[Psykopear]

I'm trying to replicate the problem (I'm on linux), but it seems to work.
This is what I did:

mkdir bytewax-bug
cd bytewax-bug
python -m venv .venv
source .venv/bin/activate
pip install bytewax
code main.py

Code has the ms-python extension installed:

Starting to write the import it works:

It also works if I run code from outside the virtualenv.
I tried pycharm community edition too, created the virtualenv and installed bytewax from whithin the IDE, and it worked there too:

Maybe it depends on the language server/python plugin used? I installed a fresh version of both code and pycharm, so the setup is pretty much the default one.

[wlwwt]

Thanks for your test. I just did a fresh installation on VSCode and entered same lines of code from your example above.
Same Extension installed. Only difference seems to be the OS.

Anybody able to confirm my behaviour on MacOS?

[kchristakis52]

I can confirm the issue on MacOS

[zhenyakovalyov]

yep, seeing the same on macOS/vscode

[awmatheson]

Unfortunately, these don't get automagically created by Maturin/PyO3 at this point and need to be manually created as a pyi stub file. There is an issue open with a proposal to automate this as it would be quite cumbersome to maintain.

[whoahbot]

I just reproduced this error on a mac mini here, and in the Python extension server log file, I'm seeing the following error:

2023-11-06 09:43:48.736 [info] [Info  - 9:43:48 AM] (841) Pylance language server 2023.11.10 (pyright 088ebaa5) starting
2023-11-06 09:43:48.736 [info] [Info  - 9:43:48 AM] (841) Server root directory: /Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist
2023-11-06 09:43:48.736 [info] [Info  - 9:43:48 AM] (841) Starting service instance "bytewax-test"
2023-11-06 09:43:48.751 [info] [Info  - 9:43:48 AM] (841) Setting pythonPath for service "bytewax-test": "/Users/whoahbot/.pyenv/versions/3.11.2/envs/test-bytewax/bin/python"
2023-11-06 09:43:48.751 [info] [Info  - 9:43:48 AM] (841) Setting environmentName for service "bytewax-test": "3.11.2 (test-bytewax venv)"
2023-11-06 09:43:48.869 [info] [Info  - 9:43:48 AM] (841) Assuming Python version 3.11
2023-11-06 09:43:48.916 [info] [Info  - 9:43:48 AM] (841) No source files found.
2023-11-06 09:45:41.582 [info] [Info  - 9:45:41 AM] (841) Found 1 source file
2023-11-06 09:46:15.015 [info] [Error - 9:46:15 AM] (841) An internal error occurred while parsing file "/Users/whoahbot/.pyenv/versions/3.11.2/envs/test-bytewax/lib/python3.11/site-packages/bytewax/bytewax.cpython-311-darwin.so": RangeError: Maximum call stack size exceeded
    at t.Tokenizer._skipToEndOfStringLiteral (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:1441887)
    at t.Tokenizer._handleString (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:1440447)
    at t.Tokenizer._handleCharacter (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:1426514)
    at t.Tokenizer._addNextToken (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:1426238)
    at t.Tokenizer.tokenize (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:1424281)
    at I._startNewParse (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:1353067)
    at /Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:1351401
    at a.timeOperation (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:1126304)
    at I.parseSourceFile (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:1351376)
    at _0x520120._parseFile (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:511687)
    at /Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:498412
    at t.LogTracker.log (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:1089150)
    at _0x520120.parse (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:497678)
    at N._parseFile (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:432471)
    at t.SourceMapper._boundSourceGetter (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:426950)
    at t.SourceMapper.findModules (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:517368)
    at g (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:546473)
    at _0x446ce3.resolveCompletionItem (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:1185672)
    at _0x22ae58.<computed>.resolveCompletionItem (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/server.bundle.js:1:569185)
    at /Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/server.bundle.js:1:998142
    at /Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/server.bundle.js:1:251718
    at /Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:419605
    at runWithCancellationToken (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:842748)
    at a.timeOperation (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:1126304)
    at N._runEvaluatorWithCancellationToken (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:424949)
    at N.run (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:419563)
    at k.run (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/pyright.bundle.js:1:467291)
    at _0x1905c1.<computed>._runCore (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/server.bundle.js:1:251706)
    at _0x1905c1.<computed>.run (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/server.bundle.js:1:250389)
    at _0x108e2b.onCompletionResolve (/Users/whoahbot/.vscode/extensions/ms-python.vscode-pylance-2023.11.10/dist/server.bundle.js:1:997498)

Are you all seeing similar errors?

[whoahbot]

Looks like this will be fixed in the next release of pylance: microsoft/pylance-release#5090 (comment)

[davidselassie]

There are a few things being discussed in this issue, but I believe on all counts this is fixed:

• The latest release of Bytewax https://github.com/bytewax/bytewax/releases/tag/v0.18.0 now has all of the dataflow building API defined Python-side and that entire API has type hints, so autocomplete and type hints should work everywhere if you use this new version. Though we do not maintain stub files for the PyO3 bindings, the amount of public API that is directly exposed through those bindings is now much smaller.

• The latest releases of Pylance properly handle native extensions, should not crash, and will thus show autocomplete for the functions that it can show. So if you are using an older version of Bytewax (this issue was opened for v0.14), you should see a few more completions, but since all the operator methods are defined through PyO3, the improvement will be minimal. We encourage you to upgrade to the latest version of Bytewax since it enables type checking in addition to completions.