Python extension locks up VSCode during startup

[PeterJCLaw]

Environment data

• VS Code version:

Version: 1.45.1
Commit: 5763d909d5f12fe19f215cbfdd29a91c0fa9208a
Date: 2020-05-14T08:27:22.494Z
Electron: 7.2.4
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Linux x64 5.3.0-59-generic

• Extension version (available under the Extensions sidebar): 2020.5.86806
• OS and version: Ubuntu 18.04.4
• Python version (& distribution if applicable, e.g. Anaconda): OS has Python 2.7.17 and 3.6.9. I've also got 3.5.9, 3.7.7 and 3.8.3 installed via the deadsnakes PPA
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): OS Python, virtualenv (via mkvirtualenv)
• Jedi or Language Server? "python.jediEnabled": true
• Value of the python.languageServer setting: Microsoft

Expected behaviour

Initial startup of the Python extension is essentially a background activity.

Actual behaviour

Initial startup of the Python extension locks up VSCode and creates sufficient load that other programs slow down.

This is only noticeable on the first startup of the extension after a reboot or a long period of inactivity. Just restarting VSCode doesn't reproduce this.

Steps to reproduce:

1. start VSCode
2. open any Python file
3. try to click around within that file while the Python extension is starting up

[PeterJCLaw]

Based on the non-reproducibility on just restarting VSCode I'm guessing that this could be related to the extension doing lots of disk IO when it first starts, but which it doesn't need to do later on? And/or those files are then cached, possibly by the OS?

I've only noticed this recently; I'd guess the last few weeks or so. In that time I've upgraded (in place) from Ubuntu 16 to Ubuntu 18 and the latest version of the extension included some changes to the way that Pythons are found (related to the removal of pythonPath).

Could this be related to changes in the way that virtualenvs are found? Does VSCode run python in each discovered virtualenv? (I have about 50 virtualenvs in ~/.virtualenvs, on a HDD, some of which I know are a bit broken due to my OS upgrade)

[karthiknadig]

which it doesn't need to do later on? And/or those files are then cached, possibly by the OS?

There is some caching that we do, but on first startup we run each python instance we find to get the full executable path by running import sys;print(sys.executable). If we get a response to this, we then run interpreterInfo.py to get additional details.

Could this be related to changes in the way that virtualenvs are found? Does VSCode run python in each discovered virtualenv?

We have not changed how virtualenvs are discovered. But as mentioned before, we do run python in each discovered virtualenv, to get the full executable path.

[PeterJCLaw]

Hrm, thanks for that.

I've removed the broken virtualenvs I had and today ran the following to warm things up. This definitely seemed to make the extension faster to start up and VSCode remained responsive while it did so.

for p in $(find ~/.virtualenvs/ -name python -not -type d)
do
  $p ~/.vscode/extensions/ms-python.python-2020.5.86806/pythonFiles/interpreterInfo.py
done

That pre-warmup took a while to run, however didn't seem to have any adverse effects while it did so.

Is the Python extension launching all the Python processes in parallel? If so, that feels like it could contribute to this.

I also wonder whether the extension needs to examine all of them at startup? Could it perhaps load the information lazily, only when the user opts to select a new interpreter?

I realise that there is a need to find an interpreter for the project (if one is not configured), though maybe it would be possible to limit that exploration to the "system" Pythons?
I also realise that "system" is a bit vague, though hopefully a heuristic could be found which meant that only a handful of paths were tried, rather than potentially all the possibly hundreds of virtualenvs a user might have?

[karthiknadig]

We are refactoring the entire python discovery process. We plan on addressing several issues with respect to performance and discovering python interpreters. One of the things we plan on doing is avoid blocking extension load on discovering interpreters.

[karthiknadig]

@PeterJCLaw Do let me know if you have any additional questions.

[PeterJCLaw]

Thanks, it sounds like this is already known/being worked on 👍.
I'm not sure why I've only started noticing this recently, though it's possible that it's related to other system changes, rather than changes in vscode-python.