Pylance language server 100% CPU usage while scanning folders in multi-root workspace #5070
Comments
|
Thanks for the issue. Can you share your code? It would also help to share the contents of your settings.json. Settings in there might cause us to open every file in the workspace, although it should just be looking for .py/.pyi files. Another thing that can affect this is what you have installed in your python environment. If you set this setting: "python.analysis.logLevel": "trace"This should show what files we're scanning. |
|
Hello @rchiodo!
If you mean my source code: my workspace spans 20GB of mostly internal non-python-related files. My python usage is mostly limited to simple scripts under 1000 lines of code. There are also lots of 3rd-party/upstream projects from github, etc... If you look for something specific, please let me know.
I found lots of settings.json files in my work directory.
This is what's left
I have set it, an will analyze the logs and report back if it happens again. |
|
My guess is this: {
"python.pythonPath": "/usr/bin/python3"
}Whatever you have installed in your root python will be scanned for each workspace. Everything installed there. For each workspace. The log should show us indexing all the files there. |
What do you mean by "root" and "workspace" here? root directory? So do you mean that each folder added to the workspace having I noticed this line now in the Pylance output:
As a fix: Should I remove all the settings.json entries with I get that trying to analyze any .py file found will be a resource hog. Can I turn off the Python extension except for projects/roots that actually contain python code I work on? I also noticed: While it is a WSL Remote workspace, when opening the Pylance log file withing VSCode itself, I get a little _ ! Select Interpreter _ button in the bottom bar. Clicking on it, it offers me a selection for each folder in my workspace, but with |
I actually meant all the packages installed in Yes, we will also look through all the workspace folders for python files, but that will start at each workspace folder root and probably won't be taking that long if you don't have a lot of python files. However, say you installed something like pandas into |
|
So is |
|
... and just now, my CPU fan started spinnning again, and I collected the start of the action: (it's now 9:32 here, and it's still spinning. Could this be the vscode settings sync with Microsoft triggering the phenomenon? |
|
I don't think it would be the settings sync. It'd probably be better to check what process is doing the spinning though. Something like Using this setting isn't normally done too: "python.pythonPath": "/usr/bin/python3"Because the Python extension will interpret that to mean the python environment installed there, which in this case is the global one. Normally you'd use this command in each workspace to set the environment:
But that doesn't actually store it in the settings.json. It's only stored on your local machine. If you wanted to use a virtual environment in the settings.json, you'd use this instead: "python.defaultInterpreterPath": ".venv/bin/python3"As described here: https://code.visualstudio.com/docs/python/environments#_manually-specify-an-interpreter |
|
The spinning (and the for-no-reason-3.5GB-RAM-eating) process is this one: I just suspected the settings sync as the trigger for pylance to start running, based on the log. And thanks for clarifying the |
|
Your log also doesn't look like it has this set: "python.analysis.logLevel": "Trace"It might be because I gave you the wrong case before. I said |
|
I added the setting to the user settings JSON, and it shows up in the GUI user settings, too. |
|
Hmm, weird. It should show a bunch of indexing and other stuff happening. Maybe it needs to be set here too:
But it doesn't for me, so I don't think that's it. Maybe the process is actually stuck before it gets to any of the work I'd expect. For example, opening a folder for me generates a whole bunch of output like so: 2023-11-02 03:24:54.396 [info] (Client) Pylance async client (2023.11.11) started with python extension (2023.20.0) [272d9b96e411ffb67ca640be28b35b424b40622b6ca3a5f79df231af3dc56a3e]
2023-11-02 03:24:56.266 [info] [Info - 3:24:56 AM] (24584) Pylance language server 2023.11.11 (pyright 64c9c3ff) starting
2023-11-02 03:24:56.267 [info] [Info - 3:24:56 AM] (24584) Server root directory: /home/rich/.vscode-server/extensions/ms-python.vscode-pylance-2023.11.11/dist
2023-11-02 03:24:56.271 [info] [Info - 3:24:56 AM] (24584) Starting service instance "test : with colon"
2023-11-02 03:24:56.467 [info] (24584) No configuration file found.
2023-11-02 03:24:56.467 [info] (24584) No pyproject.toml file found.
2023-11-02 03:24:56.467 [info] [Info - 3:24:56 AM] (24584) Setting pythonPath for service "test : with colon": "/bin/python"
2023-11-02 03:24:56.468 [info] [Info - 3:24:56 AM] (24584) Setting environmentName for service "test : with colon": "3.10.12 (global)"
2023-11-02 03:24:56.574 [info] [Info - 3:24:56 AM] (24584) Assuming Python version 3.10
2023-11-02 03:24:56.574 [info] (24584) Assuming Python platform Linux
2023-11-02 03:24:56.762 [info] [Info - 3:24:56 AM] (24584) Search paths for /home/rich/test/test : with colon
2023-11-02 03:24:56.762 [info] [Info - 3:24:56 AM] (24584) /home/rich/.vscode-server/extensions/ms-python.vscode-pylance-2023.11.11/dist/typeshed-fallback/stdlib
2023-11-02 03:24:56.762 [info] [Info - 3:24:56 AM] (24584) /home/rich/test/test : with colon
2023-11-02 03:24:56.762 [info] [Info - 3:24:56 AM] (24584) /home/rich/test/test : with colon/scripts
2023-11-02 03:24:56.762 [info] [Info - 3:24:56 AM] (24584) /home/rich/test/test : with colon/typings
2023-11-02 03:24:56.762 [info] [Info - 3:24:56 AM] (24584) /home/rich/.vscode-server/extensions/ms-python.vscode-pylance-2023.11.11/dist/typeshed-fallback/stubs/...
2023-11-02 03:24:56.763 [info] [Info - 3:24:56 AM] (24584) /home/rich/.vscode-server/extensions/ms-python.vscode-pylance-2023.11.11/dist/bundled/stubs
2023-11-02 03:24:56.763 [info] [Info - 3:24:56 AM] (24584) /usr/lib/python3.10
2023-11-02 03:24:56.763 [info] [Info - 3:24:56 AM] (24584) /usr/lib/python3.10/lib-dynload
2023-11-02 03:24:56.763 [info] [Info - 3:24:56 AM] (24584) /usr/local/lib/python3.10/dist-packages
2023-11-02 03:24:56.763 [info] [Info - 3:24:56 AM] (24584) /usr/lib/python3/dist-packages
2023-11-02 03:24:56.763 [info] [Info - 3:24:56 AM] (24584) Adding fs watcher for library directories:
/usr/lib/python3.10
/usr/lib/python3.10/lib-dynload
/usr/local/lib/python3.10/dist-packages
/usr/lib/python3/dist-packages
2023-11-02 03:24:56.763 [info] [Info - 3:24:56 AM] (24584) Adding fs watcher for directories:
/home/rich/test/test : with colon
2023-11-02 03:24:56.763 [info] (24584) Searching for source files
2023-11-02 03:24:56.814 [info] [Info - 3:24:56 AM] (24584) Found 2 source files
2023-11-02 03:24:56.814 [info] (24584) pytest configurations: {"message":"script","classes":["Test"],"files":["test_*.py","*_test.py"],"functions":["test"]}
2023-11-02 03:24:56.814 [info] (24584) pytest configurations: {"message":"script","classes":["Test"],"files":["test_*.py","*_test.py"],"functions":["test"]}
2023-11-02 03:24:57.005 [info] (24584) [FG] parsing: /home/rich/test/test : with colon/test # with hashtag/test.py (71ms)
2023-11-02 03:24:57.763 [info] (24584) [FG] parsing: /home/rich/.vscode-server/extensions/ms-python.vscode-pylance-2023.11.11/dist/typeshed-fallback/stdlib/builtins.pyi [fs read 15ms] (755ms)
2023-11-02 03:24:58.018 [info] (24584) [FG] binding: /home/rich/.vscode-server/extensions/ms-python.vscode-pylance-2023.11.11/dist/typeshed-fallback/stdlib/builtins.pyi (239ms)
2023-11-02 03:24:58.019 [info] (24584) [FG] binding: /home/rich/test/test : with colon/test # with hashtag/test.py (0ms)
2023-11-02 03:24:59.018 [info] (24584) pytest configurations: {"message":"request cancelled","classes":["Test"],"files":["test_*.py","*_test.py"],"functions":["test"]}
2023-11-02 03:24:59.279 [info] (24584) [BG(1)] getInlayHints range 0:0 - 3:9 at /home/rich/test/test : with colon/test # with hashtag/test.py ...
... |
|
Yes, where your screenshot says "Information", mine says "Trace". But my logs look similar to yours. (weekend now, be back on Monday...) |
|
I notice that it takes a very long time on my local checkout of Apache CloudStack: https://github.com/apache/cloudstack |
|
I tried apache cloudstack, no CPU hogging for me. It doesn't even have much python code in it. |
|
I tried on a different PC (Also Win10 + Debian 12 WSL) with a fresh installation of WSL. Enabled settings sync and installed my plugins as synced. I still get hogging, but not as long, and still high RAM usage of pylance (peak 1.4GB, stabilizes at 650MB). I also noticed (also on my main machine), that Git/SCM activities trigger the "hogging cycle", apparently pylance re-processes its things after being signalled by Git updates/File changes, even if no new commits came in or files were changed. When I find the time I will try more things (bisect repo combinations, etc.).... |
rchiodo
added
waiting for user response
|
Hey @rchiodo, this issue might need further attention. @MartinEmrich, you can help us out by closing this issue if the problem no longer exists, or adding more information. |
|
TBH for my and my CPU fan's sanity's sake, I lately tried to move away from vscode whenever possible.... but yes, I can still reproduce it. (Start vscode, language server scans every python file in my workspace roots. Switch over to the Git/SCM tab, and the pylance server starts rattling on). The logs also look still like the snippet above. If you need specific information, please let me know. |
github-actions
bot
added
user responded
|
I was unable to reproduce the problem, but it sounds like it has something to do with GIT. Maybe you have automatic fetch enabled? That would cause a rescan, and possibly a lot of them if we were getting file notifications for each file one at a time. Although I thought git changes all of the files in a big chunk. There would be file watcher notifications in the log in that case though. Without a way to reproduce this, there's really nothing we can do. Did you have a specific branch of cloudstack checked out? Maybe I can try going back a bunch of commits and then having VS code do a sync. |
|
I do not think I have automatic fetch on (in fact I have to click on the sync button for >20 repos every morning, and I would love to see that automated). What I notice:
On that cloudstack repo, I was on the main branch all along. An fwiw, I see not special log entries regarding the cloudstack repo, but I have a feeling that it finds more "scan targets", "work packages", whatever it is called, within that repository folder than in all other repos I have open together. I can find 1395 *.py files in my whole work dir, 630 of them in the cloudstack repo. And I guess that amplifies the effect. But that's IMHO not all: if I remove the cloudstack repo from my workspace, the effect takes not 1/2, but way less than 1/10th of the time. A nice, acceptable workaround would be an option to enable all python features only on a select opt-in set of workspace folders, thus I could only enable it for repos/folders I actually work in. |
|
Can you do this Without live reproduction or detailed logging (Trace), it is hard to determine what is happening. That said, a few things that might cause the issue you're seeing include having some settings set somewhere in However, it would be impossible for us to guess how this happened without the detailed log containing the final settings we received from your vscode and information on what is rerunning and why. also, make sure you use the latest |
heejaechang
added
waiting for user response
Why automatic fetch reruns pylance analysis? Fetch is not changing files in project folder. Only in .git folder.
To me correct solution would be to run analysis only on files where change is detected. I have also noticed that binding big file BG and FG takes around 12 seconds. Each 6 seconds. Checking and analyzing is 2 seconds in total. getSemanticTokens is 6 seconds. File size is 4MB. In file there are just variable names with type inside class. For example: If settings.json is triggering rescan maybe solution can be to detect what has changed in settings.json and if rescan is needed at all. I think this approach is the correct one to solve a lot of issues related to rescaning. I have only one settings.json file in whole project. Before you ask for the code please make a testing environment with multiple submodules and big python files. If you are too busy and there is interest to fix this issue I can prepare testing environment for you. Then I expect that someone will work on this issue. |
|
I found no unexpected I turned off the `python.analysis.indexing checkbox. On clicking it, the mouse cursor froze for a second, so something must have happened. I have no blue bubble on the plugins, so I assume I have everything up to date. After clicking the Git tab, again the python laguage server did it's rattling over ca. 2 minutes. But on a personal note: Not only due to this issue, but for the general bloat that crept into vscode and its plugin ecosystem (I have 3 idle small java projects and this many-small-projects-workspace open, and already 6GB RAM are wasted), I started moving to Neovim. Also I fear Microsoft as a company forcing their GenAI hype tech more aggressively into vscode, too, as they do in their other services. Time to jump ship. Indeed this is the first time I opened vscode on that workspace for trying your suggestions since the last message here. I still thank you all for your attempts in trying to help, but if you are busy, I won't mind if you close this issue with a "won't fix". |
github-actions
bot
added
user responded
|
any change on |
|
we found an issue that we will fix in next prerelease around |
|
is this still repro? if it does, please re-activate the issue with this info - https://github.com/microsoft/pylance-release/wiki/Collecting-data-for-an-investigation. |
|
I re-enabled it and it didn't work. |
Is there a link to commits what has been fixed? I see the issue is closed but no visible changes in pylance under release or pre-release. |
There's no way to link our commits as Pylance is private but we did fix a problem where we were incorrectly watching the '.git' folder. |
Since ca. 4-6 weeks, I irregularly notice high CPU load on a VSCode process related to pylance, CPU fan quickly goes to full throttle.
It seems like it does an expensive operation for every folder in my multi-root workspace, even though most are not python-related.
Environment data
Logs
At start:
(repeated for each of my ~20-30 folders opened in my workspace)
Regularly:
I did not configure anything to make it assume that each folder is a "service" in any sense whatsoever.
As it takes 5-15 minutes despite running on a modern 8-Core/32GB-RAM/SSD machine. I tooks like it either does way more than a
find . -name \*.pyequivalent, or something is broken.The text was updated successfully, but these errors were encountered: