Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python Language Server uses too much RAM / Memory #2594

Closed
nojvek opened this Issue Sep 14, 2018 · 45 comments

Comments

Projects
None yet
@nojvek
Copy link

nojvek commented Sep 14, 2018

Environment data

  • VS Code version: Version 1.27.0 (1.27.0)

  • Extension version (available under the Extensions sidebar): 2018.8.0

  • OS and version: Mac Snow Leopard

  • Python version (& distribution if applicable, e.g. Anaconda): Python2

Honestly there should just be a command that generates this

Actual behavior

Machine crawls to a halt. Using Python's language server using 30GB+ happens after a couple of hours of use.

Expected behavior

Sonic like universe. FAST!

Steps to reproduce:

  1. Use vscode with python for a while. We have a somewhat large repo with 10k+ python files.

@DonJayamanne DonJayamanne changed the title Python Language Server uses wayyyy too much RAM / Memory Python Language Server uses too much RAM / Memory Sep 14, 2018

@hologerry

This comment has been minimized.

Copy link

hologerry commented Sep 15, 2018

I have the same issue with VS Code version 1.27.2

@hologerry

This comment has been minimized.

Copy link

hologerry commented Sep 15, 2018

Cause to VS Code not responding...

@tadly

This comment has been minimized.

Copy link

tadly commented Sep 16, 2018

Same here and unlike #2553, I have jedi disabled.

@JPhillot

This comment has been minimized.

Copy link

JPhillot commented Sep 17, 2018

Same here.

I use vscode as git editor and also as mergetool.
My git config:

# .gitconfig
[core]
    editor = code --wait
[difftool "vscode"]
    cmd = code --wait --diff $LOCAL $REMOTE
    keepBackup = false
    trustExitCode = true
[merge]
    tool = vscode
    conflictstyle = diff3
[mergetool "vscode"]
    cmd =  code --wait $MERGED
    keepBackup = false
    trustExitCode = true

After each merge conflict fixed in vscode, the server seems to not release the used RAM.

@blaylockbk

This comment has been minimized.

Copy link

blaylockbk commented Sep 17, 2018

The python extension is using nearly all my memory on my desktop computer, too. Task Manager says memory is eaten up by Microsoft.Python.LanguageServer

@krmozejko

This comment has been minimized.

Copy link

krmozejko commented Sep 18, 2018

I have same issue on Linux Mint 18.3 with VS Code Version: 1.27.2 and extension 2018.8.0. After some time it uses all the RAM and starts flooding swap.

@brettcannon

This comment has been minimized.

Copy link
Member

brettcannon commented Sep 18, 2018

For those of you bumping up against this, a reminder that setting "python.jediEnabled": true in your settings will switch you back to Jedi and off the language server.

@brettcannon brettcannon added type-perf and removed type-bug labels Sep 18, 2018

@brettcannon

This comment has been minimized.

Copy link
Member

brettcannon commented Sep 18, 2018

And if anyone has a repo that can reproduce the problem that they can share that would be greatly appreciated as debugging a potential cause is nearly impossible otherwise.

@tadly

This comment has been minimized.

Copy link

tadly commented Sep 18, 2018

I already did switch back to jedi for the time being.

As for a share-able repo, I've mostly been working on dups, did switch between projects using the project manager extension though.
Not sure if this might have to do anything with it.

@brettcannon

This comment has been minimized.

Copy link
Member

brettcannon commented Sep 18, 2018

[ @nojvek I edited your ✉️ ⬆️ to come off as more 😃 and less 😠 ]

@MikhailArkhipov

This comment has been minimized.

@Scalamoosh

This comment has been minimized.

Copy link

Scalamoosh commented Sep 20, 2018

I have the same issue using python3 on a small/new codebase (can’t share) where colleagues on same codebase works fine.

I also have the project manager plugin installed.

@nojvek

This comment has been minimized.

Copy link
Author

nojvek commented Sep 20, 2018

Thank you @brettcannon , Apologies for the tone.

@brettcannon

This comment has been minimized.

Copy link
Member

brettcannon commented Sep 20, 2018

I'm going to leave this open, but hopefully we will have the fixes Mikhail and the language server team have made in the next release of the extension (at worst the one following).

@nojvek no problem, we all get cranky sometimes. 😄 And I'll also mention that #1119 covers your request for a command to gather system details we ask for.

@brettcannon brettcannon removed their assignment Sep 20, 2018

@heyakyra

This comment has been minimized.

Copy link

heyakyra commented Sep 26, 2018

@brettcannon are the releases on a regular schedule or just "when it's ready"?

@brettcannon

This comment has been minimized.

Copy link
Member

brettcannon commented Sep 27, 2018

@heyakyra

This comment has been minimized.

Copy link

heyakyra commented Sep 27, 2018

Thanks, for me I have found that enabling jedi solves the issue, strangely enough.

@brettcannon

This comment has been minimized.

Copy link
Member

brettcannon commented Oct 1, 2018

@tadly you mentioned that you were switching between stuff using another extension, so I had not dived into trying the repo yet. Plus we are hoping to do a new release soon that will hopefully fix this.

@scorphus

This comment has been minimized.

Copy link

scorphus commented Oct 1, 2018

Hey, @brettcannon. Thanks for your reply! I think you can reproduce this behavior with thefuck's source code (at least this is the source code I'm having memory leak problems with). I'm on Linux 4.15 x86_64 and Python 3.7.0. Please let me know if you need any additional info. Also, I'd love to test any alpha/beta/RC.

@brettcannon

This comment has been minimized.

Copy link
Member

brettcannon commented Oct 2, 2018

@scorphus if you install a development build then you should be able to test the current beta version of the language server.

@hmnd

This comment has been minimized.

Copy link

hmnd commented Oct 3, 2018

@brettcannon Still happens on vscode insider, dev build of extension.

screenshot

@xen

This comment has been minimized.

Copy link

xen commented Oct 3, 2018

I'm also have this issue with latest python (3.7), OSX (Mojave). Also, I have Visual Studio IntelliCode which require Language Server.

@NdagiStanley

This comment has been minimized.

Copy link

NdagiStanley commented Oct 3, 2018

Same here

@heyakyra

This comment has been minimized.

Copy link

heyakyra commented Oct 3, 2018

I first encountered this problem on vscode insiders (since stable doesn't work on Fedora 29)

@MikhailArkhipov

This comment has been minimized.

Copy link
Member

MikhailArkhipov commented Oct 4, 2018

@scorphus, @NdagiStanley, @xen, @hmnd

Can you try setting python.analysis.watchSearchPaths to false and see if it helps.

@hmnd

This comment has been minimized.

Copy link

hmnd commented Oct 4, 2018

@MikhailArkhipov That config setting apparently doesn't exist.

screenshot

@MikhailArkhipov

This comment has been minimized.

Copy link
Member

MikhailArkhipov commented Oct 4, 2018

@hmnd - it does, it is simply not currently public (i.e. not listed in package.json). There are some development settings that rarely matter to regular users.

@hmnd

This comment has been minimized.

Copy link

hmnd commented Oct 4, 2018

@MikhailArkhipov Looks like ram usage remains incredibly high but it doesn't grow as quickly as before...

screenshot

@xen

This comment has been minimized.

Copy link

xen commented Oct 4, 2018

@MikhailArkhipov doesn't help. I'm tried to restart VS Code after added this flag, but now it started to consume all resources again:

screen shot 2018-10-04 at 15 46 22

If I kill process I'm getting this error on Output tab:

[Info  - 3:47:40 PM] Connection to server got closed. Server will restart.
[Error - 3:47:40 PM] Request textDocument/documentSymbol failed.
Error: Connection got disposed.
	at Object.dispose (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/main.js:825:25)
	at Object.dispose (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-languageclient/lib/client.js:57:35)
	at LanguageClient.handleConnectionClosed (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-languageclient/lib/client.js:2036:42)
	at LanguageClient.handleConnectionClosed (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-languageclient/lib/main.js:127:15)
	at closeHandler (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-languageclient/lib/client.js:2023:18)
	at CallbackList.invoke (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/events.js:71:39)
	at Emitter.fire (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/events.js:135:36)
	at closeHandler (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/main.js:221:26)
	at CallbackList.invoke (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/events.js:71:39)
	at Emitter.fire (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/events.js:135:36)
	at StreamMessageReader.AbstractMessageReader.fireClose (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/messageReader.js:135:27)
	at Socket.<anonymous> (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/messageReader.js:188:62)
	at emitOne (events.js:121:20)
	at Socket.emit (events.js:211:7)
	at Pipe._handle.close [as _onclose] (net.js:554:12)
Initializing for /Users/xen/.local/share/virtualenvs/notify-1dEpo5ac/bin/python

Upd. In status line also shown spinning arrows circle with text "Analyzing workspace, 14 items remaining..."

@scorphus

This comment has been minimized.

Copy link

scorphus commented Oct 4, 2018

@MikhailArkhipov, thanks for your message.

I tried with "python.analysis.watchSearchPaths": false as you instructed but the behaviour persists, memory usage keeps increasing.

Maybe worth to mention, I'm running with only 8G of RAM and I don't know if memory usage stops increasing at some point because the system runs out of memory if I don't quit VSCode.

This is the output of grep python ~/.config/Code/User/settings.json:

    "python.linting.enabled": true,
    "python.linting.lintOnSave": true,
    "python.linting.pylintEnabled": false,
    "python.linting.pylintCategorySeverity.convention": "Hint",
    "python.linting.pylintCategorySeverity.warning": "Error",
    "python.analysis.watchSearchPaths": false,
    "python.linting.flake8Enabled": true,

After re-downloading language server, I see the following in the output panel:

Starting Microsoft Python language server.
Initializing for /.../.pyenv/versions/thefuck-370/bin/python
##########Linting Output - flake8##########
##########Linting Output - flake8##########
[Error - 3:10:43 PM] System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.Collections.Generic.HashSet`1.Enumerator.MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.OfTypeIterator[TResult](IEnumerable source)+MoveNext()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
   at Microsoft.PythonTools.Analysis.PythonAnalyzer.GetEntriesThatImportModule(String moduleName, Boolean includeUnresolved)
   at Microsoft.Python.LanguageServer.Implementation.Server.OnDocumentChangeProcessingCompleteAsync(IDocument doc, VersionCookie vc, Boolean enqueueForAnalysis, AnalysisPriority priority, IDisposable disposeWhenEnqueued)

I hope this helps. I'm still looking forward to a chance (aka free time) to test the development build, as @brettcannon suggested. I'll do so ASAP.

Thank you all for your time and support! 👍

@MikhailArkhipov

This comment has been minimized.

Copy link
Member

MikhailArkhipov commented Oct 4, 2018

@SynneK1337

This comment has been minimized.

Copy link

SynneK1337 commented Oct 7, 2018

Same on macOS 10.14 (18A391), VSCode 1.27.2 and vscode-python 2018.8.0. It used 22GB of RAM on my machine xD

@DonJayamanne

This comment has been minimized.

Copy link

DonJayamanne commented Oct 9, 2018

@nojvek @tadly @SynneK1337 @scorphus @xen @hmnd @heyakyra @ahter @blaylockbk @JPhillot

We believe this issue has been resolved.
We just released the latest version of the Python Extension, please update to this new version 2018.9.0 and give it a go.

@hmnd

This comment has been minimized.

Copy link

hmnd commented Oct 9, 2018

Looks like it's fixed for me @DonJayamanne

@xen

This comment has been minimized.

Copy link

xen commented Oct 10, 2018

I tried and its works. Thank you very much for this changes and for your amazing work.

@nojvek

This comment has been minimized.

Copy link
Author

nojvek commented Oct 10, 2018

Thanks. Y’all so 😎.

I’m just curious what the technical issue was that caused such a collosal memory conquest?

@brettcannon

This comment has been minimized.

Copy link
Member

brettcannon commented Oct 10, 2018

@nojvek If I remember correctly it was from the language server constantly re-processing modules as they changed but not freeing the memory from the old analyses.

@Microsoft Microsoft locked as resolved and limited conversation to collaborators Oct 10, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.