Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to set pylance memory max limitation? #5799

Open
Mactarvish opened this issue Apr 25, 2024 · 6 comments
Open

How to set pylance memory max limitation? #5799

Mactarvish opened this issue Apr 25, 2024 · 6 comments
Assignees
@bschnurr
Labels
needs repro Issue has not been reproduced yet user responded Was "waiting for user response" and they responded

Comments

@Mactarvish
Copy link

Pylance works fine when workspace is small but crashed when huge.
Pylance has crashed. Would you like to try using a Node.js executable to run the language server?

Report issue is

2024-04-25 11:25:35.477 [info] (Client) Pylance async client (2024.4.1) started with python extension (2024.4.1)
2024-04-25 11:25:36.064 [info] [Info  - 11:25:36 AM] (1667316) Pylance language server 2024.4.1 (pyright version 1.1.356, commit 6652c4a8) starting
2024-04-25 11:25:36.064 [info] [Info  - 11:25:36 AM] (1667316) Server root directory: file:///home/mck/.vscode-server/extensions/ms-python.vscode-pylance-2024.4.1/dist
2024-04-25 11:25:36.064 [info] [Info  - 11:25:36 AM] (1667316) Starting service instance "lrcn"
2024-04-25 11:25:36.117 [info] [Info  - 11:25:36 AM] (1667316) Setting environmentName for service "lrcn": "D:\anaconda3\python.exe"
2024-04-25 11:25:46.295 [info] [Error - 11:25:46 AM] (1667316) Enumeration of workspace source files is taking longer than 10 seconds.
This may be because:
* You have opened your home directory or entire hard drive as a workspace
* Your workspace contains a very large number of directories and files
* Your workspace contains a symlink to a directory with many files
* Your workspace is remote, and file enumeration is slow
To reduce this time, open a workspace directory with fewer files or add a pyrightconfig.json configuration file with an "exclude" section to exclude subdirectories from your workspace. For more details, refer to https://github.com/microsoft/pyright/blob/main/docs/configuration.md.
2024-04-25 11:27:50.685 [info] 
<--- Last few GCs --->

[1667316:0x5c6aa10]   132258 ms: Mark-sweep 4055.7 (4139.7) -> 4042.8 (4142.5) MB, 966.2 / 0.0 ms  (average mu = 0.340, current mu = 0.057) allocation failure; scavenge might not succeed
[1667316:0x5c6aa10]   135181 ms: Mark-sweep 4058.5 (4142.5) -> 4045.5 (4144.7) MB, 2862.7 / 0.0 ms  (average mu = 0.158, current mu = 0.021) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

2024-04-25 11:27:50.686 [info]  1: 0xb85bc0 node::Abort() [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.686 [info]  2: 0xa94834  [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.687 [info]  3: 0xd66d10 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.687 [info]  4: 0xd670b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.688 [info]  5: 0xf447c5  [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.688 [info]  6: 0xf56cad v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.689 [info]  7: 0xf313ae v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.689 [info]  8: 0xf32777 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.690 [info]  9: 0xf1394a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.691 [info] 10: 0x12d8caf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.691 [info] 11: 0x1705b39  [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:51.210 [info] [Error - 11:27:51 AM] Server process exited with signal SIGABRT.
2024-04-25 11:27:58.346 [info] [Error - 11:27:58 AM] Notify file events failed.
2024-04-25 11:27:58.346 [info] Error: Connection is disposed.
	at pe (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308075)
	at Object.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308918)
	at _.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2352539)
	at async t.Delayer.<anonymous> (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2364453)
2024-04-25 11:27:59.034 [info] [Error - 11:27:59 AM] Notify file events failed.
2024-04-25 11:27:59.034 [info] Error: Connection is disposed.
	at pe (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308075)
	at Object.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308918)
	at _.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2352539)
	at async t.Delayer.<anonymous> (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2364453)
2024-04-25 11:28:25.819 [info] [Error - 11:28:25 AM] Request textDocument/codeAction failed.
2024-04-25 11:28:25.819 [info] Error: Connection is disposed.
	at pe (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308075)
	at Object.sendRequest (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2310379)
	at _.sendRequest (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2351280)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
	at async N.provideCodeActions (/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/out/vs/workbench/api/node/extensionHostProcess.js:154:95620)
2024-04-25 11:28:25.961 [info] [Error - 11:28:25 AM] Notify file events failed.
2024-04-25 11:28:25.961 [info] Error: Connection is disposed.
	at pe (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308075)
	at Object.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308918)
	at _.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2352539)
	at async t.Delayer.<anonymous> (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2364453)
2024-04-25 11:28:26.557 [info] [Error - 11:28:26 AM] Notify file events failed.
2024-04-25 11:28:26.557 [info] Error: Connection is disposed.
	at pe (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308075)
	at Object.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308918)
	at _.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2352539)
	at async t.Delayer.<anonymous> (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2364453)
2024-04-25 11:28:29.889 [info] [Error - 11:28:29 AM] Request textDocument/definition failed.
2024-04-25 11:28:29.890 [info] Error: Connection is disposed.
	at pe (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308075)
	at Object.sendRequest (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2310379)
	at _.sendRequest (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2351280)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
	at async h.provideDefinition (/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/out/vs/workbench/api/node/extensionHostProcess.js:154:92639)

As shown above : FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

But I have 256G memory on my server which is enough for pylance. How to set the max limitation of memory occupy?
@github-actions github-actions bot added the needs repro Issue has not been reproduced yet label Apr 25, 2024
@Mactarvish
Copy link
Author

Otherwise, if there anyway to let pylance NOT analysis special directory or soft link? I've found that when I create a soft link to my HUGE dataset that causes pylance crash.

@rchiodo
Copy link
Contributor

rchiodo commented Apr 25, 2024

There is an exclude option:

https://github.com/microsoft/pyright/blob/main/docs/configuration.md

@rchiodo
Copy link
Contributor

rchiodo commented Apr 25, 2024

Additionally see the directions here:
https://github.com/microsoft/pylance-release/blob/main/TROUBLESHOOTING.md#pylance-is-crashing

@rchiodo
Copy link
Contributor

rchiodo commented Apr 25, 2024 •
edited

However, we'd rather just fix your memory issue. Can you share your source so we can try and fix the root cause of the problem?

@rchiodo rchiodo added the waiting for user response Requires more information from user label Apr 25, 2024
@rchiodo
Copy link
Contributor

rchiodo commented Apr 25, 2024 •
edited

I tried an experiment with a recursive symlinked folder.

Folder looked like this without linking:

- non_linked
  - subfolder
    - __init__.py
    - funcs.py 
  - __init__.py
- importer.py

Where importer.py looks like this:

from exporter import *

And funcs.py looks like this:

from recursive import *
def foo_func():
   return 4

And the __init__.py files just export all the symbols inside of them.

Then I symlinked non_linked to exporter and inside of subfolder I added a recursive link to non_linked.

Running this in Pylance I get indexing results like so:

2024-04-25 23:28:56.272 [info] (83551) [IDX(3)]   index execution environment file:///home/rich/tests/test_symlinks [found 1218 in 17 files] (14ms)

With output like so:


[BG(1)]                                               binding: file:///home/rich/tests/test_symlinks/non_linked/subfolder/recursive/subfolder/recursive/subfolder/recursive/subfolder/recursive/subfolder/recursive/subfolder/recursive/subfolder/recursive/subfolder/recursive/__init__.py (54ms)

And it chews up memory pretty fast. I'm honestly not sure what eventually stops it. We might hit some maximum import resolution depth.

@Mactarvish
Copy link
Author

Thanks for your helps. Seems the key point is not depth of directory but num of files. In the soft link under my workspace there's more than 50000000 .jpg files and cause pylance crash. After move the soft link out of my workspace and reload vscode, pylance reworks well.

@github-actions github-actions bot added user responded Was "waiting for user response" and they responded and removed waiting for user response Requires more information from user labels Apr 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bschnurr bschnurr

Labels
needs repro Issue has not been reproduced yet user responded Was "waiting for user response" and they responded
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bschnurr @Mactarvish @rchiodo