Out of memory crash pyright-langserver

[gilbertfrancois]

I sorry to reopen issue #3239. I'm using pyright 1.1.348. The pyright language server crashes as seen in the LspLog output of neovim:

[START][2024-01-29 14:32:07] LSP logging initiated
[ERROR][2024-01-29 14:32:07] .../vim/lsp/rpc.lua:734	"rpc"	"/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"	"stderr"	"\n<--- Last few GCs --->\n\n[1468394:0x6a30cf0]    66991 ms: Scavenge 4020.6 (4123.2) -> 4015.1 (4123.9) MB, 6.42 / 0.00 ms  (average mu = 0.472, current mu = 0.326) allocation failure; \n[1468394:0x6a30cf0]    67009 ms: Scavenge 4021.7 (4123.9) -> 4016.1 (4140.4) MB, 7.00 / 0.00 ms  (average mu = 0.472, current mu = 0.326) allocation failure; \n[1468394:0x6a30cf0]    69137 ms: Mark-Compact 4030.6 (4140.4) -> 4018.6 (4144.7) MB, 2101.94 / 0.00 ms  (average mu = 0.298, current mu = 0.092) allocation failure; scavenge might not succeed\n\n\n<--- JS stacktrace --->\n\nFATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory\n----- Native stack trace -----\n\n"
[ERROR][2024-01-29 14:32:07] .../vim/lsp/rpc.lua:734	"rpc"	"/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"	"stderr"	" 1: 0xca5430 node::Abort() [node]\n"
[ERROR][2024-01-29 14:32:07] .../vim/lsp/rpc.lua:734	"rpc"	"/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"	"stderr"	" 2: 0xb7807d  [node]\n"
[ERROR][2024-01-29 14:32:07] .../vim/lsp/rpc.lua:734	"rpc"	"/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"	"stderr"	" 3: 0xeca0b0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]\n"
[ERROR][2024-01-29 14:32:07] .../vim/lsp/rpc.lua:734	"rpc"	"/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"	"stderr"	" 4: 0xeca397 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]\n"
[ERROR][2024-01-29 14:32:07] .../vim/lsp/rpc.lua:734	"rpc"	"/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"	"stderr"	" 5: 0x10dc0e5  [node]\n"
[ERROR][2024-01-29 14:32:07] .../vim/lsp/rpc.lua:734	"rpc"	"/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"	"stderr"	" 6: 0x10f3f68 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]\n"
[ERROR][2024-01-29 14:32:07] .../vim/lsp/rpc.lua:734	"rpc"	"/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"	"stderr"	" 7: 0x10ca081 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]\n"
[ERROR][2024-01-29 14:32:07] .../vim/lsp/rpc.lua:734	"rpc"	"/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"	"stderr"	" 8: 0x10cb215 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]\n"
[ERROR][2024-01-29 14:32:07] .../vim/lsp/rpc.lua:734	"rpc"	"/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"	"stderr"	" 9: 0x10a8866 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]\n"
[ERROR][2024-01-29 14:32:07] .../vim/lsp/rpc.lua:734	"rpc"	"/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"	"stderr"	"10: 0x15035f6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]\n11: 0x7ff675e99ef6 \n"

I tried with node-v18.16.0-linux-x64 and node-v20.11.0-linux-x64.

System:
Ubuntu 22.04 64bit
neovim 0.9.5
node 18.16.0 LTS || node 20.11.0 LTS

Is there a workaround or fix available? Or am I doing something wrong? Let me know if I can provide more information.

[erictraut]

Could you post the language server logs? I'm not sure how to access these in neovim. In VS Code, you would go to the "Output" tab and choose "pyright".

Adding "verboseOutput": true to the pyrightconfig.json or pyproject.toml will also help here, as it will provide additional details about heap usage.

An out-of-memory condition like this is typically because you're configuring pyright to find and type check every Python source file on your volume — perhaps by specifying a root directory of / or having a symbolic link somewhere in your project that redirects to /. The log output will typically provide clues about what's happening.

[gilbertfrancois]

Hi @erictraut, I think I found the cause of the problem. I guess it is caused by my workflow, because when I added the pyrightconfig.json file in the source folder, it solved the problem. The way my repository is organized is as follows:

research/
├── projects/
│   ├── experiment_1/
│   ├── experiment_2/
│   ├── experiment_3/
:   :
│   ├── experiment_n-1/
│   └── experiment_n/
└── venvs/
    ├── mxnet_cuda11/
    ├── torch1_cuda11/
    ├── torch2_cuda12/
    :
    ├── venv_m-1/
    └── venv_m/

The pypoetry.toml files are located in the research/venvs/some_venv folder and the sources are in research/projects/experiment_i. When I open neovim in one of the experiments folders, I guess pyenv cannot find pyrightconfig.json or pyproject.toml or ??? file by going up in the folder tree. Therefore it starts to 'index' all that is on the disk, like you suggest.

When adding a pyrightconfig.json in the current source folder (e.g. research/projects/experiment1/pyrightconfig.json), pyright did not crash anymore. When moving the file to /home/ubuntu/pyrightconfig.json, pyright crashed again, giving the message below:

[ERROR][2024-01-29 20:43:22] ...lsp/handlers.lua:535    
'Enumeration of workspace source files is taking longer than 10 seconds.
This may be because:
* You have opened your home directory or entire hard drive as a workspace
* Your workspace contains a very large number of directories and files
* Your workspace contains a symlink to a directory with many files
* Your workspace is remote, and file enumeration is slow
To reduce this time, open a workspace directory with fewer files or add a pyrightconfig.json configuration 
file with an "exclude" section to exclude subdirectories from your workspace. For more details, refer to 
https://github.com/microsoft/pyright/blob/main/docs/configuration.md.'

Followed by the heap out of memory error that I posted before:

[START][2024-01-29 20:51:37] LSP logging initiated
[ERROR][2024-01-29 20:51:37] .../vim/lsp/rpc.lua:734    "rpc"   "/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"   "stderr"        "\n<--- Last few GCs --->\n\n[6666:0x7072cf0]    73551 ms: Mark-Compact 4027.8 (4140.2) -> 4014.9 (4143.2) MB, 1850.36 / 0.00 ms  (average mu = 0.160, current mu = 0.030) allocation failure; scavenge might not succeed\n[6666:0x7072cf0]    76055 ms: Mark-Compact 4030.7 (4143.2) -> 4017.9 (4146.2) MB, 2450.44 / 0.00 ms  (average mu = 0.086, current mu = 0.021) allocation failure; scavenge might not succeed\n\n\n<--- JS stacktrace --->\n\nFATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory\n----- Native stack trace -----\n\n"
[ERROR][2024-01-29 20:51:37] .../vim/lsp/rpc.lua:734    "rpc"   "/home/ubuntu/.local/share/nvim/mason/bin/pyright-langserver"   "stderr"        " 1: 0xca5430 node::Abort() [node]\n 2: 0xb7807d  [node]\n 3: 0xeca0b0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]\n 4: 0xeca397 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]\n 5: 0x10dc0e5  [node]\n 6: 0x10f3f68 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]\n 7: 0x10ca081 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]\n 8: 0x10cb215 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]\n 9: 0x10a8866 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]\n10: 0x15035f6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]\n11: 0x7ff98bed9ef6 \n"

Thank you for the hint to the solution. I'll close the issue as resolved.

[gilbertfrancois]

To make the comment above complete, the pyrightconfig.json that I mentioned has the following content:

{
  "verboseOutput": true
}