Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory #8072

Open
2 of 4 tasks
bhavesh2510 opened this issue May 10, 2024 · 22 comments
Open
2 of 4 tasks
Assignees

Comments

@bhavesh2510
Copy link

bhavesh2510 commented May 10, 2024

Verify latest release

  • I verified that the issue exists in the latest pnpm release

pnpm version

9.1.0

Which area(s) of pnpm are affected? (leave empty if unsure)

No response

Link to the code that reproduces this issue or a replay of the bug

Uploading image.png…

Reproduction steps

i'm using NX workspace and using pnpm to install dependencies
Build fails on pipeline
just FYI --max-size=3017 is not working, tried everything but couldn't make it work
However on local machine it works fine but on bitbucket pipeline it breaks

"build:manager": "nx run manager-dashboard:build --prod --skip-nx-cache --configuration=prod",

Here is the step

  • step:
    name: Trigger pnpm build test
    size: 2x
    image: node:18.16.0
    script:
    - corepack enable
    - corepack prepare pnpm@latest-9 --activate
    - pnpm install
    - pnpm build:manager
    caches:
    - pnpm

Describe the Bug

<--- Last few GCs --->
[328:0x6242d20] 12010 ms: Mark-sweep (reduce) 529.7 (541.8) -> 528.7 (540.8) MB, 301.9 / 0.0 ms (average mu = 0.108, current mu = 0.025) allocation failure; scavenge might not succeed
[328:0x6242d20] 12017 ms: Scavenge (reduce) 530.2 (541.0) -> 529.6 (541.0) MB, 1.4 / 0.0 ms (average mu = 0.108, current mu = 0.025) allocation failure;
[328:0x6242d20] 12024 ms: Scavenge (reduce) 530.2 (541.0) -> 529.9 (541.5) MB, 4.3 / 0.0 ms (average mu = 0.108, current mu = 0.025) allocation failure;
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xb7a940 node::Abort() [/usr/local/bin/node]
2: 0xa8e823 [/usr/local/bin/node]
3: 0xd5c940 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
4: 0xd5cce7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
5: 0xf3a3e5 [/usr/local/bin/node]
6: 0xf3b2e8 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/local/bin/node]
7: 0xf4b7f3 [/usr/local/bin/node]
8: 0xf4c668 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
9: 0xf26fce v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0xf28397 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
11: 0xf0956a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/bin/node]
12: 0x12ce7af v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
13: 0x16fb6b9 [/usr/local/bin/node]

Expected Behavior

dont know what am i missing here

Which Node.js version are you using?

18.16.0

Which operating systems have you used?

  • macOS
  • Windows
  • Linux

If your OS is a Linux based, which one it is? (Include the version if relevant)

No response

@zkochan
Copy link
Member

zkochan commented May 12, 2024

Should be fixed by #8060

@zkochan
Copy link
Member

zkochan commented May 12, 2024

🚢 9.1.1

@bhavesh2510
Copy link
Author

@zkochan this issue still persists even after using pnpm version 9.1.1

@zkochan
Copy link
Member

zkochan commented May 13, 2024

How can I reproduce this? Is the project open source?

@bhavesh2510
Copy link
Author

@zkochan it is not opensource, im using nx monorepo with node version 18.16.0 & pnpm version 9.1.1
i will try to come with some sample project if possible

@sammarks
Copy link

sammarks commented May 13, 2024

FWIW I am unfortunately not able to provide the source for my example either, but I have been doing a bit of debugging on my end.

On my machine, the process doesn't get to the point of running out of memory, but does peg the CPU.

In my case, inside resolvePeersOfChildren, calculateDepPaths has a length of 92 for one specific package (I'm guessing the package itself is irrelevant as it depends on the packages that depend on it), which seems a lot longer than the rest. That calls analyzeGraph from graph-cycles with 156 nodes, and ~a quarter of those nodes having ~20-30 dependencies. The call to analyzeGraph does eventually finish without running out of memory, but takes several minutes to complete.

I verified this by installing v1.2.1 of graph-cycles separately and passing it the same payload.

Not sure if there's a way to reduce the size of the graph passed to graph-cycles, or if there's a way to improve the performance of graph-cycles.

Our environment is a large monorepo with ~200 interconnected packages, with a lot of them referencing the others as peer dependencies (version specifier "<1.0.0" for all of the interrelated peers). The installation inside the main monorepo succeeds without any slowness, but running an installation inside another much smaller monorepo that depends on a lot of the packages from the larger monorepo is the one that stalls. The packages from the main monorepo are being downloaded from a registry instead of any local workspace linking.

So I would imagine this might be reproducible by creating a smaller monorepo (5 packages) that depend on other packages with a lot of peer dependencies to drive the size of that graph up.

@smart
Copy link

smart commented May 13, 2024

Chiming in to say we're experiencing this using github actions on a turborepo monorepo building a sveltekit app on pnpm 9.1.1.

✓ 1789 modules transformed.
rendering chunks...
<--- Last few GCs --->
[5527:0x661a640] 68118 ms: Mark-Compact (reduce) 2035.4 (2076.3) -> 2035.3 (2076.8) MB, 1244.63 / 0.02 ms (+ 84.4 ms in 7 steps since start of marking, biggest step 26.4 ms, walltime since start of marking 1353 ms) (average mu = 0.300, current mu = 0.[5527:0x661a640] 69962 ms: Mark-Compact (reduce) 2041.8 (2082.5) -> 2040.6 (2082.4) MB, 1834.44 / 0.00 ms (average mu = 0.153, current mu = 0.006) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
1: 0xb84bd6 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
2: 0xefec30 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
3: 0xefef17 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
4: 0x1110925 [node]
5: 0x11287a8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
6: 0x10fe8c1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
7: 0x10ffa55 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0x10dd0a6 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
9: 0x1537e36 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
10: 0x7fb92ce99ef6
Aborted (core dumped)
 ELIFECYCLE  Command failed with exit code 134.

@smart
Copy link

smart commented May 14, 2024

Just a follow up to this as I continue to debug. Here is my github actions file. I tried not building through turbo and going directly to build within the app directory.

jobs:
build:
name: Build and Test
timeout-minutes: 15
runs-on: ubuntu-latest

steps:
  - name: Check out code
    uses: actions/checkout@v4
    with:
      fetch-depth: 2

  - uses: pnpm/action-setup@v4
    with:
      version: 9.1.1

  - name: Setup Node.js environment
    uses: actions/setup-node@v4
    with:
      node-version: 20
      cache: 'pnpm'

  - name: Install dependencies
    run: pnpm install --no-frozen-lockfile

  - name: Install playwright browsers
    run: pnpx playwright install --with-deps

  - name: Build
    working-directory: ./apps/web
    run: pnpm build 

  - name: Test
    working-directory: ./apps/web
    run: pnpm test

Still resulted in:
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
1: 0xb84bd6 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
2: 0xefec30 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
3: 0xefef17 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
4: 0x1110925 [node]
5: 0x11287a8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
6: 0x10fe8c1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
7: 0x10ffa55 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0x10dd0a6 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
9: 0x1537fe1 v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
10: 0x7fe059e99ef6
Aborted (core dumped)
 ELIFECYCLE  Command failed with exit code 134.

@zkochan
Copy link
Member

zkochan commented May 15, 2024

I have reduced memory consumption in this PR: #8084

I don't know if it will be enough for you all. Maybe you could try to build pnpm from that branch and see if it helps.

zkochan added a commit that referenced this issue May 16, 2024
@zkochan
Copy link
Member

zkochan commented May 20, 2024

Try 9.1.2

@bhavesh2510
Copy link
Author

bhavesh2510 commented May 21, 2024

@zkochan i'm still facing this issue, happens on bitbucket pipeline
sadly i can't provide sample code for this

- step: name: Trigger pnpm build test size: 2x image: node:18.16.0 script: - corepack enable - corepack prepare pnpm@latest - pnpm install - npm run build:manager caches: - pnpm

> nx run manager-dashboard:build:prod vite v4.3.9 building for production... transforming... replace "apps/manager-dashboard/src/environments/environment.ts" with "apps/manager-dashboard/src/environments/environment.prod.ts" replace "apps/manager-dashboard/src/environments/environment.ts" with "apps/manager-dashboard/src/environments/environment.prod.ts" replace "apps/manager-dashboard/src/environments/environment.ts" with "apps/manager-dashboard/src/environments/environment.prod.ts" replace "apps/manager-dashboard/src/environments/environment.ts" with "apps/manager-dashboard/src/environments/environment.prod.ts" replace "apps/manager-dashboard/src/environments/environment.ts" with "apps/manager-dashboard/src/environments/environment.prod.ts" replace "apps/manager-dashboard/src/environments/environment.ts" with "apps/manager-dashboard/src/environments/environment.prod.ts" replace "apps/manager-dashboard/src/environments/environment.ts" with "apps/manager-dashboard/src/environments/environment.prod.ts" node_modules/.pnpm/pdfjs-dist@3.11.174/node_modules/pdfjs-dist/build/pdf.js (1982:23) Use of eval in "node_modules/.pnpm/pdfjs-dist@3.11.174/node_modules/pdfjs-dist/build/pdf.js" is strongly discouraged as it poses security risks and may cause issues with minification. ✓ 7428 modules transformed. rendering chunks... <--- Last few GCs ---> [266:0x639cd20] 156539 ms: Mark-sweep 1942.0 (2101.7) -> 1917.0 (2094.2) MB, 1895.3 / 0.0 ms (average mu = 0.735, current mu = 0.547) allocation failure; scavenge might not succeed [266:0x639cd20] 158585 ms: Mark-sweep 1951.1 (2128.3) -> 1951.1 (2122.8) MB, 1974.5 / 0.0 ms (average mu = 0.552, current mu = 0.035) allocation failure; scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory 1: 0xb7a940 node::Abort() [/usr/local/bin/node] 2: 0xa8e823 [/usr/local/bin/node] 3: 0xd5c940 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node] 4: 0xd5cce7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node] 5: 0xf3a3e5 [/usr/local/bin/node] 6: 0xf4c8cd v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node] 7: 0xf26fce v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node] 8: 0xf28397 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node] 9: 0xf088e0 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/usr/local/bin/node] 10: 0xf00354 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [/usr/local/bin/node] 11: 0xf02608 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [/usr/local/bin/node] 12: 0x1042959 v8::internal::JsonParser<unsigned short>::MakeString(v8::internal::JsonString const&, v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node] 13: 0x1044676 v8::internal::JsonParser<unsigned short>::ParseJsonValue() [/usr/local/bin/node] 14: 0x104515f v8::internal::JsonParser<unsigned short>::ParseJson() [/usr/local/bin/node] 15: 0xde2523 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node] 16: 0x16fb7b9 [/usr/local/bin/node]

@zkochan
Copy link
Member

zkochan commented May 21, 2024

I believe this should do it: #8106

@zkochan
Copy link
Member

zkochan commented May 22, 2024

Did one more memory optimization: #8109

I believe most of the issues will be solved in the next release.

@zkochan
Copy link
Member

zkochan commented May 27, 2024

Try 9.1.3

@bhavesh2510
Copy link
Author

@zkochan still same :(

@zkochan
Copy link
Member

zkochan commented May 29, 2024

I think I have found the core issue. I will try to fix it.

@zkochan
Copy link
Member

zkochan commented May 29, 2024

As a temporary workaround, you should be able to add these settings to your .npmrc in the root of your project:

dedupe-peer-dependents=false
auto-install-peers=false

zkochan added a commit that referenced this issue May 30, 2024
Reverts 4b65113

ref #8072

This change makes install faster at least 3 times in some Bit workspaces.
@zkochan
Copy link
Member

zkochan commented May 30, 2024

Try v9.1.4

@smart
Copy link

smart commented May 30, 2024

Just tried it with 9.1.4 still getting this

<--- Last few GCs --->
[4770:0x75e72e0] 74721 ms: Scavenge (reduce) 2036.0 (2079.8) -> 2035.9 (2080.8) MB, 14.05 / 0.00 ms (average mu = 0.300, current mu = 0.275) allocation failure;
[4770:0x75e72e0] 76464 ms: Mark-Compact (reduce) 2037.8 (2082.1) -> 2037.2 (2082.2) MB, 1346.10 / 0.00 ms (average mu = 0.274, current mu = 0.248) allocation failure; GC in old space requested
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
1: 0xb82c28 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
2: 0xeed540 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
3: 0xeed827 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
4: 0x10ff3c5 [node]
5: 0x1117248 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
6: 0x10ed361 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
7: 0x10ee4f5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0x10cbb46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
9: 0x1527b21 v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
10: 0x7f8d34e99ef6
Aborted (core dumped)
 ELIFECYCLE  Command failed with exit code 134.

@zkochan
Copy link
Member

zkochan commented May 30, 2024

The projects that reproduced the issue for me work now. If you have sharable examples that reproduce the issue, please attach them.

@sammarks
Copy link

sammarks commented May 30, 2024

Just got the opportunity to check it again in my environment; in my case the fix in 9.1.3 appears to have fixed it.

However, now I get a new error after the install is complete and dependencies are built:

I get this same error with 9.1.4, so ignore the 9.1.3 in the path in the error report. I've upgraded to 9.1.4 and tried again to find the same issue.

pnpm: Invalid string length
    at writeBlockMapping (/Users/sammarks/Library/pnpm/global/5/.pnpm/pnpm@9.1.3/node_modules/pnpm/dist/pnpm.cjs:16817:20)
    at writeNode (/Users/sammarks/Library/pnpm/global/5/.pnpm/pnpm@9.1.3/node_modules/pnpm/dist/pnpm.cjs:16883:13)
    at writeBlockMapping (/Users/sammarks/Library/pnpm/global/5/.pnpm/pnpm@9.1.3/node_modules/pnpm/dist/pnpm.cjs:16808:14)
    at writeNode (/Users/sammarks/Library/pnpm/global/5/.pnpm/pnpm@9.1.3/node_modules/pnpm/dist/pnpm.cjs:16883:13)
    at Object.dump (/Users/sammarks/Library/pnpm/global/5/.pnpm/pnpm@9.1.3/node_modules/pnpm/dist/pnpm.cjs:16977:11)
    at yamlStringify (/Users/sammarks/Library/pnpm/global/5/.pnpm/pnpm@9.1.3/node_modules/pnpm/dist/pnpm.cjs:114456:32)
    at writeLockfiles (/Users/sammarks/Library/pnpm/global/5/.pnpm/pnpm@9.1.3/node_modules/pnpm/dist/pnpm.cjs:114470:23)
    at async Promise.all (index 0)
    at async _installInContext (/Users/sammarks/Library/pnpm/global/5/.pnpm/pnpm@9.1.3/node_modules/pnpm/dist/pnpm.cjs:192699:9)
    at async installInContext (/Users/sammarks/Library/pnpm/global/5/.pnpm/pnpm@9.1.3/node_modules/pnpm/dist/pnpm.cjs:192812:16)

Modified that bit of code in the 9.1.4 transpiled JS to be:

} else {
  pairBuffer += ": ";
}
pairBuffer += state.dump;
try {
  _result += pairBuffer;
} catch (err) {
  console.error('would have run into a problem here')
  console.info(_result.length)
  console.info(pairBuffer.length)
  process.exit(1)
}

And that outputs:

would have run into a problem here
19313147
533455214

So looks like we're running into some very large strings here. Even if it were able to handle that, it seems like the lockfile would be a few hundred MB in size.

That was with:

auto-install-peers=false
dedupe-peer-dependents=true

Running it with:

auto-install-peers=false
dedupe-peer-dependents=false

results in the same error, but it seems to happen later in the process. This time here's the output I get:

would have run into a problem here
526644599
77564213

Happy to help debug some more if you tell me what variables you want to see, but unfortunately I'm not able to provide access to the specific repository where it's happening.

If it helps here's my full config:

link-workspace-packages=deep
strict-peer-dependencies=false

auto-install-peers=false
resolution-mode=lowest-direct
dedupe-peer-dependents=false
disallow-workspace-cycles=true
resolve-peers-from-workspace-root=false

I can also make a separate issue for this or participate in the existing thread to keep things organized; just let me know.

@zkochan
Copy link
Member

zkochan commented May 30, 2024

@sammarks put your observations to the existing issue about the same error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants