FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

[mjtechguy]

When adding smaller repos, the system seems to work fine. I am able to ask questions about those github repos.

I am working on some projects that require MinIO Object Storage integration and I wanted to ask questions of their docs.

When I load this repo https://github.com/minio/docs from main, it waits a bit then throws a Java Heap error.

Any ideas? Should I switch to Supabase instead of PGVector?

docker compose logs -f output:

dialoqbase           | Cloning into './uploads/minio-docs-main'...
dialoqbase           |
dialoqbase           | <--- Last few GCs --->
dialoqbase           |
dialoqbase           | [81:0x7222e80]   226205 ms: Mark-sweep 2002.8 (2090.1) -> 1990.7 (2091.9) MB, 99.6 / 0.0 ms  (average mu = 0.853, current mu = 0.568) allocation failure; scavenge might not succeed
dialoqbase           | [81:0x7222e80]   226456 ms: Mark-sweep 2008.3 (2093.9) -> 1998.0 (2100.1) MB, 217.0 / 0.0 ms  (average mu = 0.638, current mu = 0.133) allocation failure; scavenge might not succeed
dialoqbase           |
dialoqbase           |
dialoqbase           | <--- JS stacktrace --->
dialoqbase           |
dialoqbase           | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
dialoqbase           |  1: 0xb83f50 node::Abort() [/usr/local/bin/node]
dialoqbase           |  2: 0xa94834  [/usr/local/bin/node]
dialoqbase           |  3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
dialoqbase           |  4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
dialoqbase           |  5: 0xf42265  [/usr/local/bin/node]
dialoqbase           |  6: 0xf5474d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
dialoqbase           |  7: 0xf2ee4e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
dialoqbase           |  8: 0xf30217 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
dialoqbase           |  9: 0xf10760 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/usr/local/bin/node]
dialoqbase           | 10: 0xf081d4 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [/usr/local/bin/node]
dialoqbase           | 11: 0xf0a568 v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString(int, v8::internal::AllocationType) [/usr/local/bin/node]
dialoqbase           | 12: 0x11e7bb5 v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::ConsString>, v8::internal::AllocationType) [/usr/local/bin/node]
dialoqbase           | 13: 0x1051cd2 v8::internal::JsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node]
dialoqbase           | 14: 0x10528b1 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
dialoqbase           | 15: 0x105448f v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
dialoqbase           | 16: 0x105448f v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
dialoqbase           | 17: 0x1056cef v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
dialoqbase           | 18: 0x1057a3f v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
dialoqbase           | 19: 0xdea777 v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
dialoqbase           | 20: 0x17036b9  [/usr/local/bin/node]
dialoqbase           | Aborted (core dumped)
dialoqbase           | error Command failed with exit code 134.
dialoqbase           | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
dialoqbase exited with code 0
dialoqbase           | yarn run v1.22.19
dialoqbase           | $ npx prisma migrate deploy && fastify start  app.js
dialoqbase           | Prisma schema loaded from prisma/schema.prisma
dialoqbase           | Datasource "db": PostgreSQL database "dialoqbase", schema "public" at "dialoqbase-pg:5432"
dialoqbase           |
dialoqbase           | 9 migrations found in prisma/migrations
dialoqbase           |
dialoqbase           |
dialoqbase           | No pending migrations to apply.
dialoqbase           | [info] use ffmpeg.wasm v0.12.0
dialoqbase           | Connecting pm2...
dialoqbase           | Connecting to database...

[mjtechguy]

Also, it will not let me delete the datasource now.

[n4ze3m]

hey, it's 'heap out of memory.' There is some memory usage limit on the V8 engine, I guess :/ How much memory do you have?

Can you add this to .env and restart the server: NODE_OPTIONS=--max_old_space_size=4096 ? It may fix the issue.

https://stackoverflow.com/questions/38558989/node-js-heap-out-of-memory

[mjtechguy]

I am on a VM with 16GB of RAM. Thanks for the recommendation. I will try 4096 and 8192 and so on.

Will report back.

[mjtechguy]

That worked. I increased it to 8192 and I saw the system memory usage reach 6GB. But it was able to process it and I am now able to ask questions against it.

Thank you!