[v4] Relational fields to large tables causes Strapi to crash (memory leak?)

[bhark]

Describe the bug

I've migrated an editorial site to Strapi. There are a little over 32.000 articles total.

All relational fields to these articles are unusable, and causes Strapi to consume an ungodly amount of system resources, hang for a few minutes and eventually crash.

As an example, I have a Single Type called Frontpage. The frontpage should contain some articles, manually picked out using a relational field. When the relational field is clicked, Strapi becomes unresponsive, the host system slows down and Strapi crashes.

In another case, I have a content type called Themes. The themes refer to articles in a many-to-many relationship. Trying to edit a theme (simply clicking it in the content manager) with more than ~900 articles leads to the same result.

Steps to reproduce the behavior

1. Have a collection type with 10.000+ items
2. Create a relational field to the collection type
3. Edit the relational field
4. Let the terror engulf you

Expected behavior

Strapi should be able to support large collections.

System

Node.js version: 14.17.5
NPM version: 6.14.14
Strapi version: 4.0.0
Database: Postgres 13.4
Operating system: Arch Linux

Error logs

After hanging for some time, the server coughs up a stacktrace and dies:

<--- Last few GCs --->

[6634:0x4fa1c20]   461850 ms: Mark-sweep (reduce) 2034.4 (2050.5) -> 2033.6 (2051.8) MB, 1504.0 / 0.0 ms  (average mu = 0.131, current mu = 0.003) allocation failure scavenge might not succeed
[6634:0x4fa1c20]   463354 ms: Mark-sweep (reduce) 2034.6 (2050.8) -> 2033.7 (2052.0) MB, 1499.2 / 0.0 ms  (average mu = 0.069, current mu = 0.003) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa25510 node::Abort() [/home/akhefale/.nvm/versions/node/v14.17.5/bin/node]
 2: 0x9664d3 node::FatalError(char const*, char const*) [/home/akhefale/.nvm/versions/node/v14.17.5/bin/node]
 3: 0xb9a8be v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/akhefale/.nvm/versions/node/v14.17.5/bin/node]
 4: 0xb9ac37 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/akhefale/.nvm/versions/node/v14.17.5/bin/node]
 5: 0xd56ca5  [/home/akhefale/.nvm/versions/node/v14.17.5/bin/node]
 6: 0xd5782f  [/home/akhefale/.nvm/versions/node/v14.17.5/bin/node]
 7: 0xd6566b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/akhefale/.nvm/versions/node/v14.17.5/bin/node]
 8: 0xd6922c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/akhefale/.nvm/versions/node/v14.17.5/bin/node]
 9: 0xd3790b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/home/akhefale/.nvm/versions/node/v14.17.5/bin/node]
10: 0x107fbef v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/home/akhefale/.nvm/versions/node/v14.17.5/bin/node]
11: 0x1426919  [/home/akhefale/.nvm/versions/node/v14.17.5/bin/node]

Analysis

• Strapi seems to query all articles in any case involving relational fields
• Perhaps a workaround can be found by overwriting all affected query controllers

[bhark]

This seems related to #2992 from 2019, still unresolved. They are not the same, though - this is Strapi core, not users-permissions. It seems to be a general problem plaguing Strapi and the ecosystem.

[strapi-bot]

This issue has been mentioned on Strapi Community Forum. There might be relevant details there:

https://forum.strapi.io/t/v4-relational-fields-to-large-collections-causes-strapi-to-crash-need-a-workaround/14791/1

[derrickmehaffy]

duplicate: #12004