[v4] - Strapi can't handle many records.

[rserafim]

Bug report

Describe the bug

Error accessing a record in admin.

Strapi can't handle working with too many records. I only have 1 million and he is crying. I still need to go up another 5 million.

This error only happens when I click on a record that is related to a city that has hundreds of companies.

If I click on a record, in which the city has dozens of companies, the error doesn't happen.

That's because it's running on localhost.

Steps to reproduce the behavior

1. Go to /content-manager/collection-types
2. Click on any register
3. See error

INFO REGISTER COMPANY ID = 1314737

total companies same city = 235611
total companies same category = 272

Expected behavior

Show the data like the example below.

INFO REGISTER COMPANY ID = 2
total companies same city = 1
total companies same category = 2

Screenshots

INFO REGISTER COMPANY ID = 1314737

Searching via Graphql the result comes correctly.

Code snippets

[2021-12-22 21:10:46.340] error: select distinct "t0".*, "t1"."company_id" from "categories" as "t0" left join "companies_category_links" as "t1" on "t0"."id" = "t1"."category_id" where ("t1"."company_id" in ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, ....)) - bind message has 39003 parameter formats but 0 parameters at Parser.parseErrorMessage (/Volumes/SSD Externo/_PROJETOS/XXX/XXX/api-portalfull-v2/node_modules/pg-protocol/dist/parser.js:287:98) at Parser.handlePacket (/Volumes/SSD Externo/_PROJETOS/XXX/XXX/api-portalfull-v2/node_modules/pg-protocol/dist/parser.js:126:29) at Parser.parse (/Volumes/SSD Externo/_PROJETOS/XXX/XXX/api-portalfull-v2/node_modules/pg-protocol/dist/parser.js:39:38) at Socket.<anonymous> (/Volumes/SSD Externo/_PROJETOS/XXX/XXX/api-portalfull-v2/node_modules/pg-protocol/dist/index.js:11:42) at Socket.emit (events.js:315:20) at Socket.EventEmitter.emit (domain.js:467:12) at addChunk (internal/streams/readable.js:309:12) at readableAddChunk (internal/streams/readable.js:284:9) at Socket.Readable.push (internal/streams/readable.js:223:10) at TCP.onStreamRead (internal/stream_base_commons.js:188:23)

And if it keeps trying to show the registration data (blank screen print), some time later this error will appear.
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

System

• Node.js version: v14.16.0
• YARN version: 1.22.17
• Strapi version: v4
• Database: DOCKER/POSTGRES
• Operating system: macos

Content-Type COMPANY

DATA INFOS

total companies = 1.229.134
total cities = 10.186
total categories = 2.351

[qant]

Imo you should try get less data, its seems not strapi error, its your error because of to many data in one query.
So try to avoid is some how, its your app, you need to think how to organize data, for example separate the queries or put some limits etc. I hope it helps

[rserafim]

Imo you should try get less data, its seems not strapi error, its your error because of to many data in one query. So try to avoid is some how, its your app, you need to think how to organize data, for example separate the queries or put some limits etc. I hope it helps

I'm not fetching lots of data. It is only 1 record with a relation of 1 city and 1 category. Just it!

Strapi's Dashboard should look under all companies that have the same city and category as the record in question.

[derrickmehaffy]

Can you please test with v4.0.2 (also in the future please specify your exact version not just v4)

We modified the logic in the list and edit views for population.

[rserafim]

Can you please test with v4.0.2 (also in the future please specify your exact version not just v4)

We modified the logic in the list and edit views for population.

I upgraded and rebuilt the dashboard. But nothing has changed, the error continues.

And note at the end of the code, it shows the total number of companies linked to the city.
5, $235606, $235607, $235608, $235609, $235610, **$235611**)) - bind message has 39003 parameter formats but 0 parameters at Parser.parseErrorMessage (/XXX/node_modules/pg-protocol/dist/parser.js:287:98) at Parser.handlePacket (/XXX/node_modules/pg-protocol/dist/parser.js:126:29) at Parser.parse (/XXX/node_modules/pg-protocol/dist/parser.js:39:38) at Socket.<anonymous> (/XXX/node_modules/pg-protocol/dist/index.js:11:42) at Socket.emit (node:events:390:28) at Socket.emit (node:domain:475:12) at addChunk (node:internal/streams/readable:315:12) at readableAddChunk (node:internal/streams/readable:289:9) at Socket.Readable.push (node:internal/streams/readable:228:10) at TCP.onStreamRead (node:internal/stream_base_commons:199:23)

[derrickmehaffy]

Even trying to remotely create the number of entities you have I can't reproduce this at all.

[rserafim]

derrickmehaffy

I created a new repository and a new database with fake data. I sent the invitation and you can reproduce the same error.
Thank's for your time.

Probably all records with the city "São Paulo" will present the mentioned error.

The same for the "Empresas Variadas" category

I believe all of these will work.

[derrickmehaffy]

You didn't link to the repo? 😕

nvm it's a private one, I see

[derrickmehaffy]

It doesn't appear your repo has the database though

[rserafim]

It doesn't appear your repo has the database though

is configured directly in database.js

[derrickmehaffy]

It doesn't appear your repo has the database though

is configured directly in database.js

I mean.... Using a PostgreSQL database over the internet is obviously not a good idea, the latency alone would trigger basically every warning in the tarn.js pooling system and probably just kill the connections.

Are you attempting to use a remote database also?

---

Edit: I'm grabbing a DB backup of this remote database so I can test locally. If you are trying to use this remotely that is almost certainly going to be a problem.

[rserafim]

It doesn't appear your repo has the database though

is configured directly in database.js

I mean.... Using a PostgreSQL database over the internet is obviously not a good idea, the latency alone would trigger basically every warning in the tarn.js pooling system and probably just kill the connections.

Are you attempting to use a remote database also?

Edit: I'm grabbing a DB backup of this remote database so I can test locally. If you are trying to use this remotely that is almost certainly going to be a problem.

No. I'm using localhost (Docker).

It's just for you to access and download.

[supryantowp]

same issue