Longer response time in sending 1 million records using express.js

[sachinavinaw]

Hi there,

I have developed an application that rely heavily on database on following environment-

node v10.15.3
postgresql v10.5

200 concurrent applications access API to execute read all records from database. In response, 1 million record is returned by the database to node server and node server returns the same packet to client application. The response time is 6407.9 msec which is very high.

200 concurrent API call is made every second.
I also used compression module, but it did not help much.

Can anyone please let me know how to speed up the response time?

[achillesoo7]

@sachinavinaw I would like to point out something, that response is not getting delayed by the express it's waiting for your DB to complete the query and return the response. So the actual problem is with the query itself.

Several solutions:

Adding pagination would be a quick and efficient one.

Other then this, you can try using streams for PostgreSQL
for implementing streams in pg, take reference from the link below
https://www.npmjs.com/package/pg-query-stream

[wesleytodd]

I agree, sending one million of anything in a single request is something to avoid. Pagination is the best approach IMO. Either way this is not an express issue, if you would like more support with how to design your application I would post on Stack Overflow or the express gitter