Allow fetching all items with multiple calls instead of a very large one #199

philippeauriach · 2021-04-30T16:16:30Z

Especially usefull on servers limiting the duration of requests (for example Heroku, which limits to 30sec each request).

Passing the loop parameter will perform multiple calls until all items are downloaded.

Readme.md updated with an example.

strapi-cla · 2021-04-30T16:16:34Z

All committers have signed the CLA.

Especially usefull on servers limiting the duration of requests (for example Heroku, which limits to 30sec each request). Passing the loop parameter will perform multiple calls until all items are downloaded.

philippeauriach · 2021-07-11T15:06:56Z

Hi @tonoli, thanks for reviewing. Could you tell what's missing in this issue before it could be merged ?

remidej

Thanks for taking the time to work on this! It can be a pretty good feature for a future release. I've added my review, don't hesitate to comment if you have questions

remidej · 2021-07-27T09:17:01Z

README.md

+          api: { qs: { _sort: 'drop_date:DESC' } }, 
+          loop: 100,


The loop is an implementation detail of how we connect to the API. So I'd recommend moving the loop option inside of the api object

remidej · 2021-07-27T09:18:23Z

README.md


+#### Multiple requests instead of a single large one
+
+If you have many items in a collection and a single API call is a bit heavy for your server, you can split it into multiple requests.


Suggested change

If you have many items in a collection and a single API call is a bit heavy for your server, you can split it into multiple requests.

When you have many items in a collection, a single API call can be a bit heavy for your server. In that case, you can split your data fetching into multiple requests.

remidej · 2021-07-27T09:18:59Z

src/fetch.js

  const { apiURL, queryLimit, jwtToken, reporter } = ctx;

-  const { endpoint, api } = entityDefinition;
+  const { endpoint, api, loop } = entityDefinition;


Same as above, I recommended getting the loop from the api object

remidej · 2021-07-27T09:19:58Z

src/fetch.js

  );

  try {
+    if (typeof loop === 'number') {


Please also check if it's a integer (no decimals) and >= 1

remidej · 2021-07-27T09:21:48Z

src/fetch.js

+  reporter,
+  queryLimit,
+  requestOptions,
+  loop,
+  resultsAggreg = [],
+  _start = 0


Can you replace these arguments with an object? This way it will be easier to read when calling that function

remidej · 2021-07-27T09:22:43Z

src/fetch.js

+  reporter,
+  queryLimit,
+  requestOptions,
+  loop,


We can make the default value 1

remidej · 2021-07-27T09:25:02Z

src/fetch.js

+  const newResults = resultsAggreg.concat(newlyFetched);
+  if (newlyFetched.length < loop || newResults.length >= queryLimit) {
+    reporter.info(`Loop fetching ended with ${newResults.length} items - ${requestOptions.url}`);
+    //ended


Please make this comment more explicit

remidej · 2021-07-27T09:25:45Z

src/fetch.js

+    //ended
+    return newResults;
+  }
+  return fetchLoop(reporter, queryLimit, requestOptions, loop, newResults, _start + loop);


Please add a comment to make this recursion easier to read

remidej · 2021-07-27T09:27:08Z

src/index.js

  if (typeof type === 'object') {
    return {
      endpoint: type.endpoint || (single ? type.name : pluralize(type.name)),
+      loop: type.loop,


This shouldn't be needed if we move it to the API object

zsrl · 2021-08-08T13:46:27Z

When can this feature be released?
I think I need it.
@philippeauriach @remidej

philippeauriach · 2021-08-08T19:45:40Z

@remidej Changes made according to suggestions, thanks for your review. The only issue i didn't address is the part y ou ask for an explicit comment about the recursion, if you have an idea that would be nice ?
thanks

FredWahlqvistDrop · 2021-09-09T22:42:32Z

Hey @remidej, is there anything we can do to so this can be merged?

laurenskling · 2021-10-07T21:26:10Z

I would really like to have this. Currently I'm trying to load 90k models, but whatever I try, this this plugin wants to fetch them all at once. I've even tried to split request by adding multiple types in a loop with the same name with api: { qs: { _limit: limit, _start: limit * index } },, but that fetches all those queries at once, still making Strapi crash out of memory.

Grouviere · 2021-10-27T07:33:21Z

We're also needing this to be able to deploy on heroku. Is the only change blocking a missing comment ?

soupette · 2022-02-17T09:40:25Z

I am closing the PR as a new version has been released, feel free to reopen it.

laurenskling · 2022-02-17T09:44:12Z

Since v2 is only compatible for Strapi 4, this fix would still be highly appreciated for us using Strapi 3 (MongoDB say hello 👋)

laurenskling · 2022-03-10T20:46:05Z

Would it be possible for this to be merged in as 1.1 @soupette ? Really, this plugin cannot be run without this fix. Without this, how would one load any data over a couple of hundred entries?
Since you are planning to go v2 for Strapi v4. Please merge this in as 1.1 for us poor souls running Strapi v3.

laurenskling · 2022-06-18T10:05:53Z

I've been running this code for a while now using patch-package and it's been working great for me. On cold build.

I just realized that on incremental builds on Gatsby Cloud, the loop isn't running. My assumption here is that Gatsby Cloud incremental builds (which detect a Strapi webhook) run their installed version of the source plugin, not my patched one.

So still I would really really really love to see this feature released as a version 1.1 or something for us (still stuck at) running projects on Strapi 3. I don't see any downsides in doing so.

If this PR needs any work still, I would be super happy to work on it.

soupette · 2022-06-28T08:24:23Z

@laurenskling have tried to the new version with the Strapi V4? It does support Gatsby incremental build

laurenskling · 2022-06-28T11:26:00Z

@soupette yes I know the v4 version is updated, but i'm running 10+ codebases on Strapi 3, and for all of them I need to: switch from mongo to sql, change all database fields to snake_case, update all my code, update all my plugins code, update my frontend code to use the new GraphQL output. It's not something I can do overnight, it will take me months, many months, before I'm ready to fully switch to v4.
I am in the process, but until it's ready (probably need a full year for it), i would really really like it if we can get these little code snippets with big impact into v3. Thanks.

remidej · 2022-11-30T10:51:04Z

Closing this as it's targeting Strapi v3, which reaches end of life in a month, so we won't maintain the v3 version of this source plugin either

philippeauriach mentioned this pull request May 3, 2021

Paginate Strapi data requests #104

Closed

Allow fetching all items with multiple calls instead of a very large one

9c27d2f

Especially usefull on servers limiting the duration of requests (for example Heroku, which limits to 30sec each request). Passing the loop parameter will perform multiple calls until all items are downloaded.

philippeauriach force-pushed the fetch-loop branch from a9bc0d7 to 9c27d2f Compare June 1, 2021 11:30

tonoli previously approved these changes Jul 8, 2021

View reviewed changes

remidej suggested changes Jul 27, 2021

View reviewed changes

Move "loop" option inside "api" object

ddbd691

philippeauriach dismissed tonoli’s stale review via ddbd691 August 8, 2021 19:44

philippeauriach requested a review from remidej August 8, 2021 19:44

soupette closed this Feb 17, 2022

soupette reopened this Mar 11, 2022

alexandrebodin assigned remidej Jul 11, 2022

remidej closed this Nov 30, 2022


		#### Multiple requests instead of a single large one

		If you have many items in a collection and a single API call is a bit heavy for your server, you can split it into multiple requests.

	If you have many items in a collection and a single API call is a bit heavy for your server, you can split it into multiple requests.
	When you have many items in a collection, a single API call can be a bit heavy for your server. In that case, you can split your data fetching into multiple requests.

Allow fetching all items with multiple calls instead of a very large one #199

Allow fetching all items with multiple calls instead of a very large one #199

Uh oh!

Conversation

philippeauriach commented Apr 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

strapi-cla commented Apr 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philippeauriach commented Jul 11, 2021

Uh oh!

remidej left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zsrl commented Aug 8, 2021

Uh oh!

philippeauriach commented Aug 8, 2021

Uh oh!

FredWahlqvistDrop commented Sep 9, 2021

Uh oh!

laurenskling commented Oct 7, 2021

Uh oh!

Grouviere commented Oct 27, 2021

Uh oh!

soupette commented Feb 17, 2022

Uh oh!

laurenskling commented Feb 17, 2022

Uh oh!

laurenskling commented Mar 10, 2022

Uh oh!

laurenskling commented Jun 18, 2022

Uh oh!

soupette commented Jun 28, 2022

Uh oh!

laurenskling commented Jun 28, 2022

Uh oh!

remidej commented Nov 30, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

philippeauriach commented Apr 30, 2021 •

edited

Loading

strapi-cla commented Apr 30, 2021 •

edited

Loading