[gatsby-source-filesystem] Make concurrency of createRemoteFileNode configurable

[madeleineostoja]

Summary

This feature request would address one of the issues discovered in #6654, that the high concurrency (200) of the createRemoteFileNode action can overload some remote sources (WP servers, content platforms) and cause the source and transform nodes build step to hang and fail silently.

Basic example

Users should be able to configure the concurrency of that action by passing a new option to gatsby-source-filesystem, maybe simply concurrency: <Integer>, or a more user-friendly param like 'intensity' or 'load' or something.

The reasoning behind this configuration should also be documented, since it's a currently obscure gotcha of processing large repositories of content (particularly media) from certain remote servers.

[alexanderwe]

Just saw that there is an environment variable which can be set to control the concurrent downloads, does this maybe already solve the issue ? 90aa247#diff-1864dd21828754bdbc63f22b895bee8e

I tried it with my current project of having a Wordpress Site with other 500 images being in size of 4-6mb which previously suffered the issue described in #6654 and it seems to work for now.

[madeleineostoja]

Ha! So there is. Weird that it's an env var rather than config passed in gatsby-config like every other source, but that seems to address this issue.

Maintainers feel free to close. Still think it should be consistent with other Configs (and better documented), but 🤷‍♂️

[alexanderwe]

I am not 100% that this will address all of those issues that were raised with failing builds with large number of images/big image files but at least it seems to work for me right now.

But I am with you saying that a config would be more consistent here. But I also see the positive aspects of an environment variable to be set only when needed by the users.

[DSchau]

Glad to hear it's working nicely!

Re: environment variables, we're not 100% sold on that approach yet, either. Check out #14636 for something that we think is far improved.

Going to close -- please reply or re-open if we can help further! 💜