Skip to content

Commit

Permalink
feat(gatsby-source-filesystem): allow adjusting `createRemoteFileNode…
Browse files Browse the repository at this point in the history
…` retry/timeout settings via env vars (#24535)

* (gatsby-source-filesystem) Remote File Node env variables

For developers with awful internet connection (like me), when using createRemoteFileNode, default timeouts of 30000ms are too short. Out of 300 files, it'd fail 1-4 images due to timeout and mess up the build process. 
Thus, it would be useful to overwrite default remote file download settings (retry limits and timeouts) with .env variables. 
For my specific case, just doubling all the values solved all issues.

* Update create-remote-file-node.js

removed ";"s

* updated readme

* update

Co-authored-by: Michal Piechowiak <misiek.piechowiak@gmail.com>
  • Loading branch information
webbugt and pieh committed Jul 16, 2020
1 parent e8ec13b commit c08fa4a
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 3 deletions.
8 changes: 8 additions & 0 deletions packages/gatsby-source-filesystem/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,3 +335,11 @@ function createMySqlNodes({ name, __sql, idField, keys }, results, ctx) {

module.exports = createMySqlNodes
```

## Troubleshooting

In case that due to spotty network, or slow connection, some remote files fail to download. Even after multiple retries and adjusting concurrent downloads, you can adjust timeout and retry settings with these environment variables:

- `STALL_RETRY_LIMIT`, default: `3`
- `STALL_TIMEOUT`, default: `30000`
- `CONNECTION_TIMEOUT`, default: `30000`
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,10 @@ let totalJobs = 0
* @param {Reporter} [options.reporter]
*/

const STALL_RETRY_LIMIT = 3
const STALL_TIMEOUT = 30000
const STALL_RETRY_LIMIT = process.env.GATSBY_STALL_RETRY_LIMIT || 3
const STALL_TIMEOUT = process.env.GATSBY_STALL_TIMEOUT || 30000

const CONNECTION_TIMEOUT = 30000
const CONNECTION_TIMEOUT = process.env.GATSBY_CONNECTION_TIMEOUT || 30000

/********************
* Queue Management *
Expand Down

0 comments on commit c08fa4a

Please sign in to comment.