image cache fails when path to Gatsby src changes

[aheissenberger]

Description

the build process will generate all images again if the path of the Gatsby src folder changes and the cache exists from a previous build

Steps to reproduce

npx gatsby new gatsby-cache-problem
cd gatsby-cache-problem
yarn upgrade --latest
GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true yarn build --write-to-file --log-pages

rename src folder (this simulates the situation in a code build ci where the source is checked out in a temp folder and the public and .cache folder are restored from the last run.

cd ..
mv gatsby-cache-problem gatsby-cache-problem-changed
GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true yarn build --write-to-file --log-pages

Expected result

$ gatsby build --write-to-file --log-pages
success open and validate gatsby-configs - 0.039s
success load plugins - 0.517s
success onPreInit - 0.017s
success initialize cache - 0.021s
success copy gatsby files - 0.120s
success onPreBootstrap - 0.010s
success createSchemaCustomization - 0.053s
success source and transform nodes - 0.106s
success building schema - 0.422s
success createPages - 0.004s
success createPagesStatefully - 0.098s
success onPreExtractQueries - 0.003s
success update schema - 0.042s
success extract queries from components - 0.321s
success write out requires - 0.015s
success write out redirect data - 0.004s
success Build manifest and related icons - 0.259s
success onPostBootstrap - 0.279s
⠀
info bootstrap finished - 5.179s
⠀
success Building production JavaScript and CSS bundles - 5.296s
success run queries - 5.443s - 4/4 0.73/s
success Building static HTML for pages - 0.708s - 0/0 0.00/s
success Delete previous page data - 0.003s
success onPostBuild - 0.010s
info Done building in 11.469421385 sec

Actual result

Images are regenerated success Generating image thumbnails - 8.606s - 6/6 0.70/s

$ gatsby build --write-to-file --log-pages
success open and validate gatsby-configs - 0.053s
success load plugins - 0.647s
success onPreInit - 0.028s
success initialize cache - 0.013s
success copy gatsby files - 0.128s
success onPreBootstrap - 0.011s
success createSchemaCustomization - 0.008s
success source and transform nodes - 0.081s
success building schema - 0.344s
success createPages - 0.002s
success createPagesStatefully - 0.085s
success onPreExtractQueries - 0.002s
success update schema - 0.031s
success extract queries from components - 0.331s
success write out requires - 0.006s
success write out redirect data - 0.003s
success Build manifest and related icons - 0.184s
success onPostBootstrap - 0.190s
⠀
info bootstrap finished - 4.864s
⠀
success Building production JavaScript and CSS bundles - 6.661s
success run queries - 7.095s - 6/6 0.85/s
success Building static HTML for pages - 1.457s - 0/0 0.00/s
success Delete previous page data - 0.003s
success Generating image thumbnails - 8.606s - 6/6 0.70/s
success onPostBuild - 0.011s
info Done building in 13.532112445 sec

Environment

System:
OS: macOS Mojave 10.14.6
CPU: (4) x64 Intel(R) Core(TM) i7-6567U CPU @ 3.30GHz
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 14.3.0 - /var/folders/c3/xp8j5jx90s37thp58v2_76t80000gr/T/yarn--1590669969769-0.11111835422606897/node
Yarn: 1.22.4 - /var/folders/c3/xp8j5jx90s37thp58v2_76t80000gr/T/yarn--1590669969769-0.11111835422606897/yarn
npm: 6.14.4 - /usr/local/bin/npm
Languages:
Python: 2.7.16 - /usr/bin/python
Browsers:
Chrome: 83.0.4103.61
Firefox: 75.0
Safari: 13.1
npmPackages:
gatsby: ^2.22.11 => 2.22.11
gatsby-image: ^2.4.5 => 2.4.5
gatsby-plugin-manifest: ^2.4.9 => 2.4.9
gatsby-plugin-offline: ^3.2.7 => 3.2.7
gatsby-plugin-react-helmet: ^3.3.2 => 3.3.2
gatsby-plugin-sharp: ^2.6.9 => 2.6.9
gatsby-source-filesystem: ^2.3.8 => 2.3.8
gatsby-transformer-sharp: ^2.5.3 => 2.5.3

[aheissenberger]

Problem:
Gatsby creates a Hash for each queue job based on the content of the job.
The internalJob.outputDir and the internalJob.plugin.resolve contain absolute paths.
If the absolute path to the Gatsby source directory changes the internalJob.contentDigest does no longer match and the jobs will rerun.

Here is a long blog post about my findings:
https://www.heissenberger.at/en/blog/gatsby-conditional-build-broken-debugging/

Solutions:
A) QUICKFIX: copy the source code to a stable folder path (e.g. /tmp/gatsby) before running gatsby build
B) change the attributes of the createContentDigest to either remove the absolute path or generally avoid to create any kind of internal content with absolute path (e.g. internalJob.plugin.resolve)

[LpmRaven]

This is still an issue - my build times are 2 hours because the cache is invalidated. I have a 50GB website, so a tmp folder is a workaround (that will become worse as my site grows) but not a fix, unfortunately.

[rogermparent]

We have also run into this issue on dvc.org. On trying to diagnose it roughly a year ago we got the same symptoms from this issue's OP, but on a more recent attempt we had a completely new symptom.

Basically, in our normal Heroku builds any time the image cache is reused, we get this error:

       failed Building static HTML for pages - 1.279s
error Building static HTML failed for path "/blog"
       
         75 |     return context[query].data
         76 |   } else {
       > 77 |     throw new Error(
            |           ^
         78 |       `The result of this StaticQuery could not be fetched.\n\n` +
         79 |         `This is likely a bug in Gatsby and if refreshing the page does not fix it, ` +
         80 |         `please open an issue in https://github.com/gatsbyjs/gatsby/issues`
       
       
         WebpackError: The result of this StaticQuery could not be fetched.
         
         - gatsby-browser-entry.js:77 
           website/.cache/gatsby-browser-entry.js:77:11
         
         - siteMeta.ts:13 
           website/src/queries/siteMeta.ts:13:21
         
         - index.tsx:18 
           website/src/components/Page/DefaultSEO/index.tsx:18:31
         
         - defineProperty.js:11 
           [website]/[@babel]/runtime/helpers/defineProperty.js:11:1
         
         - defineProperty.js:14 
           [website]/[@babel]/runtime/helpers/defineProperty.js:14:1
         
         - extends.js:7 
           [website]/[@babel]/runtime/helpers/extends.js:7:1
         
         - static-entry.js:299 
           website/.cache/static-entry.js:299:22
         
         - utils.js:14 
           [website]/[@gatsbyjs]/reach-router/lib/utils.js:14:1
         
       
       not finished Caching JavaScript and CSS webpack compilation - 19.569s
       not finished Caching HTML renderer compilation - 1.356s
error Command failed with exit code 1.
       info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: The first Gatsby build attempt failed, and DEPLOY_OPTIONS does not include "retry"
    at main (/tmp/build_718f3f5e/scripts/deploy-with-s3.js:117:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
error Command failed with exit code 1.
       info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

It took a while for us to diagnose that the cause was this Issue.

Going from absolute to relative paths in cache hashing seems like it would be a minor change that would improve portability and compatability of Gatsby's OSS offering with other platforms. We've managed to work around it by implementing the quick fix described in the previous comment, but it's obviously less than ideal.

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions]

Hey again!

It’s been 60 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.
Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to comment on this issue or create a new one if you need anything else.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜