Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug]: ☂️ umbrella issue for cache and over-eager persistence issues #11747

Closed
DSchau opened this issue Feb 13, 2019 · 36 comments
Closed

[bug]: ☂️ umbrella issue for cache and over-eager persistence issues #11747

DSchau opened this issue Feb 13, 2019 · 36 comments
Labels
type: bug

Comments

@DSchau
Copy link
Contributor

@DSchau DSchau commented Feb 13, 2019

Description

In this umbrella issue, I'd like to provide a means for others to share any issues encountered regarding caching, specifically in instances when the solution to a problem you were facing was remediated by removing the cache (e.g. rm -rf .cache).

Issues that we have seen before (but generally have not been able to reproduce) can usually be classified as one of the following:

  1. Remote content sourced by a plugin is available (e.g. in the database, CMS, etc.) but is not available after a local re-build
  2. A local plugin source code changed, but the change(s) do not seem to have taken effect
  3. Local content (e.g. images, Markdown, etc.) were changed, but the changes do not seem to show up after a build

If any of these scenarios, or more broadly speaking, if you've ran into a problem where content wasn't appearing appropriately and deleting the cache solved this problem--then please chime in!

Intent

If we can reliably reproduce these types of issues and scenarios, we can harden our cache and improve the experience for all Gatsby users. Specifically, if we can reliably reproduce we can fix the underlying issue and then author end-to-end tests to ensure that the caching issue remains fixed.

Template

Please use the following template to report an issue so that we can most effectively debug the underlying issue.

Note: it is extremely important that you provide as much information as possible for the reproduction step. The more info and clearer the reproduction, the better we will be able to debug and ship fixes to these issues!

## Description

<!-- What led you to delete the cache? What was the exact problem that deleting the cache seemed to have fixed? -->

## Reproduction

<!-- Please, please provide detailed steps to reproduce. Ideally a Github repository where we can reproduce the issue(s) ourselves, as well. -->

## Environment Info

<!-- Run `npx gatsby info --clipboard` in the root of your Gatsby app and paste the output here -->
@DSchau DSchau added type: bug status: needs reproduction no triage labels Feb 13, 2019
@mrispoli24
Copy link

@mrispoli24 mrispoli24 commented Feb 13, 2019

I think I ran into this issue and after much frustration I started this issue but ended up closing it after deleting the public folder and getting everything to work. I do believe this sounds a lot like what others are experiencing though.

@nickytonline
Copy link
Contributor

@nickytonline nickytonline commented Feb 13, 2019

I built my site off of the Netlify CMS starter, https://github.com/nickytonline/www.iamdeveloper.com. ‪It has deleting the .cache as part of their build steps. See https://github.com/netlify-templates/gatsby-starter-netlify-cms/blob/master/package.json#L40‬

Not sure why, but they must have a good reason to delete the .cache.

I remember when I started off with another starter site, I had to delete the .cache frequently, but this was pre-Gatsby 2.

@keithmifsud
Copy link

@keithmifsud keithmifsud commented Feb 14, 2019

Please bear in mind that I am not 100% sure this was the cause of the issue forcing me to delete the .cache directory but it did fix the issue as soon as I did. I have a json file with an array of objects. I decided to give each object an identifier key. Gatsby does not like this key name, maybe it conflicts with an internal key name. I renamed the identifier to name but the related changes were not being built until I deleted the .cache directory.

@endymion1818
Copy link
Contributor

@endymion1818 endymion1818 commented Feb 14, 2019

In working with several developers using GatsbyJS over the past 9 months I’ve only had this isssue come up once. A colleague was experiencing graphql errors because an API wasn’t being reached. We had recently changed how we fetch data (using gatsby-source-Wordpress) in Gatsby-config.js. When he updated his master branch, running the cli command “gatsby develop” still used the old settings. Deleting .cache solved the issue.

@oobidin
Copy link

@oobidin oobidin commented Feb 14, 2019

I've been working with GatsbyJS last 6 months and I faced some issues when you update some local content (MDs and JSONs in our case) and it doesn't update. Actually, we've created the yarn clean command with the content: rm -rf public && rm -rf .cache to solve such kind of issues.

@trotzig
Copy link
Contributor

@trotzig trotzig commented Feb 14, 2019

We updated our content types in contentful (removed some types, added some). Removing the cache folder made the dev server start up again. Cc @axhamre.

@kylefraser
Copy link

@kylefraser kylefraser commented Feb 14, 2019

We've run into an issue with contentful where, when we've added images via a rich-text, we've had to delete the .cache folder in order to query additional properties with GraphQL on the embedded asset.

For instance, we've added images, re-started the dev server, and were still unable to query image properties like

target {
   fields {
      file {
         url
      }
   }
}

deleting the .cache folder, solved this for us.

@vacas5
Copy link
Contributor

@vacas5 vacas5 commented Feb 14, 2019

I abstracted a lot of functions utilized inside my sourceNodes function into a separate file. Changes made in that file were not detected by cache busting, and so any time I changed the code in my abstracted file, I would have to clear the cache manually. Any way that Gatsby could detect changes in required files within gatsby-node.js would be advantageous to my use case.

@amankkg
Copy link

@amankkg amankkg commented Feb 17, 2019

Not sure if it counts...

Recently, I had to press Clear cache and retry deploy button in Netlify's CI during this PR - that checks turned green only after cache clean.

It started to fail with adding of prismjs and its theme CSS import in gatsby-browser.js.

@u12206050
Copy link
Contributor

@u12206050 u12206050 commented Feb 21, 2019

Though it is undocumented it seems one can use Gatsby's cache object to set and get json objects. Data is stored in the .cache/cache/{plugin-name}/ directory.

In your plugin gatsby-node.js in one of the many apis include cache as an argument; eg.

exports.onPostBuild = async function(
  { cache, store, graphql },
  { some, plugin, options} ) {
    const cacheKey = 'some-key-name';
    ...
    let obj = await cache.get(cacheKey);

    if (!obj) {
      obj = { created: Date.now() };
    }
    ...
    obj.modified = Date.now();

    await cache.set(cacheKey, obj);
  }

@DSchau
Copy link
Contributor Author

@DSchau DSchau commented Feb 21, 2019

@u12206050 correct! A bit out-of-scope for this discussion though, right?

We'd love if you were able to rectify the "not documented" piece and document that API :)

@danoc
Copy link
Contributor

@danoc danoc commented Feb 22, 2019

Hi @DSchau! Thanks for making this.

Here's one that me and my team have been struggling with lately: #11109

I spent some time debugging but couldn't figure it out.

@wardpeet wardpeet changed the title [bug]: umbrella issue for cache and over-eager persistence issues [bug]: ☂️ umbrella issue for cache and over-eager persistence issues Feb 22, 2019
@nishantdania nishantdania unpinned this issue Feb 24, 2019
@nishantdania nishantdania pinned this issue Feb 24, 2019
@kbrgl kbrgl unpinned this issue Feb 26, 2019
@kbrgl
Copy link
Contributor

@kbrgl kbrgl commented Feb 26, 2019

Oops, unpinned this accidentally. Sorry!

(Perhaps GitHub should make repository actions more obvious 😛)

@kbrgl kbrgl pinned this issue Feb 26, 2019
@aarondack
Copy link

@aarondack aarondack commented Feb 26, 2019

This is a pretty systemic problem whenever adding a new image. Pretty much exclusively I have to rm -rf .cache/ and re-install. Sometimes I also just nuke the public/ folder. I know this is sort of already known, just figured that I'd share that I have had this experience.

@JonnyHayes
Copy link

@JonnyHayes JonnyHayes commented Apr 5, 2019

This is happening to us a lot now. Message keeps changing -

Error: Invariant Violation: Encountered an error trying to infer a GraphQL type for: block > teaser___NODE.
Error: Invariant Violation: Encountered an error trying to infer a GraphQL type for: navigation___NODE.
Error: Invariant Violation: Encountered an error trying to infer a GraphQL type for: body___NODE.

We are also using contentful.

@lannonbr lannonbr added not stale and removed stale? labels Apr 11, 2019
@smurrayatwork smurrayatwork unpinned this issue Apr 11, 2019
@LekoArts LekoArts pinned this issue Apr 17, 2019
@mrispoli24
Copy link

@mrispoli24 mrispoli24 commented May 11, 2019

Running into this issue again myself now, deleting the public folder and .cache folders don't seem to do the trick anymore. The images are there, I am willing to invite someone to the repository to check it out if you like but it seems that when working with static image files this happens a lot. They are present in the gh-pages branch but showing up broken on the deployed site.

@mrispoli24
Copy link

@mrispoli24 mrispoli24 commented May 15, 2019

I figured out what was going on in the above case. I have a series of image files in a /static directory that are divided by folders. In my template I had the wrong case for a character so /static/Folder instead of /static/folder. In development and on local building this ran just fine, upon deployment to github this rightly broke. Not sure exactly how gatsby builds things under the hood but it would have been a faster catch had it broken in development mode locally. This might not have had anything to do with the cache but I think it's worth letting ya'll know in case this is a symptom or indicator of something here.

@smerth
Copy link

@smerth smerth commented Jun 8, 2019

I wrote plugin gatsby-source-gh-readme to pull file content data from a user's github account, it does the job it should do but I need to clear the cache before running gatsby develop or most of the time the build fails. I added a comment about this here because I think it might be related

@mkaradeniz
Copy link

@mkaradeniz mkaradeniz commented Sep 7, 2019

I'm running into trouble with gatsby-source-filesystem and remote assets.

I'm using gatsby-source-graphql to use GraphCMS in my project. I added a custom resolver to download the GraphCMS assets via createRemoteNode to use gatsby-image later.

gatsby-node.js:

exports.createResolvers = ({ actions, cache, createNodeId, createResolvers, store, reporter }) => {
  const { createNode } = actions;

  createResolvers({
    GCMS_Asset: {
      file: {
        type: `File`,
        resolve(source, args, context, info) {
          return createRemoteFileNode({
            url: source.url,
            store,
            cache,
            createNode,
            createNodeId,
            reporter,
          });
        },
      },
    },
  });
};

When first starting gatsby develop or gatsby build without having a .cache, it will download the asset(s) and this query works fine:

    query {
      gcms {
        assets(first: 1) {
          url
          file {
            childImageSharp {
              fluid {
                src
                originalName
              }
            }
          }
        }
      }
    }

the second time I start develop or build with the .cache present, I get an ENOENT error:

Unexpected error value: "failed to process https://media.graphcms.com/ASSET_HANDLE\nError: ENOENT: no such file or directo
ry, stat 'C:\\...\\.cache\\gatsby-source-filesystem\\a7762a6a4f2571864804231ff8b216e6\\ASSET_HANDLE
gR'"

The file does exist:

C:\..\.cache\gatsby-source-filesystem\a7762a6a4f2571864804231ff8b216e6\ASSET_HANDLE.jpg

The only thing that I do find odd, is that it's looking for the file ASSET_HANDLE not ASSET_HANDLE.jpg.

Deleting the .cache would work, but in the future I'll have about 3k images in my project, so downloading on every build would be a waste of bandwith and time.

Edit:
After some digging, I think my suspicion was right. The file can't be found because we don't get a file extensions from looking at the url, so ext is empty. When the file is not in cache, we get the ext from the buffer, but that will not be called when trying to get the file from .cache.

As a workaround I could add the ext: '.jpg' when using createRemoteFileNode, but that would limit me to only use one extension for all assets, which might not be what I want or, since I'm using GraphCMS in this example, I could get the original filename and get the extension from that, but that also doesn't seem optimal.

@DmacTorstar
Copy link

@DmacTorstar DmacTorstar commented Oct 2, 2019

Having this issue, and detailed my experience in this ticket:
#18030

@m-allanson m-allanson unpinned this issue Oct 8, 2019
@slaterbbx
Copy link

@slaterbbx slaterbbx commented Dec 8, 2019

I think I have found that when using gatsby-background-image I have random .cache issues that are only on the client side when viewing the website, If I clear my local cach or turn off the local cache and then reload the pages, it will work fine after. Anyone have a similar issue?

Has me scared to do webhook based rebuilds on my project, If I do, can I just run gatsby clean then gatsby build to guarantee a clean build?

@sidharthachatterjee
Copy link
Contributor

@sidharthachatterjee sidharthachatterjee commented Jan 16, 2020

Let’s go ahead and close this for now since it’s been relatively quiet and people seem to have been hitting this less often than before.

Please open independent issues with reproductions for cases where caching breaks your gatsby build so that we can action those!

Thanks everyone!

@jonahallibone
Copy link

@jonahallibone jonahallibone commented Feb 27, 2020

Unsure if this is an issue related solely to Gatsby, but I've had to use this command twice in the last two days. I am using the gatsby-source-contentful plugin, and have found that it will often cache images when content changes, leading to GraphQL throwing errors on those models. This appears to only happen with SVG images, but it's possible I am changing those most often.

@LekoArts LekoArts removed status: needs reproduction not stale labels May 26, 2020
@NigeAtAnd
Copy link

@NigeAtAnd NigeAtAnd commented May 26, 2021

I'm fairly new to Gatsby and also had to implement a clean step to get it to pick up changes in page property values from gatsby-source-contentful. The issue seems to be that despite the Contentful plugin seeing that the item in Contentful has been updated, the page's HTML isn't being regenerated - after the build, the index.html file has an old timestamp.

Here's a log in case this helps:

> gatsby build

success open and validate gatsby-configs - 0.185s
success load plugins - 2.945s
success onPreInit - 0.049s
success initialize cache - 0.006s
success copy gatsby files - 0.581s
success onPreBootstrap - 0.036s
success createSchemaCustomization - 0.014s
success Contentful: Sync changed items - 0.289s - 1/1000 3461.54/s
success Contentful: Fetch data (xxx) - 1.235s
info Updated entries 1
info Deleted entries 0
info Updated assets 0
info Deleted assets 0
success Contentful: Process data (xxx) - 0.131s
info Creating 2 Contentful -- App download nodes
info Creating 2 Contentful -- Pull quote nodes
info Creating 1 Contentful -- HTML Template nodes
info Creating 19 Contentful -- Image and text card nodes
info Creating 23 Contentful -- Internal link nodes
info Creating 18 Contentful -- External link nodes
info Creating 63 Contentful -- Text nodes
info Creating 28 Contentful -- Hero nodes
info Creating 2 Contentful -- Video nodes
info Creating 14 Contentful -- Signpost link nodes
info Creating 7 Contentful -- Image nodes
info Creating 73 Contentful -- Section nodes
info Creating 1 Contentful -- Statement block nodes
info Creating 31 Contentful - Page nodes        
info Creating 27 Contentful -- Editorial card nodes
info Creating 38 Contentful asset nodes
success Contentful: Create nodes (xxx) - 0.106s
success Checking for changed pages - 0.003s
success source and transform nodes - 1.743s
success building schema - 2.078s
info Total nodes: 755, SitePage nodes: 32 (use --verbose for breakdown)
success createPages - 1.125s
success Checking for changed pages - 0.001s
success createPagesStatefully - 0.064s
success Cleaning up stale page-data - 0.007s
success update schema - 0.404s
success onPreExtractQueries - 0.005s
success extract queries from components - 2.220s
success write out redirect data - 0.006s
success Build manifest and related icons - 0.100s
success onPostBootstrap - 0.106s
info bootstrap finished - 19.075s
success write out requires - 0.019s
success Building production JavaScript and CSS bundles - 21.221s
success Building HTML renderer - 1.888s
info There are no new or changed html files to build.
success onPostBuild - 0.001s
info Done building in 42.3978705 sec

Environment:

  System:
    OS: Windows 10 10.0.19042
    CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
  Binaries:
    Node: 14.16.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.4 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 6.14.11 - C:\Program Files\nodejs\npm.CMD
  Languages:
    Python: 3.8.8 - C:\Users\Admin\.pyenv\pyenv-win\shims\python.BAT
  Browsers:
    Chrome: 90.0.4430.212
    Edge: Spartan (44.19041.906.0), Chromium (90.0.818.66)

Also from package.json:

    "gatsby": "^3.3.1",
    "gatsby-plugin-catch-links": "^3.4.0",
    "gatsby-plugin-image": "^1.4.0",
    "gatsby-plugin-manifest": "^3.5.0",
    "gatsby-plugin-postcss": "^4.4.0",
    "gatsby-plugin-react-helmet": "^4.4.0",
    "gatsby-plugin-sharp": "^3.4.0",
    "gatsby-source-contentful": "^5.4.0",
    "gatsby-source-filesystem": "^3.4.0",
    "gatsby-transformer-remark": "^4.1.0",
    "gatsby-transformer-sharp": "^3.4.0",

@edsu
Copy link

@edsu edsu commented Jul 10, 2021

I've definitely observed strange behavior when I:

gatsby clean
gatsby develop
# gatsby app runs fine ...
<ctrl-c>
gatsby develop
# gatsby app broken, graphql editor shows missing data

and I get a cannot query field GraphQL error. Then if I do a gatsby clean I can run gatsby develop again, but only once.

I noticed in a comment above that @keithmifsud thought that changing the property name in his JSON data from 'identifier to name helped. I had an id property in my JSON objects. Changing this to personId made the entire problem go away and I can repeatedly run gatsby develop without needing to clean the cache.

Does anyone know what might be going on in situations like this? My GraphQL query included a clause to another JSON file with an id property. Were they clashing somehow? Is there a list of reserved words that can't be used in JSON objects when inferring the schema? I spent hours, maybe days debugging this, and the fix is entirely unsatisfying.

  query($id: String!) {
    peopleJson(id: {eq: $id}) {
      id
      name
      type
    }
    wikipediaJson(personId: {eq: $id}) {
      personId
      abstract
      image {
        childImageSharp {
          gatsbyImageData(width: 300)
        }
      }
    }

edsu added a commit to umd-mith/airwaves that referenced this issue Jul 10, 2021
This fixes the problem we started seeing recently when running `npm run
start` more than once caused a Missing field GraphQL error, which
resulted in the site failing to run and the GraphQL editor to show that
the wikipedia.json file was no longer queryable. Running `npm run clean`
would then allow things to work again. But this caused problems in
environments when we wanted to keep the cache around for a faster build,
and it was just annoying when developing the site.

I tried a bunch of things but ran across an issue on GitHub where
someone said that changing the name of a property from `identifier` to
`name` in the JSON data they were querying made this problem go away. I
didn't have an `identifier` property but I did have an `id` and thought
changing it to `personId` might help. It did make the problem go away. I
tried to document this on GitHub for Gatsby here:

gatsbyjs/gatsby#11747 (comment)

This was a difficult problem to debug, and the solution was really
unsatisfying because I don't understand why it helped. I think if we are
going to continue using Gatsby in the future we might want to invest
some time learning how to define our schema rather than letting Gatsby
infer it.
@KyleAMathews
Copy link
Contributor

@KyleAMathews KyleAMathews commented Jul 10, 2021

My GraphQL query included a clause to another JSON file with an id property. Were they clashing somehow?

@edsu This is likely it. gatsby-transformer-json uses your id if you supply one but Node IDs must be universally unique as all nodes live in the same namespace — see https://www.gatsbyjs.com/docs/reference/config-files/actions/#createNode

@ibrahimaq
Copy link

@ibrahimaq ibrahimaq commented Mar 26, 2022

Ran into this issue today. I was setting up Gatsby with Contentful and wrote some dummy text in the Rich Text Field in Contentful to fetch through GraphQL. It was running smooth until I edited the rich text/embedded assets. It through this error:

Missing onError handler for invocation 'building-schema', error was 'Invariant Violation: Encountered an error trying to infer a GraphQL type for: `blog    
post___NODE`. There is no corresponding node with the `id` field matching: "6G3Uz8VEYCJCNEsOk5z5jP".'. Stacktrace was 'Invariant Violation: Encountered an  
error trying to infer a GraphQL type for: `blog post___NODE`. There is no corresponding node with the `id` field matching: "6G3Uz8VEYCJCNEsOk5z5jP".        
    at invariant (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\invariant\invariant.js:40:15)
    at getFieldConfigFromFieldNameConvention (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\add-inferred-fields.js:254:3)  
    at getFieldConfig (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\add-inferred-fields.js:143:19)
    at forEach (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\add-inferred-fields.js:88:25)
    at Array.forEach (<anonymous>)
    at addInferredFieldsImpl (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\add-inferred-fields.js:72:28)
    at addInferredFields (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\add-inferred-fields.js:27:3)
    at addInferredType (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\index.js:107:3)
    at map (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\index.js:65:5)
    at addInferredTypes (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\index.js:64:23)
    at updateSchemaComposer (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\schema.js:137:9)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processTimers (node:internal/timers:497:9)
    at build (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\index.js:112:18)'

⠸ building schema

and building scheme never ended until I Ctrl C.

I ran gatsby clean to clear the .cache and public folder. Started dev server and it worked.
After endless digging I think I'm going to clean .cache and public folder as part of my build from now on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug
Projects
None yet
Development

No branches or pull requests