Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[gatsby-source-contentful] What's the cache directory location for this plugin? #16590

Open
tmilewski opened this issue Aug 13, 2019 · 2 comments

Comments

@tmilewski
Copy link

commented Aug 13, 2019

Summary

I'm using the gatsby-plugin-netlify-cache plugin which, by default, caches the .cache directory on Netlify. I've confirmed that it's successfully caching the appropriate data/directories, but gatsby-source-contentful is still re-requesting all data from Contentful on every build.

I noticed locally that I don't see any data from gatsby-source-contentful in .cache, despite the data not being fully re-requested on every update/build.

So, my simple question is, where is this data being stored, if not in .cache?

Relevant information

Environment

$ gatsby info

  System:
    OS: macOS High Sierra 10.13.6
    CPU: (8) x64 Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz
    Shell: 5.6.2 - /usr/local/bin/zsh
  Binaries:
    Node: 12.6.0 - /var/folders/_x/sz_v8_d550v91rpxrz2txstr0000gq/T/yarn--1565711575669-0.9816823406056241/node
    Yarn: 1.17.3 - /var/folders/_x/sz_v8_d550v91rpxrz2txstr0000gq/T/yarn--1565711575669-0.9816823406056241/yarn
    npm: 6.9.0 - /usr/local/bin/npm
  Languages:
    Python: 2.7.10 - /usr/bin/python
  Browsers:
    Chrome: 75.0.3770.100
    Firefox: 68.0.1
    Safari: 12.1.1
  npmPackages:
    gatsby: ^2.13.31 => 2.13.31
    gatsby-image: ^2.2.6 => 2.2.6
    gatsby-plugin-google-fonts: ^1.0.1 => 1.0.1
    gatsby-plugin-google-tagmanager: ^2.1.4 => 2.1.4
    gatsby-plugin-manifest: ^2.2.3 => 2.2.3
    gatsby-plugin-offline: ^2.2.4 => 2.2.4
    gatsby-plugin-purgecss: ^4.0.0 => 4.0.0
    gatsby-plugin-react-helmet: ^3.1.2 => 3.1.2
    gatsby-plugin-sass: ^2.1.3 => 2.1.3
    gatsby-plugin-sharp: ^2.2.8 => 2.2.8
    gatsby-plugin-sitemap: ^2.2.3 => 2.2.3
    gatsby-source-contentful: ^2.1.19 => 2.1.19
    gatsby-source-filesystem: ^2.1.5 => 2.1.5
    gatsby-transformer-remark: ^2.6.7 => 2.6.7
    gatsby-transformer-sharp: ^2.2.4 => 2.2.4

File contents

gatsby-config.js:

/* eslint-disable max-len */
const activeEnv =
 process.env.GATSBY_ACTIVE_ENV || process.env.NODE_ENV || 'development'

require('dotenv').config({
 path: `.env.${activeEnv}`
})

module.exports = {
 siteMetadata: {
   title: `...`,
   description: `...`,
   author: `@...`,
   siteUrl: `https://www.abc.com`,
 },
 plugins: [
   {
     resolve: 'gatsby-plugin-netlify-cache',
     options: {
       cachePublic: true
     }
   },
   {
     resolve: `gatsby-source-contentful`,
     options: {
       spaceId: process.env.CONTENTFUL_SPACE_ID,
       accessToken: process.env.CONTENTFUL_ACCESS_TOKEN,
       host: (process.env.CONTENTFUL_HOST || 'cdn.contentful.com'),
       environment: (process.env.CONTENTFUL_ENV || 'master'),
       downloadLocal: (!!process.env.CONTENTFUL_DOWNLOAD_LOCAL)
     },
   },
   {
     resolve: `gatsby-transformer-remark`
   },
   {
     resolve: `gatsby-plugin-sass`,
     options: {
       implementation: require('sass'),
       postCssPlugins: [
         require('postcss-flexbugs-fixes'),
         require('autoprefixer')
       ],
       cssLoaderOptions: {
         localIdentName: activeEnv === 'development' ?
           '[name]--[local]-[hash:base64:5]' :
           '[local]-[hash:base64:6]'
       }
     },
   },
   `gatsby-plugin-react-helmet`,
   {
     resolve: `gatsby-source-filesystem`,
     options: {
       name: `images`,
       path: `${__dirname}/src/images`,
     },
   },
   `gatsby-transformer-sharp`,
   `gatsby-plugin-sharp`,
   {
     resolve: `gatsby-plugin-manifest`,
     options: {
       name: `...`,
       short_name: `...`,
       start_url: `/`,
       background_color: `#00b3e6`,
       theme_color: `#00b3e6`,
       display: `minimal-ui`,
       icon: `src/images/favicon.ico`,
       crossOrigin: `use-credentials`
     },
   },
   {
     resolve: `gatsby-plugin-google-tagmanager`,
     options: {
       id: 'GTM-...',
       includeInDevelopment: true
     },
   },
   {
     resolve: 'gatsby-plugin-purgecss',
     options: {
       develop: true,
       printRejected: false,
       purgeOnly : ['src/components/Layout/']
     },
   },
   {
     resolve: `gatsby-plugin-sitemap`,
     options: {
       exclude: [
         `/...`,
       ],
       query: `
       {
         site {
           siteMetadata {
             siteUrl
           }
         }

         allSitePage(filter: {fields: {hideFromSearchEngines: {eq: false}}}) {
           edges {
             node {
               path
               fields {
                 lastmod
                 changefreq
                 priority
               }
             }
           }
         }
       }`,
       serialize: ({ site, allSitePage }) =>
         allSitePage.edges.map(edge => {
           return {
             url: site.siteMetadata.siteUrl + edge.node.path,
             lastmod: edge.node.fields.lastmod,
             changefreq: edge.node.fields.changefreq,
             priority: edge.node.fields.priority,
           }
         }),
     }
   }
 ],
}

package.json: N/A

{
  "name": "abc.com",
  "private": true,
  "description": "...",
  "version": "5.0.0",
  "author": "... <info@abc.com>",
  "browserslist": [
    ">1%",
    "last 4 versions",
    "not dead"
  ],
  "dependencies": {
    "@contentful/rich-text-plain-text-renderer": "^13.1.0",
    "@contentful/rich-text-react-renderer": "^13.2.0",
    "@reach/tabs": "^0.1.6",
    "airbrake-js": "^1.6.8",
    "autoprefixer": "^9.6.1",
    "axios": "^0.19.0",
    "babel-runtime": "^6.26.0",
    "classnames": "^2.2.6",
    "fs-extra": "^7.0.0",
    "gatsby": "^2.13.31",
    "gatsby-image": "^2.2.6",
    "gatsby-plugin-google-fonts": "^1.0.1",
    "gatsby-plugin-google-tagmanager": "^2.1.4",
    "gatsby-plugin-manifest": "^2.2.3",
    "gatsby-plugin-offline": "^2.2.4",
    "gatsby-plugin-purgecss": "^4.0.0",
    "gatsby-plugin-react-helmet": "^3.1.2",
    "gatsby-plugin-sass": "^2.1.3",
    "gatsby-plugin-sharp": "^2.2.8",
    "gatsby-plugin-sitemap": "^2.2.3",
    "gatsby-source-contentful": "^2.1.19",
    "gatsby-source-filesystem": "^2.1.5",
    "gatsby-transformer-remark": "^2.6.7",
    "gatsby-transformer-sharp": "^2.2.4",
    "js-cookie": "^2.2.0",
    "lodash.kebabcase": "^4.1.1",
    "node-sass": "^4.11.0",
    "postcss-flexbugs-fixes": "^4.1.0",
    "program-language-detector": "^0.0.12",
    "prop-types": "^15.7.2",
    "query-string": "^6.8.1",
    "react": "^16.8.2",
    "react-collapsible": "^2.6.0",
    "react-contentful-image": "^1.3.2",
    "react-device-detect": "^1.6.5",
    "react-dom": "npm:@hot-loader/react-dom",
    "react-helmet": "^5.2.1",
    "react-modal": "^3.9.1",
    "react-player": "^1.11.1",
    "react-reveal": "^1.2.2",
    "react-slick": "^0.24.0",
    "react-sticky-el": "^1.0.20",
    "react-syntax-highlighter": "^11.0.2",
    "react-visibility-sensor": "^5.1.0",
    "sass": "^1.22.7",
    "scroll-into-view": "^1.9.7",
    "sharp": "^0.22.1",
    "slick-carousel": "^1.8.1",
    "url-search-params-polyfill": "^7.0.0",
    "uuid": "^3.3.2"
  },
  "keywords": [
    "gatsby"
  ],
  "license": "MIT",
  "scripts": {
    "build": "gatsby build",
    "develop": "gatsby develop",
    "start": "npm run develop",
    "serve": "gatsby serve",
    "test": "jest src",
    "coverage": "jest src --coverage",
    "storybook": "STORYBOOK_ENV=true start-storybook -p 6006",
    "build-storybook": "build-storybook"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/..."
  },
  "bugs": {
    "url": "https://github.com/.../issues"
  },
  "devDependencies": {
    "@babel/core": "^7.5.5",
    "@storybook/addon-a11y": "^5.1.9",
    "@storybook/addon-actions": "^5.1.9",
    "@storybook/addon-backgrounds": "^5.1.9",
    "@storybook/addon-links": "^5.1.9",
    "@storybook/addon-notes": "^5.1.9",
    "@storybook/addon-storysource": "^5.1.9",
    "@storybook/addon-viewport": "^5.1.9",
    "@storybook/addons": "^5.1.9",
    "@storybook/react": "^5.1.9",
    "@storybook/theming": "^5.1.9",
    "@testing-library/jest-dom": "^4.0.0",
    "@testing-library/react": "^8.0.5",
    "@testing-library/react-hooks": "^1.1.0",
    "axios-mock-adapter": "^1.17.0",
    "babel-eslint": "^10.0.2",
    "babel-jest": "^24.7.1",
    "babel-loader": "^8.0.5",
    "babel-plugin-remove-graphql-queries": "^2.7.2",
    "babel-preset-gatsby": "^0.2.7",
    "cross-env": "^5.2.0",
    "eslint": "^6.0.1",
    "eslint-config-fbjs": "^3.1.0",
    "eslint-plugin-babel": "^5.3.0",
    "eslint-plugin-flowtype": "^3.12.1",
    "eslint-plugin-jsx-a11y": "^6.2.3",
    "eslint-plugin-react": "^7.14.2",
    "eslint-plugin-relay": "^1.3.5",
    "identity-obj-proxy": "^3.0.0",
    "jest": "^24.7.1",
    "jest-localstorage-mock": "^2.4.0",
    "react-test-renderer": "^16.8.6"
  }
}

gatsby-node.js: Can't currently share.
gatsby-browser.js: N/A
gatsby-ssr.js: N/A

@axe312ger

This comment has been minimized.

Copy link
Member

commented Aug 13, 2019

It uses gatsby-source-contentfuls createRemoteFileNode to cache it, which stores in PROJECT_DIR/.cache/gatsby-source-filesystem

This will be cached when using gatsby-plugin-netlify-cache, but all files get deleted when the gatsby cache is reset. (Which can happen more often as you would think)

I can see how this can be very bad for huge sites. SQIP and the new video transformer will use node_modules/.cache/XYZ. We could make source-filesystem use that folder as well? Files won't be deleted on Gatsby cache reset and caching on netlify would even work without the netlify-cache plugin.

@tmilewski

This comment has been minimized.

Copy link
Author

commented Aug 14, 2019

Thanks for the reply, @axe312ger!

The approach of saving it in node_modules/.cache/XYZ would likely help. Long-term we might want to enable a simple command to clear the cache for each plugin as it might not be 100% clear where this data resides.


In the meantime, I don't know that the issue is that the cache is being reset. See below for the process:

Clean Build

rm -r .cache; yarn build yields the following:

Starting to fetch data from Contentful
Fetching default locale
default locale is : en-US
contentTypes fetched 28
Updated entries  2343
Deleted entries  0
Updated assets  2929
Deleted assets  0
Fetch Contentful data: 11828.563ms

Checking Cache Directories

No cached files to be found.

🔌  2:16 in dotcom/ on develop 
› ls -al .cache/gatsby-source-filesystem      
gls: cannot access '.cache/gatsby-source-filesystem': No such file or directory

🔌  2:16 in dotcom/ on develop 
› ls -al .cache/caches                         
total 0
drwxr-xr-x 18 tmilewski staff  576 Aug 14 09:41 ./
drwxr-xr-x 47 tmilewski staff 1504 Aug 14 09:41 ../
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 default-site-plugin/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 dev-404-page/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-plugin-manifest/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-plugin-page-creator/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-plugin-purgecss/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-plugin-sass/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-plugin-sharp/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:41 gatsby-plugin-sitemap/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-source-contentful/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-source-filesystem/
drwxr-xr-x 35 tmilewski staff 1120 Aug 14 09:41 gatsby-transformer-remark/
drwxr-xr-x  3 tmilewski staff   96 Aug 14 09:40 gatsby-transformer-sharp/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 internal-data-bridge/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 load-babel-config/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 prod-404/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 webpack-theme-component-shadowing/

🔌  2:16 in dotcom/ on develop 
› ls -al .cache/caches/gatsby-source-filesystem
total 0
drwxr-xr-x  2 tmilewski staff  64 Aug 14 09:40 ./
drwxr-xr-x 18 tmilewski staff 576 Aug 14 09:41 ../

🔌  2:16 in dotcom/ on develop
› ls -al .cache/caches/gatsby-source-contentful 
total 0
drwxr-xr-x  2 tmilewski staff  64 Aug 14 09:40 ./
drwxr-xr-x 18 tmilewski staff 576 Aug 14 09:41 ../

Second Build

yarn build yields the following, apparently with cached files:

Starting to fetch data from Contentful
Fetching default locale
default locale is : en-US
contentTypes fetched 28
Updated entries  0
Deleted entries  0
Updated assets  0
Deleted assets  0
Fetch Contentful data: 598.905ms
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.