Builds start failing after awhile

[jasonLaster]

Describe the bug

It looks like our builds start failing after 5-10 builds. My hunch is that the .cache is filling up as we're using the netlify gatsby build plugin.

https://app.netlify.com/sites/sustainability-landscape/deploys?filter=deploy%20previews

[jasonLaster]

CC @jlengstorf have you noticed any issues in the past with the cache not clearing over time?

[ehmicky]

Hi @jasonLaster, thanks a lot for reporting this.

From looking at the deploys, it looks like what's making builds slower and slower is the very end of the build, where we upload files to the CDN.

You are probably right, where the .cache directory gets bigger and bigger on each build. It actually does not have a huge impact on cache downloading/uploading itself (it remains around 1.3GB), but it might have an impact on uploading files to the CDN.

The .cache should be cleared between builds instead of augmenting incrementally.

We will look into this issue as soon as possible. I would recommend turning off this Build plugin until this issue is fixed.

[jasonLaster]

Okay, i'll try disabling the cache and see how it does. We don't need to publish the .cache to the CDN so if there is a way to just publish the public dir, that would speed things up too

[ehmicky]

There are a couple of issues with the Gatsby cache plugin at the moment. I have filed the following ones:

• Issue when caching the whole repository jlengstorf/netlify-plugin-gatsby-cache#8
• Should this plugin cache the publish directory? jlengstorf/netlify-plugin-gatsby-cache#9
• Issue caching the .cache directory jlengstorf/netlify-plugin-gatsby-cache#10

This GitHub issue is most likely due to one or several of the issues listed above.

I recommend not using this plugin until those issues are fixed, which we will aim at solving as soon as possible.

[ehmicky]

After investigating it, it looks like this issue might not be due to Netlify nor to netlify-plugin-gatsby-cache but to gatsby-source-filesystem and/or Airtable.

Some builds crash suddenly but this seems related to the problems highlighted in the following issues: climatescape/climatescape.org#199 and climatescape/climatescape.org#81. While enabling the netlify-plugin-gatsby-cache randomly triggers that crash, its root cause seems to be the following issues and libraries. Running gatsby build locally (without Netlify) several times in a row also seem to produce the above warnings.

It turns out caching seems to correctly work for Gatsby, at least on the latest version of Netlify:

• When a new Site is created and Gatsby is detected, the "Publish directory" is set to public (which is correct).
• Only the publish directory is published to the CDN.
• netlify-plugin-gatsby-cache correctly caches the public and .cache directories, which amounts to ~1.3GB on that repository. The next builds re-use that cache, making their duration much faster (gatsby build goes from 9 to 2 minutes).

After fixing the problems above, you should be able to re-enable the netlify-plugin-gatsby-cache. I have opened a PR to help re-enabling it then. If the problem still occurs, please re-open this issue. I will close it for the moment since Netlify does not seem to be the root cause.

There are few additional bugs in the buildbot that can be experienced in your problem but are actually separate:

• Caching of new PR/branches sometimes fails (https://github.com/netlify/buildbot/issues/575). For the moment, using the "Clear cache and deploy site" button in the Netlify app usually fixes this problem.
• The bug described above makes the whole build crash. We should handle these types of errors instead of abruptly terminating the builds and printing no logs (https://github.com/netlify/buildbot/issues/655).

[bloudermilk]

@ehmicky somehow missed this detailed analysis. Thank you for digging in and providing so much context. We'll circle back after we've had a chance to evaluate on our end!

[jbolda]

👋 (I built the gatsby-source-airtable plugin.)

I was chatting with Jason about this trying to decide the root cause. I am suspecting it is an issue with the way that createRemoteFileNode does the cache. It is a function that comes from gatsby-source-filesystem. Part of me is wondering if some of the issue is related to how it is inferring the file type and long file names. I would be curious if your results if you have a chance to test out the next version: jbolda/gatsby-source-airtable#187

also related: jbolda/gatsby-source-airtable#170