Questions about Gatsby Cloud performance

[antonbuks]

👋

I recently received a newsletter about an update to Gatsby Cloud that boasts some incredible figures

up to 5x faster publishing times

and

up to 2x improvements in site performance

This is great stuff and I wanted to inquire a little bit about how you guys are achieving this or what kind of hardware you suggest using to speed up build performance by as much as 5x for those who are building on custom hardware.

I understand that this is a paid service so you might not be able to share much and as much as we would love to offload this workload to Gatsby Cloud without asking too many questions, it's just not possible for us for several reasons:

1. We're using a private gitlab enterprise repo
2. We're subhosting the gatsby site under a legacy php monolith thats handling a lot of redirects and cookies and stuff
3. We're using gitlab CI pipelines for everything from generating and importing translation files to running tests to deploying the subhosted files with rsync
4. We're not using a cms (we used to use drupal but eventually found it redundant as most of our pages require a lot of css and svg animations and we already had an internal translation tool that handles translations through git so there was little benefit to using a cms back in the day it was just easier to write a lot of react components)
5. We're using a variety of other services to pull in our data that would make a migration very complicated (internal rest apis with source-rest etc)

Gatsby is by far the most versatile framework we tested but our build times keep rising with every page we add; we're at about 10-15 mins with just under 1500 nodes on 8x core runners to build and deploy if we edit something like a shared component and all pages need to be rebuilt.

Any advice on where to look to optimise or what kind of hardware could help? Any suggestions on setting up a cms (self hosted) for a use case like this and which would you recommend? Even with a cms, if any shared component needs a code update (for something like animations or css) the build times will still be long 😕

we were looking into trying the webpack cache flag PRESERVE_WEBPACK_CACHE but haven't been able to make it work yet, some weird webpack errors were popping like warning WebpackError: Unsafe builtin usage fs.existsSync: (node 14.2) so we abandoned that for the time being 😄

[KyleAMathews]

Check out this doc for tips https://www.gatsbyjs.com/docs/how-to/performance/improving-build-performance/#gatsby-skip-here

1st thing I'd look at from your description is if you're preserving the .cache and public folder in between builds. That speeds things up a lot.

we were looking into trying the webpack cache flag PRESERVE_WEBPACK_CACHE but haven't been able to make it work yet, some weird webpack errors were popping like warning WebpackError: Unsafe builtin usage fs.existsSync: (node 14.2) so we abandoned that for the time being 😄

That's a warning btw not an error. It's just saying that Gatsby can't enable incremental builds as using fs means builds aren't deterministic. Also on later versions of v3, we enabled webpack caching for everyone by default.

[antonbuks]

We are preserving .cache and public folders yes, without it the builds are about twice as long so about 20-30 min. I think I tested the webpack flag on an early version of v3 and the warning you are talking about broke the build, I'll try it again tomorrow on latest.

[antonbuks]

I actually started going through that doc a while back and these bits here were what peeked my interest:

Run builds on a machine with more and higher-powered CPUs
The better the underlying hardware / VM / container you have, the faster your builds will go. As an anecdote, a Gatsby core team developer ran the same build on two machines they owned — a five-year-old low-powered Windows machine and a new MacBook Pro — and found that the latter gave a 30x build speed improvement.

Use Gatsby Cloud
Gatsby Cloud offers image parallelization during image processing and other build optimizations that can speed things up significantly. Try running your site on Gatsby Cloud and comparing the speeds there (both warm and cold builds) to your current CI/CD setup.

I was kind of hoping to get some more info on what specific hardware you are using for Cloud and what image parellelization entails

[antonbuks]

also something I have yet to try is setting GATSBY_CPU_COUNT, is this something that would help for our runners with 8 core cpu?