Slow build times

[willmcl]

Thanks for the plugin. :)

I have been using it for a gallery website. I have 5 tables.

One that holds assets - 450 records
4 others that hold 150, 120, 500 and 5 records respectively
Those tables have many linked fields particularly to the assets table.

The issue is that my build times are fluctuating between 6 and 10 minutes (depending on whether there are new records in airtable or not) on Gatsby cloud (free plan) and longer on my machine when running gatsby develop. (I know this last onemight be partly due to my machine. It has a tendency to run a bit slow, though not on any of my other gatsby sites sourcing content from other CMSs like Contentful or Prismic)

I have reduced my concurrency to 3 in the plugin options to avoid fails on my build, so I know that could be adding to it, but I still would like to reduce it somehow. I have included below a log from one of the longer builds.

I have looked at the timings and it seems the "source and transform nodes" (admittedly on this build it is not too bad at all) and the "running page queries" steps are the longer ones.

Restoring previous build cache and dependencies 42.67s
Install project dependencies 104.01s
Pulling latest commits from repository 1.48s
open and validate gatsby-configs, load plugins 2.18s
onPreInit 0.09s
initialize cache 0.29s
onPreBootstrap 0.82s
Compiling Gatsby Functions 0.79s
source and transform nodes 89.22s
createSchemaCustomization 0.01s
building schema 2.44s
Checking for changed pages 0.02s
createPages 0.69s
createPagesStatefully 0.34s
Checking for changed pages 0.00s
Cleaning up stale page-data 0.03s
extract queries from components 4.30s
onPreExtractQueries 0.00s
onPostBootstrap 1.81s
run static queries 1 / 1 0.16s
run page queries 80 / 80 371.52s
Building production JavaScript and CSS bundles 6.49s
Caching JavaScript and CSS webpack compilation 0.02s
Writing page-data.json files to public directory 0 / 0 0.39s
Building HTML renderer 4.35s
Building static HTML for pages 0 / 06.51s
Caching HTML renderer compilation 0.00s
onPostBuild 0.18s
Calculating files to upload 23.39s
Uploading 1 / 17.24s
Deploying Build to Netlify 188 / 188 23.92s
Running series of Lighthouse reports 7 / 7

I have included an example of one of my queries form an offending page that is flagged with a warning during build that the query is taking too long.

query exhibitionListByArtistPageQuery($artistId: String!, $skip: Int!, $limit: Int!) {
      allAirtableExhibitions(
        sort: {order: DESC, fields: data___From}
        filter: {data: {Artists: {elemMatch: {id: {eq: $artistId}}}}}
        limit: $limit
        skip: $skip
      ) {
        nodes {
          id
          data {
            Name
            Location
            From
            To
            Year
            Artists {
              id
              data {
                Name
                Represented
              }
            }
            Assets {
              id
              data {
                Image {
                  localFiles {
                    childImageSharp {
                      gatsbyImageData(
                        
                        layout: FULL_WIDTH
                        placeholder: BLURRED
                      )
                    }
                  }
                }
                Video_embed_code
                Custom_caption
                Installation
                Work {
                  data {
                    Title
                    Year
                    Size
                    Media
                    Edition
                    Artists {
                      id
                      data {
                        Name
                        Represented
                      }
                    }
                  }
                }
                Work_2 {
                  data {
                    Title
                    Year
                    Size
                    Media
                    Edition
                    Artists {
                      id
                      data {
                        Name
                        Represented
                      }
                    }
                  }
                }
              }
            }
            Hero_Image {
              id
              data {
                Image {
                  localFiles {
                    childImageSharp {
                      gatsbyImageData(
                        
                        layout: FULL_WIDTH
                        placeholder: BLURRED
                      )
                    }
                  }
                }
                Custom_caption
              }
            }
          }
        }
      }
  }

As you can see I query the linked fields (Assets and then Artists linked with those assets) heavily but should this cause slowness?

Any tips would be greatly appreciated.

[willmcl]

For anyone else reading this I am just noting a few things I did to improve the speed on these builds.

1. I refactored the pagination following the tip from point 5 in this article: https://dev.to/ascorbic/top-6-reasons-gatsby-incremental-builds-can-be-slow-and-how-to-make-them-fast-b75

2. I created a view with reduced content (through some filtering) for my assets that I could use when developing locally. This reduced the image processing from 3000 images to about 300. This was a massive win as the time to run gatsby develop was making the project unworkable.
   tableView: (process.env.NODE_ENV === 'production' ? Production:Development),

3. I upgraded to a paid gatsby plan in order to make use of gatsby cloud build. I don't fully understand why certain builds use this and some don't (When you look at your production history there is a cloud icon next to the builds that do use them), however on the builds that do use them they are much quicker.

The conclusion I have come to is that if you have thousands of images it might be good to use a image hosting service and just provide a link to the image in airtable. This would probably speed everything up. I usually advise clients to upload very large images and then I reduce them using gatsby-plugin-image options for display on the site. However, in this case that was probably not the best option. Downloading 3000 images on every build using a concurrency setting of 3 (to avoid errors) in the plugin settings is a slow process.

Hope this helps someone.