question - incremental builds support = part II

[ekdev2]

#4981
I think @LekoArts is right. What I mean is if you generate a site with 2000 pages and deploy to aws, then one of those content pages changes in the cms, can you generate just that one page and deploy it.

[m-allanson]

It's not something Gatsby does at the moment, but it is something people have asked for. There's been work in version 2 to improve performance on larger sites, but there's no release date for that yet.

[mattferderer]

@m-allanson is there a discussion/issue on how to handle this? I didn't see it in the link you listed. I'm curious to hear conversations on how to handle doing this on a host like Netlify & using a CMS like Wordpress/Drupal that currently require a lot of HTTP requests during build.

[pieh]

AFAIK you wouldn't been able to use incremental builds on netlify because .cache and public directories are not preserved between builds, so it will always do clean build

[mattferderer]

That's good to know. I'm tossing around a ton of ideas that aren't well thought out. So even if we could eliminate the need for HTTP requests, we still need to make sure the .cache and public directories can be referenced by the build tool which eliminates many of the hosts that lower the bar to entry.

[robertschneiderman]

Another use case for incremental building is when you have a very large site that you want to build in parts. I was getting "heap out of memory error" when building ~5k pages at once.

We plan on our site getting very large, so we're testing Gatsby at larger scales. We've tried doing something like this path: './src/pages/${subPath}', where subPath is process.argv[3]. This works nicely when we host parts of our site with gatsby develop. It also circumvents the problems with the memory heap when using gatsby build for a 5k+ page site. For it to really be a solution, it would probably depend on the ability to specify an output subdirectory within the public folder: #4756

[ekdev2]

what if another approach is used to achieve the same goal. I wanted to run an idea by you guys and see what people think. So lets say you have a 5k page web site. The initial pages would be generated statically but each page will have a sub component that will load on top of the static content with the same content thats read from static json files. This way if a user wanted to update one page in the CMS in the middle of the day, they can make the update and just that static json file would be regenerated and deployed to a CDN. Then you can just regenerate the whole site maybe once a day as a nightly process. The seo static content might not be the most up to date during the day but I dont see that as a big deal. It will just get updated during the nightly process.

[tsimons]

@robertschneiderman we've run in to the memory issue as well. We're closer to 1500 pages, but an insane amount of images (design blog). We've turned off source maps, and stopped the build from downloading image files, but ultimately had to edit the build command to increase the memory allocated to the node instance. via the --max_old_space_size flag.

One thing that worries me about this feature is schema building. If we don't have every post available for gatsby to build a schema from, our queries will throw errors. It would be really nice if there was a way to pass schemas to gatsby, or at least provide dummy entities during the build to demonstrate the different shapes they may take.

[agonsalves]

I am considering using Gatsby to build the UI for a content site with over 5000 items, most with interconnected relationships to each other. The data will come from a database-driven CMS.

The benefit to using Gatsby over a standard API-driven React site is that I would spend a fraction of the time building and maintaining the data API and state management system that loads the remote data and stores it. (Since I plan on deploying this application for multiple sites of similar size, this seems like a very valuable benefit.)

The downside to using Gatsby in this case would be the fact that the entire site would need to be rebuilt for even the most insignificant content update. Forgot to add a comma? Rebuild all 5000 pages! Who even knows how long that would take? This is even more of an issue when considering the experience of the CMS users - they're used to seeing changes appear on the site immediately after they save them. With Gatsby, we're looking at a few minutes' wait (at least) before the change appears.

If there were a way to trigger builds for a subset of pages, it would make Gatsby the clear, definitive choice. At this moment, though, it's a tough sell.

[KyleAMathews]

BTW, I've been working a lot on improving speeds for larger site builds for v2. On the latest v2 beta — you might be able to build 5000 pages in < 1:30. There'll be more speed improvements coming.

[tsimons]

That's amazing @KyleAMathews! I definitely look forward to that! Let me know if you want to test against an image heavy blog

[brod-ie]

@KyleAMathews 5K is nice but we need 1M 😉

[Tawfiqh]

If we want to compile parts of the site separately, we can set flags on build so that gatsby-node knows only to generate the parts of the site specified. We could then add back in the previously generated static files. This works for us as long as we link to the previously generated files with a basic <a href> as opposed to a <Link to >.

I'm wondering if we can make <Link to> work when linking to previously generated files if we merge in some of the previous data.json at build time. Looking into that a bit more at the moment.

[rbmedia]

I have no worry with the build time but more with the volume of static files that I need to upload for any update, we launched a large visual portfolio with Gatsby and the static site to upload is over 150 MB
Mostly images.
This makes the site unavailable around 40 minutes during an update
The availability to rebuild a part of the site is definitely a feature that would boost Gatsby.
I plan to use Gatsby for a new site but I will divide the site in a static and dynamic part using a traditional php CMS for the news part.

[mattferderer]

@rbmedia you might want to consider a host that does deployment switching like Netlify so your current site stays running until your new version is ready.

[rbmedia]

Thanks Matt, I will consider it!
I did built some News websites with Drupal in the past, any update had to be online within a short lapse of time (less than 2 minutes). I would love to use Gatsby in the future for this kind of sites.

[KnisterPeter]

Any news on this issue? We plan a site with around 100k pages and incremental builds would be awesome.

[Leocn]

make another path as default static page folder, not '/public'.
After run gatsby build, copy the ../public/* to the default path.

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[Tawfiqh]

I still don't think this is fixed/supported in Gatsby. Any news @ TeamGatsby?

[wardpeet]

it's a long-standing issue because it's really hard to fix without thinking heavily about it. @Moocar has an issue open to at least get us a step in the right direction.

[xaviemirmon]

Incremental builds on Gatsby Cloud fixes these issues. You can signup for the private beta here https://www.gatsbyjs.com/builds-beta/

[dwightwatson]

Nothing about that seems to suggest it supports incremental builds though - just that it has the "fastest build times for Gatsby sites".

I'd be concerned about the implication that incremental builds would only be available on a hosted Gatsby service rather than available to be used standalone.

[xaviemirmon]

I see what you mean @dwightwatson there's nothing on the website that says it's "incremental." At Gatsby Days London they demoed builds and it was definitely incremental builds. Not sure how it's done though and if it will be apart of the Gatsby package or if it's just going to be a service they provide.

[agonsalves]

Investors gotta make their money back somehow. 🙄

[gomflo]

trying to build very large website 140k+ pages

gatsby build is somewhat good… but doing the deployment its painful (zeit.co)

[scandeezy]

Unsure how to add a label to this, but I'm putting this down as still an issue.

[pvdz]

@gomflo is there a way I can build your site? There might be some low hanging fruit to resolve to improve build times :) No promises.

[ethannkschneider]

Nothing about that seems to suggest it supports incremental builds though - just that it has the "fastest build times for Gatsby sites".

I'd be concerned about the implication that incremental builds would only be available on a hosted Gatsby service rather than available to be used standalone.

re this ^: If my gatsby repo is in gitlab and not github, will I be able to use gatsby cloud/build features?

[sielay]

I might have mentioned that before, but regards original issue/feature. For publishers, Gatsby will make sense if we could trigger the only generation of new pages plus maybe update to indexes. Hardly any publisher would care about updating old canonical pages.

[mushkin-v]

So will we have a standalone partial update or no chances? Maybe there is some another way to update only a few pages and don't rebuild the whole project?

[dominicfallows]

Just wanted to update that my team is close to publishing a PR into the Gatsby repo that we think enables incremental builds. We're just taking some time to write a good PR and tighten up the code, but I will update here when we are done (in the next week or so).

[dominicfallows]

Just wanted to update that my team is close to publishing a PR into the Gatsby repo that we think enables incremental builds. We're just taking some time to write a good PR and tighten up the code, but I will update here when we are done (in the next week or so).

Here's the PR #20785

[dominicfallows]

Further update to the PR: #20785 (comment)

[dominicfallows]

New PR, focusing in on incremental data changes #21523

[lannonbr]

With #21523 merged in and Incremental Builds available in Gatsby Cloud, I believe this issue is resolved. It doesn't support all workflows, but I am going to close this for now and it may be better to open a new issue in the future for future endeavors if need be.

[agonsalves]

Should it really be closed? The optimization was just that - an optimization. It wasn't truly incremental builds. On top of that, whatever is available through Gatsby Cloud is not available through usage of the public package. For the specific intent of this ticket, nothing has been resolved.

[dandv]

Should it really be closed?

Based on #5496 (comment), I don't think this issue should stay closed, and I don't understand why the not stale label was removed.

[datakurre]

Has anyone here tried, or know if it is possible, to tweak GatsbyJS webpack configuration to simultaneously produce both development preview and production build version with ”gatsby develop”? (Possibly resulting ”incremental builds” with cost of forever running development server.)