Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large sites ~200k pages, incremental builds? #9083

Closed
Tawfiqh opened this issue Oct 13, 2018 · 9 comments
Closed

Large sites ~200k pages, incremental builds? #9083

Tawfiqh opened this issue Oct 13, 2018 · 9 comments

Comments

@Tawfiqh
Copy link
Contributor

@Tawfiqh Tawfiqh commented Oct 13, 2018

Summary

We have a large site of about 200k pages and we don't want to have to rebuild the whole site every time we make a change and deploy.

Relevant information

A lot of our pages are set pages that pull in a lot of data and take a while to build but they won't change often.

We could just build them in a separate Gatsby project to the rest of our site. Having two Gatsby projects set-pages and main-site and then combining the two built /public directories. But as far as I can tell, that won't allow us to use the <Link to=...> component to get quick SPA performance
when transitioning between pages from the main-site and the set pages.

Ideal situation

Ideally we want to build those "set-pages" once, and then each time we update our gatbsy project we don't have to rebuild them.

Is there a good/recommended way we can selectively rebuild just parts of the site on deploy? We would prefer not to rebuild the whole site on every deployment.

Current attempts

Currently our best thinking is to split the set-pages into a separate project from the main-site, combine the two outputs and just accept that when linking between the two we should use an <a href...> instead of a <Link to=...>.

Have read a lot of closed issues on this I can see that the faster build times in Gatsby v2 help alleviate this issue for a lot of sites, but 200k pages is where we are and we're wondering if build times will be manageable as we increase scale. Mainly we don't want small blog updates having to rebuild the entire set-pages.

@pieh

This comment has been minimized.

Copy link
Contributor

@pieh pieh commented Oct 15, 2018

At this moment it's not possible to do incremental builds. It's definitely something we will want to support but it's not trivial to do as currently all code expects that everything is built when running gatsby build.

Here's open issue for that #5002, so I'll close this to not have multiple open duplicate issues

@dcworldwide

This comment has been minimized.

Copy link

@dcworldwide dcworldwide commented Apr 3, 2019

I also have a site with a large # of pages (160k and growing). I need partial updates (1k page updates a day). Would love to learn more about whether this can be solved.

@justinwilson-apollidon

This comment has been minimized.

Copy link

@justinwilson-apollidon justinwilson-apollidon commented Apr 5, 2019

+1 for this.

@dskvr

This comment has been minimized.

Copy link

@dskvr dskvr commented Apr 8, 2019

+1

2 similar comments
@parkerproject

This comment has been minimized.

Copy link

@parkerproject parkerproject commented Apr 11, 2019

+1

@cullophid

This comment has been minimized.

Copy link

@cullophid cullophid commented May 9, 2019

+1

@DSchau

This comment has been minimized.

Copy link
Contributor

@DSchau DSchau commented Jun 3, 2019

@fenchi thanks for showing your support, but if you can just react to the original issue with an emoji, that's a much more helpful way of showing your support. Thanks!

(For full context, we can then sort issues by # of reactions, which can be a helpful signal to help us prioritize)

@sanjaySP

This comment has been minimized.

Copy link

@sanjaySP sanjaySP commented Jun 6, 2019

+1

@b1lb0

This comment has been minimized.

Copy link

@b1lb0 b1lb0 commented Oct 29, 2019

+10k :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
You can’t perform that action at this time.