New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incremental Build (while watching) #1643

Open
spf13 opened this Issue Nov 29, 2015 · 6 comments

Comments

Projects
None yet
6 participants
@spf13
Copy link
Contributor

spf13 commented Nov 29, 2015

Hugo today rebuilds the entire site each time it is rendered. This isn't necessary and Hugo could be a lot more intelligent particularly during while watching where it already has read in the entire source and built up a tree.

What would need to happen...

  1. Rethink nodes a bit and come up with a better node system, one that has a repository of nodes like pages does.
  2. Add a dirty flag to pages & nodes
  3. When a file change is detected mark everything that file used to touch and will touch as dirty.
  4. Render & write only the dirty things.

@spf13 spf13 changed the title Incremental Build (during server) Incremental Build (while watching) Nov 29, 2015

@kindlychung

This comment has been minimized.

Copy link

kindlychung commented Aug 11, 2016

Looking forward to this feature!

@bep

This comment has been minimized.

Copy link
Member

bep commented Jul 1, 2017

This issue has been automatically marked as stale because it has not been commented on for at least six months.

The resources of the Hugo team are limited, and so we are asking for your help.

If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.

If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.

This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

@bep bep added the Stale label Jul 1, 2017

@jonathanulco

This comment has been minimized.

Copy link

jonathanulco commented Jul 1, 2017

In my view this is a huge feature request. Hugo should not restrict itself to documentation sites or small blogs but to prove to the whole of the computer services that it is today the state of the art and this even for very big websites. One of the difficulties of static site generation is the management of the generation of content in real time. Workarounds exist like making exist a hugo in page mode and hugo in site mode. I think the incremental buid could solve this problem.

@bep bep added Keep and removed Stale labels Jul 1, 2017

@bep

This comment has been minimized.

Copy link
Member

bep commented Oct 13, 2017

As I suspect some people will arrive here from #3962 -- and ask: "That 4 point list from @spf13 looks simple enough, why not implement that instead?"

Because this is a much harder problem. And it would be good if @spf13 could update his description to match that reality. He may be correct saying that this is a "well-understood problem" in the general sense of it, but that is like saying "so, Java and C++ have Generics, why haven't Go added support for it?"

The core of the problem comes from the freedom we give to the template developers (you): We give you a complete site object graph to play with for every template, which makes the "is this page stale?" questions harder to answer than you might think (think where clauses etc.). Taking that freedom away isn't an option, and it isn't obvious (to me at least) how to track that outside of the templates.

While still incredibly hard, this is easier and more possible now compared to when this issue was created. Back then there was no unified object graph. Now it is: The root is the Site. But it is still very difficult, and I challenge any person claiming otherwise to describe his or her solution in deeper detail and/or create a working Proof of Concept in Hugo.

@bep

This comment has been minimized.

Copy link
Member

bep commented Oct 13, 2017

OK, so all of this rubber ducking actually helps. I think I may have wrapped my head around the hardest problem. I'll write down the gist of it so I don't forget.

In Hugo, we have done two recent improvements in the template area that is unrelated, but may help:

  • We have an AST transformation of the templates to help with the total .Params.lowercase confusion.
  • We have namespaced all the template funcs. They now all have a pointer receiver.

Common for all of the template executions is that they start with a template and *Page pair.

We should be able to inject a proxy in the templates so we can associate the *Page with the data it receives.

There is still a lot of work in the above, but now it at least looks possible.

/cc @budparr @spf13

bep added a commit that referenced this issue Oct 14, 2017

Only re-render the view(s) you're working on
Hugo already, in its server mode,  support partial rebuilds. To put it simply: If you change `about.md`, only that content page is read and processed, then Hugo does some processing (taxonomies etc.) and the full site is rendered.
This commit covers the rendering part: We now only re-render the pages you work on, i.e. the last n pages you watched in the browser (which obviously also includes the  page in the example above).

To be more specific: When you are running the hugo server in watch (aka. livereload) mode, and change a template or a content file, then we do a partial re-rendering of the following:

* The current content page (if it is a content change)
* The home page
* Up to the last 10 pages you visited on the site.

This should in most cases be enough, but if you navigate to something completely different, you may see stale content. Doing an edit will then refresh that page.

Note that this feature is enabled by default. To turn it off, run `hugo server --disableFastRender`.

Fixes #3962
See  #1643
@Yajo

This comment has been minimized.

Copy link

Yajo commented Sep 29, 2018

Why should we restrict incremental builds to watching mode?

Wouldn't it be even better if Hugo could compile static sites incrementally too?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment