Optimization on Scaling Site for 50K pages

[AdamFermier]

Hi Folks!

We are expanding / exploding the content on our site to project ~50K pages (programmatically created) - when we do it - we're hitting some "build time issues" and from what I read with this thread - there's ways of doing it with Hugo Caching: https://discourse.gohugo.io/t/building-a-site-with-50-000-pages-taking-a-very-long-time/43037/8

Here's the readout of our builds - can you recommend where I should / could do the caching as to help on the themes ability to scale?

Any and all help useful - thanks! Adam

cumulative	average	maximum	cache	percent	cached	total
duration	duration	duration	potential	cached	count	count	template
1m18.7160868s	11.675µs	268.8927ms	100	0	0	6741782	partials/menu-pre.html
1m16.0059684s	11.273µs	156.6573ms	100	0	0	6741782	partials/menu-post.html
2.205538s	254.651µs	44.5732ms	100	0	0	8661	partials/output-format.hugo
2.1791429s	280.6µs	99.6735ms	100	66	5164	7766	partials/output-suffix.hugo

[McShelby]

This kind of optimization needs to be built into the theme. See #685.

Having a caching potential of 100% may mean that it is cacheable for your project. The theme runs with many different configurations and there may be constallations out in the wild were caching leads to unwanted results.

Anyways I suspect menu-pre and menu-post are only the tip of your iceberg, as I expect other templates to be far more costly.

[AdamFermier]

Any recommendations how to proceed - I did replace some partial with partialCache and led to the unwanted results :) I'm not even sure what the menu-pre and menu-post are doing - any insights to help me learn would be great!

Thanks for all your tireless efforts! Adam

[McShelby]

Well, partials are at least used for two main cases:

• to DRY the code when doing repetitiv stuff
• to offer extension capabilties to the user

Sometimes a partial does both things.

The latter kind of partials are documented here. menu-pre and menu-post are both of that kind. They allow the theme user to modify menu entries in a global way, eg. by adding icons in front of the text.

Those templates usually take the current page as its context parameter and that is the reason why you can't cache the results inside of the theme, as the output of the partial may or may not differ per the current page.

Most of those partials are initially empty. So they will result in no output regardless of the current page. That's the reason why the profiler suspects them as 100% cachable. But that's just an assumption on behalf of your site's content. Other users may have overridden the partial and then it becomes non-cachable. The theme must account for the worst, so the partial is non-cachable.

If, in your case, the partial always creates the same output regardless of the current page, then - theoretically - you could call it with partialCached instead of partial or even remove the call completly if the partial doesn't generate content at all. But this would mean, you would have to change various files all over the theme, effectively resulting in a fork of the theme as you are now loosing an easy upgrade path in case the theme applies changes in your edited files (most likely resulting in merge conflicts).

BUT: Your profiler output is incomplete. From my experience menu-pre and menu-post aren't the heavy hitters here. The main consumption of time happens somewhere else and #685 gives you the hint why that is. Fixing this isn't easy. That's the main reason why it's still unresolved.

For now you are left with the following options:

• wait till theme: reduce build time #685 is fixed (to be honest, I am not quite sure how good performance can get here, as the runtime complexity may not be avoided to a certain degree)
• edit the theme extensivley (I still doubt that this will improve performance significantly)
• use a different theme (that's a tough one even for me)

Sorry, that I can not offer better options at the moment.

[McShelby]

If #685 is fixed, performance might get significantly better if you are avoiding big menus by utilizing alwaysopen=false and collapsibleMenu=false. Sadly, while having the same visual outcome, it currently does not help in reducing built time.

[AdamFermier]

Oh my goodness gracious! Thanks so much @McShelby - I did not see this till now - we'll try to build it again and report on our results :)