Performance problems with large wiki, with long history and large pages

[felixwellen]

I tried to use gollum (latest master) with this repo:

https://github.com/ncatlab/nlab-content

but it is somewhat too slow to be usable:

• Browsing the directory structure takes 2-3 seconds per click
• Loading small pages takes 5-10 seconds each time

I guess the second problem could be solved by caching (I don't mind if there is some wating time after updating a page) and essentially amounts to finishing up this:

• Persist cached formatted_data gollum-lib#172
• Add caching adapter gollum-lib#77

For the first problem I only have the wild guess, that it might come from the long history (>100k commits), since it also seemed to cause problems here:

• Extremely slow page loads on large wiki? #1078

Closing with some details of my setup, just in case that plays a role:

• I'm running on a fedora 37 with recent hardware and lot's of ram
• On startup, gollum tells me (twice), it is ignoring some gems, since they haven't been built with native extensions:

Ignoring psych-5.0.1 because its extensions are not built. Try: gem pristine psych --version 5.0.1
Ignoring racc-1.6.2 because its extensions are not built. Try: gem pristine racc --version 1.6.2
Ignoring sassc-2.4.0 because its extensions are not built. Try: gem pristine sassc --version 2.4.0
Ignoring stringio-3.0.4 because its extensions are not built. Try: gem pristine stringio --version 3.0.4
Ignoring unf_ext-0.0.8.2 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.8.2

[felixwellen]

I managed to make the Ignoring ...-messages go away. If there is any speedup, it is barely noticeable.
Really small pages with only a couple of lines actually take only 1 or 2 seconds. Looking at the history of a page takes a long time, even if it does not have many entries.

[felixwellen]

Maybe I should add, that the nlab-content repo has a weird structure from a wiki-standpoint, since, so far, it is only used as a backup method (for this wiki https://ncatlab.org/). In particular, it has a deep folder structure, the folder names are number, each page is called content.md and all the internal links do not work as they are.

[dometto]

Thanks for letting us know, and for providing a test repo. See gollum/gollum-lib#437 for an explanation of what's causing the poor performance. Note: the fix there only takes care of the page load times, the Overview logic still uses the slower complete tree map approach.

We could ultimately combine the approach in gollum/gollum-lib#437 with caching, but I think caching is less important than improving our logic at this point!

[dometto]

I should also say: any feedback and help is appreciated!

[felixwellen]

Many thanks for looking into this!
I haven't figured out yet, how I can use your PR. Is that documented somewhere?