Builds are very slow with lots of pages

[gilbertbw]

Describe the bug

context
When I do build our project (a fairly large set of help, with 1300 topics) the build is taking much longer than with the '' theme. * * With 'alabaster' the build takes under 2 minutes.

• With 'sphinx_book_theme' the build takes over 10 minutes.

expectation
I expected the build to be not much slower than 'alabaster'

bug
But instead the build takes more than 5 times longer.

problem
This is a problem for people doing builds of large projects because the build time is so long.

Reproduce the bug

1. Use a project with many topics
2. Build the project

List your environment

I am not using jupyter-book

OS: Windows 11 Enterprise 22H2 OS Build 22621.1

sphinx 4.5.0
sphinx-book-theme 0.3.2

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[mmcky]

@gilbertbw I am not sure why the theme would affect these build times.

Is this project public (i.e. is there a link the repo?)

@AakashGfude any ideas here?

[gilbertbw]

@gilbertbw I am not sure why the theme would affect these build times.

After a bit of digging I have found this issue pydata/pydata-sphinx-theme#381 which I suspect is the cause, I will try the mitigations they propose and see if that leads to a significant speed up.

Is this project public (i.e. is there a link the repo?)

No, this is a private project

[avivajpeyi]

I am running into the same issue! I have 5000+ pre-executed notebooks (trying to make a catalog of exoplanets), and would like to use the sphinx-book-theme to build webpages.

I was initially using jupyter-book but decided that maybe using sphinx-build with the sphinx-book-theme might be faster.

Happy to make a small working example -- uploading the 5000+ pages might be a bit challenging! Will share an example shortly.

[avivajpeyi]

Here is a really small working example (only a few files). To build the HTML pages, cd into that zipped dir and use sphinx-build -b html . _build -E

To generate extra notebooks (demonstrating the slow speed), you can duplicate the files in content/toi_notebooks/toi_....ipynb. On duplicating such that there are ~35 files, I notice that the runtime on avg is 5min16 sec.

mwe.zip

Interestingly -- when I switch the theme to alabaster the runtime is still quite slow (4min58sec). I don't get the same speedup as @gilbertbw

[gilbertbw]

I tried

html_theme_options = {
  ...
  "collapse_navigation": True,
  ...
}

This had no effect. I realised this was because the config option is not available in sphinx-book-theme. Instead I modified
sphinx_book_theme\theme\sphinx_book_theme\components\sbt-sidebar-nav.html to collapse=True. This reduced the build time to around 2 minutes.

A more important issue with writing the full document tree into every page is that each page ends up around 0.25 MB. This is with around 1500 help topics. The whole help ends up around 380 MB, compared to around 3 MB for the .chm file this is replacing.

I wonder if it would be possible to replace the sidebar being repeated on every page with an <iframe/>?

[avivajpeyi]

@gilbertbw -- interesting!
I got the following error when testing with collapse=False

preparing documents... WARNING: unsupported theme option 'collapse' given

[pradyunsg]

I wonder if it would be possible to replace the sidebar being repeated on every page with an <iframe/>?

The sort of reuse you're hoping for is not possible without enforcing some sort of JS at runtime requirements -- each page has a slightly different markup there, since the "current page" classes in the various nodes are different.

[choldgraf]

I believe that the core challenge here is that when we include all of the site's page links in the sidebar, and when there are many sidebar links, then Sphinx takes a long time to resolve all of the cross refs.

• This section in the PyData theme may have suggestions relevant to solving this

I think that we could add more documentation in this theme that cross-links to https://pydata-sphinx-theme.readthedocs.io/en/stable/user_guide/configuring.html#navigation-depth-and-collapsing-the-sidebar but other than that I don't think there's much we can do to speed up site builds if you also want all of the sub-pages in the navigation bar.

[gilbertbw]

I may be going off topic for this issue now, but for us the build size is a blocker, rather than the build time. We ship our help with our software, and can not take up this amount of space!

I wonder if it would be possible to replace the sidebar being repeated on every page with an <iframe/>?

The sort of reuse you're hoping for is not possible without enforcing some sort of JS at runtime requirements -- each page has a slightly different markup there, since the "current page" classes in the various nodes are different.

@pradyunsg I have a proposal that I believe works without JS. We prepopulate the sidebar with a unique class for each <li> in the tree. Then when building each documentation page, add a <style> tag that shows and colours the relevant links and lists.

e.g. with a tree like this, where the class is on the <li>

• Top 1 class="t1"
  • Top 1 Child 1 class="t1c1"
  • Top 1 Child 2 class="t1c2"
• Top 2 class="t2"
  • Top 2 Child 1 class="t2c1"
  • Top 2 Child 2 class="t2c2"
  • Top 2 Child 3 class="t2c3"
    • Top 2 Child 3 Child 1 class="t2c3c1"
    • Top 2 Child 3 Child 2 class="t2c3c2"
    • Top 2 Child 3 Child 3 class="t2c3c3"
• Top 3 class="t3"
  • Top 3 Child 1 class="t3c1"

on the page Top 1 add a

<style>
#site-navigation .t1>a { color: rgba(var(--pst-color-link),1); }
#site-navigation .t1>ul { display:block; }
</style>

or on Top 2 Child 3 Child 2 add

<style>
#site-navigation .t2>a { color: rgba(var(--pst-color-link),1); }
#site-navigation .t2>ul { display:block; }

#site-navigation .t2c3>a { color: rgba(var(--pst-color-link),1); }
#site-navigation .t2c3>ul { display:block; }

#site-navigation .t2c3c3>a { color: rgba(var(--pst-color-link),1); }
</style>

---

I believe that the core challenge here is that when we include all of the site's page links in the sidebar, and when there are many sidebar links, then Sphinx takes a long time to resolve all of the cross refs.

• This section in the PyData theme may have suggestions relevant to solving this

I think that we could add more documentation in this theme that cross-links to https://pydata-sphinx-theme.readthedocs.io/en/stable/user_guide/configuring.html#navigation-depth-and-collapsing-the-sidebar but other than that I don't think there's much we can do to speed up site builds if you also want all of the sub-pages in the navigation bar.

@choldgraf When testing this I tried #561 (comment) setting "collapse_navigation": True. It did not work, I assume because it is hardcoded to False here

sphinx-book-theme/src/sphinx_book_theme/theme/sphinx_book_theme/components/sbt-sidebar-nav.html

Line 19 in 357c659

collapse=False,

---

@gilbertbw -- interesting! I got the following error when testing with collapse=False

preparing documents... WARNING: unsupported theme option 'collapse' given

@avivajpeyi Sorry for the confusion. I mean I literally opened up ...\virtualenv\Lib\site-packages\sphinx_book_theme\theme\sphinx_book_theme\components\sbt-sidebar-nav.html on my computer and changed collapse=False to collapse=True.

[choldgraf]

@gilbertbw for complex / large Sphinx sites, I'd recommend giving this tool a shot: https://github.com/executablebooks/sphinx-remove-toctrees#install

maybe that'll help

[choldgraf]

Closing this one as it's been resolved upstream, and now has documented workarounds

[avivajpeyi]

Hey @choldgraf -- could you point me to the documented workarounds?

This page from your comment seems to be broken:
https://pydata-sphinx-theme.readthedocs.io/en/stable/user_guide/configuring.html#navigation-depth

Im still having some trouble for my project (may need 5000+ pages 😅 )
http://catalog.tess-atlas.cloud.edu.au/content/toi_notebooks/toi_101.html

[choldgraf]

Yep - check out the docs about that here:

https://pydata-sphinx-theme.readthedocs.io/en/stable/user_guide/performance.html

[avivajpeyi]

Another change that helped me in the +5000 ipynb case was removing all matplotlib-inline plots and instead displaying images using markdown cells (e.g. with ![]()). This can drastically reduce the size of the ipynbs (depending on the number of images within an ipynb).