Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expandable sidebar sections with huge sphinx sites (eg large API reference) results in a very slow build #364

Closed
jorisvandenbossche opened this issue Apr 2, 2021 · 10 comments

Comments

@jorisvandenbossche
Copy link
Member

jorisvandenbossche commented Apr 2, 2021

Now that the collapsible sections in the sidebar are in master, I am noticing that this gives some challenges in the pandas docs, and especially the API reference part.

If you have a huge site with many pages, the change in #349 has the consequence that each page is now included in the toctree (we use collapse=False now), which means that 1) the build is much slower and 2) the size of the html build artefact is much bigger.

Specifically for the pandas API docs (https://pandas.pydata.org/docs/dev/reference/index.html), we have more than 1000 API pages, and since the docs are built with #349 being merged, the build time went from ~15min to ~55min and the build artifact from 180Mb to 1.1GB. The html of a single page now contains 15,000 lines for the sidebar ..

(aside: we have too many API pages in pandas, as some are identical (duplicated) pages for shared methods accross subclasses, this is something we can improve and deduplicate. But even with less API pages, you will still notice the difference)

Brainstorming some ways this issue could be addressed (in general, and for pandas specifically):

  • Speed-wise: could we cache some results? Since each page shares almost the same html, but not exactly (the current/active page is different), I am not sure this is possible
  • The fact that autosummary automatically adds those pages to the toctree, so they end up in the sidebar, is IMO not really necessary (eg see https://pandas.pydata.org/docs/dev/reference/io.html, all functions are already listed on the page, so that they are also included in the sidebar is a bit duplicative, and it makes the sidebar also very long). But I am not sure how avoid this using sphinx.
  • We could set collapse=True specifically in the API reference section of the pandas docs. Pandas could do this by overriding the template and depending on the pagename set collapse to True or False. Or would there be another way to configure this?
@choldgraf
Copy link
Collaborator

Could you make the api pages orphans, and then manually link to them in the toctree?

@ChaiByte
Copy link
Contributor

ChaiByte commented Apr 5, 2021

Ref: #321

Users have could gain clear API information from the summary table in the main part. The automatically generated subsections in the side-navbar are a bit needless. Now any :toctree: in the .rst file can be captured. It would be a disaster if autosummary and the default :toctree: are used on the same page.

@jorisvandenbossche jorisvandenbossche changed the title Collapsed sidebar sections with huge sphinx sites (eg large API reference) Expandable sidebar sections with huge sphinx sites (eg large API reference) results in a very slow build Apr 14, 2021
@jorisvandenbossche
Copy link
Member Author

I opened #386 to make this configurable, so at least it's easy to turn off expandable navigation in the sidebar in case you don't want it.

In pandas, I solved it for now by turning it off only specifically for the API section of the docs with a small template override: pandas-dev/pandas#40761

But I think the :orphan: approach is also interesting to look into (that would specifically be a solution for #321)

@jorisvandenbossche
Copy link
Member Author

Thinking a bit more about the :orphan: idea to not include generated API stub pages in the toctree: based on my current understanding of sphinx, I think that wouldn't work by itself, because it's still the .. autosummary:: directive that includes a toctree node for those items. So updating the autosummary templates to include :orphan: won't help I think.

In addition, we would need to instruct autosummary not to include such a toctree. That's probably only possible upstream in sphinx itself.

@choldgraf
Copy link
Collaborator

choldgraf commented Oct 8, 2021

I was just playing around with this since the Dask folks just ran into this as well (👋 @jacobtomlinson)

EDIT: I just realized that I was encountering an artifact from previous builds, and my nifty suggestion wasn't so good after all 😬

@choldgraf
Copy link
Collaborator

I think that I have a fix for this here: #492

@gilbertbw
Copy link

Hi,

I have run into this issue with the sphinx-book-theme here executablebooks/sphinx-book-theme#561

Looking into this, I have come up with the following proposal executablebooks/sphinx-book-theme#561 (comment):

I may be going off topic for this issue now, but for us the build size is a blocker, rather than the build time. We ship our help with our software, and can not take up this amount of space!

I wonder if it would be possible to replace the sidebar being repeated on every page with an <iframe/>?

The sort of reuse you're hoping for is not possible without enforcing some sort of JS at runtime requirements -- each page has a slightly different markup there, since the "current page" classes in the various nodes are different.

@pradyunsg I have a proposal that I believe works without JS. We prepopulate the sidebar with a unique class for each <li> in the tree. Then when building each documentation page, add a <style> tag that shows and colours the relevant links and lists.

e.g. with a tree like this, where the class is on the <li>

  • Top 1 class="t1"

    • Top 1 Child 1 class="t1c1"
    • Top 1 Child 2 class="t1c2"
  • Top 2 class="t2"

    • Top 2 Child 1 class="t2c1"

    • Top 2 Child 2 class="t2c2"

    • Top 2 Child 3 class="t2c3"

      • Top 2 Child 3 Child 1 class="t2c3c1"
      • Top 2 Child 3 Child 2 class="t2c3c2"
      • Top 2 Child 3 Child 3 class="t2c3c3"
  • Top 3 class="t3"

    • Top 3 Child 1 class="t3c1"

on the page Top 1 add a

<style>
#site-navigation .t1>a { color: rgba(var(--pst-color-link),1); }
#site-navigation .t1>ul { display:block; }
</style>

or on Top 2 Child 3 Child 2 add

<style>
#site-navigation .t2>a { color: rgba(var(--pst-color-link),1); }
#site-navigation .t2>ul { display:block; }

#site-navigation .t2c3>a { color: rgba(var(--pst-color-link),1); }
#site-navigation .t2c3>ul { display:block; }

#site-navigation .t2c3c3>a { color: rgba(var(--pst-color-link),1); }
</style>

This will require some changes in this repository. I wanted to get a feel for how this would be received if I open a PR:

  1. update generate_nav_tree
    def generate_nav_html(kind, startdepth=None, show_nav_level=1, **kwargs):
    to optionally write unique classes for each page <li>.
  2. add a new toctree function to write out the <style>...</style> to show the current page.

@choldgraf
Copy link
Collaborator

Can you make a clear write-up of the problem that this would resolve? Why does adding a unique CSS class to every sidebar item result in smaller build sizes?

I'd recommend writing this up as a separate issue so that it is easier to discuss the pros / cons of the proposal. In my opinion this original issue can be considered resolved now that we have documentation to get around it here: https://pydata-sphinx-theme.readthedocs.io/en/stable/user_guide/configuring.html#navigation-depth

@gilbertbw
Copy link

@choldgraf as requested, I have written this up (hopefully more clearly) into #762

@choldgraf
Copy link
Collaborator

sounds good, I will also close this one as resolved via documentation: https://pydata-sphinx-theme.readthedocs.io/en/stable/user_guide/configuring.html#navigation-depth

and we can discuss more specific proposals to improve the build times / size in #762 (or other issues)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants