Building HTML with sphinx_rtd_theme is very slow with a huge top-level page

[ulfalizer]

Describe the bug

Building HTML documentation with the sphinx_rtd_theme is very slow (over a second for an empty page for us) when the top-level page is huge. The top-level is page similar to this one, which is generated by this script.

Likely cause of bug

sphinx_rtd_theme/layout.html has this, which is used for the <div> with the navigation menu on the left. It is based on the top-level TOC tree.

<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
  {% block menu %}
    {#
      The singlehtml builder doesn't handle this toctree call when the
      toctree is empty. Skip building this for now.
    #}
    {% if 'singlehtml' not in builder %}
      {% set global_toc = toctree(maxdepth=theme_navigation_depth|int,
                                  collapse=theme_collapse_navigation|tobool,
                                  includehidden=theme_includehidden|tobool,
                                  titles_only=theme_titles_only|tobool) %}
    {% endif %}
    {% if global_toc %}
      {{ global_toc }}
    {% else %}
      <!-- Local TOC -->
      <div class="local-toc">{{ toc }}</div>
    {% endif %}
  {% endblock %}
</div>

That ends up unpickling the doctree for the top-level page, like this:

...
File "/usr/local/lib/python3.7/dist-packages/sphinx_rtd_theme/layout.html", line 345, in block_menu
File "/usr/local/lib/python3.7/dist-packages/jinja2/sandbox.py", line 440, in call
  return __context.call(__obj, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/jinja2/runtime.py", line 262, in call
  return __obj(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/sphinx/builders/html.py", line 1003, in <lambda>
  ctx['toctree'] = lambda **kw: self._get_local_toctree(pagename, **kw)
File "/usr/local/lib/python3.7/dist-packages/sphinx/builders/html.py", line 898, in _get_local_toctree
  docname, self, collapse, **kwds))['fragment']
File "/usr/local/lib/python3.7/dist-packages/sphinx/environment/adapters/toctree.py", line 308, in get_toctree_for
  doctree = self.env.get_doctree(self.env.config.master_doc)
File "/usr/local/lib/python3.7/dist-packages/sphinx/environment/__init__.py", line 783, in get_doctree
  doctree = pickle.load(f)

The unpickling is super slow, since the top-level page is so large.

Possible fixes

Not sure if this could be fixed in any other way than by caching the top-level TOC tree. I can see if you'd consider that overkill. I just wanted to document the issue at least.

Other suggestions are very welcome too. I'm not a Sphinx expert.

Your project

Internal project, but the documentation is similar to the page generated above. It's generated by the same script.

For the linked page, things work out due to that page not being the top-level page, but no such luck for the internal project.

Environment info

• OS: Ubuntu 19.10
• Python version: 3.7.5
• Sphinx version: 1.7.5