@mitsuhiko - Thanks for fixing it. Could you please make a patch release soon?
This issue is affecting other Sphinx projects, e.g. the sidebar from the Astropy docs disappeared recently and it took me some time to find this issue: astropy/astropy#5679 (comment)
It's particularly nasty since there's no warning or error from Sphinx, but the generated HTML is corrupted.