Remove non-determinism

[mitya57]

To enable packages using Sphinx to build reproducibly, its output needs to be the same from one build to another.

Its output now strips memory references such as:

<__main__.A at 0x7f68cb685710>

In addition, various generated files (objects.inv, searchindex.js, translations) are now written with their keys in a determinstic order.

Initial patch by @lamby, rebased against master by @mitya57.

[birkenfeld]

sorted(self.env.domains) should suffice.

[birkenfeld]

Good idea, apart from removing non-determinism it also makes the repr's much more readable.

[mitya57]

Updated.

[birkenfeld]

LGTM, just waiting for the tests to complete.

[lamby]

Thanks for merging and apologies for some of the—err—rough edges in the original patch. Mea culpa.

On those lines, I think we are actually missing something. For example, take a look here:

https://reproducible.debian.net/rb-pkg/sphinx.html

... and grep for "genindex.html". Note that the command line option ordering is different (-q vs -Q). I had a quick glance at the Sphinx source to work out where this could be remedied but it was not immediately forthcoming.. More than happy to look further into this if nobody else immediately knows.

[lamby]

Did anyone have any thoughts re. making the regex "safer"? I don't like how it's applied to every output of repr, but wasn't particularly convinced by a whitelist of isinstance params when I tried that..

[birkenfeld]

I think it is quite safe. We're not generating code here, so the very unlikely false positive will be much less important than the better readability for 99%+ of instances.

[lehmannro]

If you wanted to limit it you could only apply memory_address_re when type(obj).__repr__ is object.__repr__. This falls down when you're handling collections which recursively call repr (eg. lists) or user-defined objects which indeed include id(self) somewhere.

[mitya57]

@lamby The code that generates index is in create_index method in sphinx/environment.py. The code that writes it into genindex.html file is in write_genindex method in sphinx/builders/html.py. Hope that helps.

(For everyone else: the diff @lamby was referring to is here).