sphinx.search._JavaScriptIndex: non-determinite searchindex.js output

[jayaddison]

Describe the bug

While looking at some recent Reproducible Build test failures, I noticed that one of the remaining differences between builds of quodlibet is due to sorting differences in searchindex.js, a file produced to enable client-side content search in web browsers.

Looking at the relevant code in the sphinx.search._JavaScriptIndex class, the code to output the searchindex.js content is a fairly straightforward Python built-in json.dumps call.

The failures mentioned occurred with Sphinx v5.3.0 -- I believe they would occur when using the latest release v7.2.2 also (the index export code appears unchanged).

I think that a fix may be as straightforward as sending a sort_keys=True argument to the json.dumps call. Writing test coverage to provide varying inputs that sort unequally otherwise could be the tricky part.

cc @lamby

How to Reproduce

I do not yet have a local reproducer for this bug, but the differences in output can be observed in the diffoscope results for quodlibet on Sphinx v5.3.0 here: https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/quodlibet.html

Environment Information

Since the reproducer case I'm reporting from was a hosted build, I can't easily run the environment-reporting commands, but some relevant details are:

Sphinx: v5.3.0
Operating System: Debian Linux (unstable / trixie)
Documentation-package-under-test: quodlibet v4.5.0

Sphinx extensions

extensions = ['sphinx.ext.autodoc', 'sphinx.ext.extlinks', 'contributors']

Additional context

No response

[AA-Turner]

Is sort_keys=True recursive? I.e. would it sort the list under alltitles, which isn't top level?

[jayaddison]

Is sort_keys=True recursive? I.e. would it sort the list under alltitles, which isn't top level?

I believe it is, yep. It won't sort list/array elements, so if we have variance in those, then it won't help - but I think that the problem here is mostly, if not entirely, related to dictionary keys.

[picnixz]

If you are interested in solving this issue, I think it would also be good to study #11578 at the same time.

[jayaddison]

Thanks @picnixz - yep, I'd like to solve this, although am feeling quite slow at the moment. Reading the linked issue now.

[pietroalbini]

We're encountering this in our project too, where we need full reproducibility to digitally sign the output of Sphinx.

@jayaddison are you still intending to work on this? Otherwise I can open a PR later today or tomorrow fixing the issue (I don't want to step on your toes here if you already have a work in progress branch 🙂).

[jayaddison]

Hi @pietroalbini! - thanks for asking, and nope, I haven't started on this (other than to read the code), so please go ahead if you have time 👍

[pietroalbini]

Opened #11665 with the fix to this.

[jayaddison]

Thanks @pietroalbini @AA-Turner! I'll suggest a version-bump to the folks at quodlibet after this has been in place for a release and some bake-in time.