Docfx generated contents are not stable across multiple executions

[lextm]

DocFX Version Used: 2.37.2

Template used: (default or statictoc or contain custom template)
Used custom template.

Steps to Reproduce:

1. Use a large enough sample project.
2. Compile it multiple times and compare the generated files.

Expected Behavior:
The files should be the same (as the source project never changes).

Actual Behavior:
Files like index.json and manifest.json seem to always change (items being reordered).

This is very annoying in a CI/CD system, because such changes can trigger changes sensitive actions configured, while in fact the actions should not work at all.

[herohua]

@lextm , thanks for reporting the issue. Is ordering the only thing that changes from run to run? Could you help attach two manifest.json (or index.json) for us to analyze?

[lextm]

I attached two revisions for you to review,
help1.zip
help2.zip

[airbreather]

I was noticing the same thing all the time, NetTopologySuite/NetTopologySuite.IO.GPX@d82241e is an example.

After NetTopologySuite/NetTopologySuite.IO.GPX@a407093, the next 3 clean rebuilds in a row gave me the exact same content.

[airbreather]

NetTopologySuite/NetTopologySuite.IO.GPX@0e5d0c0 might be necessary too.

[lextm]

In fact, there are other instability issues more complicated than this one. I noticed that for the same method, sometimes a parameter is generated correctly, and sometimes not. But as that's intermittent, I will report back once I find a way to constantly reproduce it.

[superyyrrzz]

@airbreather A small tip: you can call docfx docfx.json -f and the caches can be ignored automatically, without the need to remove them in scripts.

[lextm]

@airbreather I reviewed the workaround you provided.

• Forcing docfx to run without parallel execution can lose too much performance.
• Clearing caches is a waste of resources.

A simpler workaround might be to write a custom script to reorder the items as post processing.

It would be responsibilities of the developers to revise their code base to fix the ordering stability issue.

For example, parallel execution might generate items randomly in cache, but the generation of manifest.json and/or index.json should be changed to reorder the items (or at least give the users an option to reorder the items).

[airbreather]

@superyyrrzz that almost gets me what I need, but there's still the possibility of NetTopologySuite/NetTopologySuite.IO.GPX@049f086 showing up depending on whether or not I've done a build since the last time my working tree was clean. Thanks, though.

@lextm I don't disagree. I just started using DocFX a few days ago, and I found a simple workaround that works for me today (at a cost, of course), so I wanted to share it in case it helps someone in the meantime.

[yufeih]

We have updated docfx to produce deterministic output for the majority of the cases and used it to publish our own docs to github pages incrementally. There could be cases we haven't covered yet, if you encountered indeterministic behaviors, create a new bug with a minimal repo so we could take a look and fix.