Avoid build.ninja changes due to order of hash table iteration

[bonzini]

The ordering of elements in sets and of keys in dictionaries cannot be relied upon, because the hash values are randomized by Python. Whenever the processing of meson.build or the operation of the backend is affected by this ordering, random changes in the command line can occur and cause ninja to rebuild a lot of files unnecessarily.

This pull request fixes the instances that affect QEMU, and which were causing about 40% of the build to be repeated after touch ../meson.build (most of them due to sourcesets).

[jpakkane]

This should probably use OrderedMap and OrderedSet consistently rather than fixing it later with sorted.

[bonzini]

Sometimes it should (and the patches do), especially if that affects the build command line directly. In fact for sourcesets the pull request is already using OrderedSet. However, I'm not sure that should always be the case.

For dictionaries in meson.build I am afraid of users relying on the iteration order being the same as the assignment order, which I don't think should be guaranteed; it feels wrong that Meson dictionaries are OrderedDicts rather than just associative arrays. In this case, sorting is just the easy answer to the immediate need I had, which is to stabilize the order of iteration. Yes, perhaps dictionaries shouldn't be a SortedDict either but it seemed to me that it's less likely for meson.build to rely on sorting order than on assignment order.

In the remaining cases (depfiles in the first patch and ninjabackend) Meson uses set just to get rid of duplicates but it doesn't need the original order. In these cases I chose to sort just because it produces neater results. It is easier to scan for a missing file if you are debugging meson.build, for example; using an OrderedSet to keep the order didn't strike me as particularly useful. In the case of ninjabackend, in addition, the order of the resulting set would also depend on implementation details of the backend which should not necessarily be visible in build.ninja.

[bonzini]

By the way, there is a precedent for sorting the keys in the output in mesonlib.dump_conf_header. The third patch in this PR merely extends the behavior to (some parts of) build.ninja.

[jpakkane]

I thought about this some more and it is in fact important that we preserve the order given. For example:

executable(, dependencies: [lib1_dep, lib2_dep])

If both lib1 and lib2 are static libraries and 1 depends on symbols from 2, it must come first on the link line at least with GNU ld. Now it could be said that this is wrong as 1 should have a dependency on 2 but sometimes that does not work (especially if those are internal helper libraries).

Now the fact that we have not gotten bug reports about this implies that this is not actually happening in the real world. Still, I'd prefer OrderedSet as doing that fixes this iteration order everywhere, even for code that might get added in the future.

[bonzini]

You're probably thinking of commit "ninjabackend: stabilize order of dependencies and order-only dependencies" which changes the order of ninja dependencies, not Meson dependency objects. That is, in a rule

rule output.o: c_COMPILER output.c | header1.h header2.h || order-only1.in order-only2.in

The header1.h header2.h part and the order-only1.in order-only2.in parts are kept sorted in order to stabilize the output. The order of these does not matter.

The sourceset patch is the only one that deals with Meson dependencies and also the only one that uses OrderedSet, so we should be good.