`ca-derivations`: pushing a second build traces to the same s3 based cache is destructive

[mschwaig]

Describe the bug

I am not sure if we are switching to the term build traces, but I am using it in this post.

Since today it is not possible to make Nix only trust a build trace when multiple parties agree about how it should be resolved, this is not a bug.

Eventually I would like it to be possible to aggregate information about realisations across various builders.
One way to enable this is to feed a central cache from a number of builders, which all add additional information.

Currently when a second builder builds something with the same resolved derivation hash, the second builder writes over the metadata and signatures that the first builder uploaded. This is the case at least when using nix copy to fill an s3-based cache.

Steps To Reproduce

I made a branch in my repo, where I can show this issue by filling a cache from two builders, and looking at the relevant realisation/ folder. When the test is run interactively this can be obtained from a web-ui. Non-interactively won't work for this, because the data we care about is only a side-effect of the test.
See commit below for credentials and details:
mschwaig/laut@7e1e987

1. nix build github:mschwaig/laut/7e1e987e68f531f85d4fb10a748cb1d9171427cb#checks.x86_64-linux.fullReproVM.driverInteractive
2. ./result/bin/nixos-test-driver --interactive
3. Go to the minio web ui at port 9001 and download the relevant folders from there
4. You will see that only one signature ever makes it into each realisation file, even though two builders are feeding the cache

Instead of running this yourself, you can also take a look at the attached .zip file:
s3-realisation-upload-bug.zip

(I also still have a similar problem in my own project, where the first ever uploaded trace also only has one signature.)

Expected behavior

IMO aggregation of this data should be supported somehow, though implementing this on the builder-side, where each builder introduce updates that deliberately erase signatures by other builders, is also not ideal for in a distributed trust setting.

Additional context

What I describe in here is not the worst of bugs, and not core to getting CA derivations to work at all, but IMO it's central to support something like this to support how CA derivations can help distribute trust across builders.

I'm trying to keep track of a bunch of issues like this based on what I need for my own project at mschwaig/laut#12. My plan is to only post the ones here where I have gained confidence that they are relevant enough to bring up here, and not just polluting the issue tracker. Let me know when I miss the mark.

---

Add 👍 to issues you find important.