Native Nix store implementation / standardized REST API

[roberth]

Both cachix and attic implement their own push-related endpoints, that are, for good reason, incompatible with Nix's own HTTP binary cache store. This is not so great for interoperability.

What does the attic API look like? Is it compatible with the cachix API?
Would you be interested in working towards a native HttpBinaryCacheStore subclass in Nix?

Some benefits of a store implementation:

• No need to install an extra package in the CIs like GitHub Actions
• Automatic support in deployment tools that use nix copy
• Support for queries like nix path-info --store

[zhaofengli]

For pushing, Attic has two main endpoints:

• _api/v1/get-missing-paths which takes a list of store path hashes and returns a list of missing hashes for a given cache
• _api/v1/upload-path which takes both narinfo (in a custom header) and the NAR stream (as the PUT payload)
  • Currently there isn't a separate endpoint to upload only a NAR like in Cachix, because narinfos in the database basically grant the local cache access to the NAR in the global cache.
  • If the NAR hash already exists in the global cache, the server only hashes the stream and silently discards the data. Finally, it adds both the global NAR (if it's a new NAR) and the local narinfo to the database.
  • In the future, the server may decide to return early without receiving the entire NAR stream, if the NAR hash already exists and proof of possession is disabled or the NAR hash is reachable from some other narinfo in the cache (i.e., the cache has already proven possession of the NAR in the past).

I agree about the native HttpBinaryCacheStore and am working to implement it in Attic. It will involve adding separate endpoints to upload NARs (PUT /nar/$fileHash.nar) and narinfos (PUT $storePathHash.narinfo) separately. However, the client must use ?compression=none (similar to nix-casync and spongix), because the server may transform the uncompressed NAR in all sorts of ways (#5) to achieve more efficient deduplication. Compression is handled server-side, and the global NAR table is keyed to uncompressed NAR hashes.

Edit: Regarding improvements to the existing HTTP binary cache protocol, it would be nice to have two things:

• Support an endpoint to query for missing store paths instead of HEAD-ing everything in the closure.
• Also inform the server about the associated narinfo in a header while it's uploading a new NAR.

[lheckemann]

Support an endpoint to query for missing store paths instead of HEAD-ing everything in the closure.

It would also be nice to have a way to query the narinfos for the whole closure of a path, which -- while less efficient than the missing-store-paths approach -- still allows massively reducing the number of requests necessary for fetching a closure and can still be served by a static HTTP server. Something along the lines of having an optional ys4dl9p8f47b6lcl9j4ss7x9dfyfw1x2.closure.narinfo in addition to the plain narinfo, which contains the information for the whole dependency closure of the path.