Create checksum from more reproducible artifact

[est31]

PR #6317 has been quite disruptive (you can call it detrimental) to my attempts at building a reversable storage of cargo crates.

The goal of the project is to deduplicate files present in multiple versions of the crate and do the compression on the fly. This gives sweet improvements. The reason why #6317 was so disruptive is that I want to replicate the compression 100% so that there are no changes in Cargo.lock hashes. But for this I need to be able to run the the exact compression algorithm used. Now there are countless versions of the zlib library used in various OSs and I'm not sure whether all of them are even open source. That's why I said detrimental, as if proprietary versions of zlib are used, the project would be impossible.

A short term fix for the issue would be to revert the PR. A more long term fix would be to use a different format for the hashes used in the registry and Cargo.lock. Instead of hashing the entire tar.gz files, you could hash the .tar files only. There is a new upcoming version of Cargo.lock in which it could be adopted. Do you think that this is a good idea?

[link2xt]

Instead of hashing the entire tar.gz files, you could hash the .tar files only.

Another option is to hash the files, write the hashes into the manifest and sign the manifest itself. This is how JAR file signatures work. This way even the .tar format can be replaced with a different format later.

[est31]

There is a large number of security vulnerabilities around that, and the jar model is generally seen as bad now in the security community.

[Eh2406]

For my own edification, where can I learn more about these security problems?

[est31]

Having done a small search I found:

• "MasterKey" vulnerability Android bug 8219321 (CVE-2013-4787), discovered by Bluebox security in 2013, article 1, article 2, black hat talk, was about apks allowing multiple entries with the same name, and the signature verifier and the extraction code taking two different ones.
• Various unnamed vulnerabilities (one, two).
• Janus vulnerability (CVE-2017–13156), discovered by GuardSquare. The version 2 signatures and above are not vulnerable to it, but version 1 is (disallowed completely since Android 11). versions 3 and 4, which came later, add some refinements like key rotation or better streaming support.

Right now cargo/crates.io does it really well with hashing the entire tar.gz file's contents. My suggestion is to only use the tar file's contents for the hashing, allowing for better compression and deduplication to be done on the backend side. If we ever want to do signatures embedded in the tar.gz, some alternatives

• via duplicate tarring, so having a .tar.gz file comprised of two files: signature.toml and files.tar, where signature.toml contains a textual representation of the raw bytes of the files.tar file. one needs to ensure that the outer .tar.gz file does not contain any files beyond those two, say files/foo/bar, which would then be added upon extraction.
• one could also add signature.toml to the tar file itself as the very last file, the signature being computed from the tar file's contents up to the signature.toml entry, but this is a bit more dangerous as it is more complex and one needs to forbid any content after the signature.toml entry.

[epage]

This might benefit #2526 as it would let us change migrate to additional compression algorithms while the checksum stays stable across all of them.

[Eh2406]

I was reminded of this at PackagingCon, and wanted to write it down before it slipped my mind. There is an existing battle hardened (file system|operating system|compression) agnostic file structure hash, the hashing algorithm used for git trees. It should be possible to compute this hash over any compressed version of the file or a checkout of the tree on the file system and get the same result. It does treat several filesystem properties as irrelevant and hashes other properties either of which could be wrong decisions in our use case. But it has the advantage of being battle hardened. Just a thought.