Checksum of yanked version causing crate download to fail

[tomb-msft]

Problem

The package ndk-sys has a latest version 0.4.0 and a more recent yanked version 0.4.0+25.0.8775105. Cargo is downloading the 0.4.0 crate, but comparing the hash with the checksum of the yanked version, and therefore failing with a "failed to verify checksum of ndk-sys v0.4.0" error.

It seems to be a sporadic issue - sometimes the download will complete successfully, sometimes it will fail due to the checksum not matching.

This happens even though the lockfile has the correct version and checksum for v0.4.0.

Checksum of 0.4.0: 21d83ec9c63ec5bf950200a8e508bdad6659972187b625469f58ef8c08e29046
Checksum of 0.4.0+25.0.8775105: f74ddd54b7da8d38d399faf43472ac9759f1a028a45c83154bff603e0f56385a

Steps

1. Create a new binary package with cargo init.
2. Add ndk-sys v0.4.0 to the package's dependencies:

   [dependencies]
   ndk-sys = "0.4.0"
   

3. Clear your Cargo cache:

   cargo install cargo-cache && cargo cache --remove-dir registry-crate-cache
   

4. Fetch the dependencies for the new crate:

   cargo fetch
   

5. Cargo will (sometimes) fail with the error:

   error: failed to verify the checksum of `ndk-sys v0.4.0`
   

6. Verify that the lockfile has version 0.4.0 and checksum 21d83ec9c63ec5bf950200a8e508bdad6659972187b625469f58ef8c08e29046

You might need to repeat steps 3 and 4 a few times.

Possible Solution(s)

No response

Notes

As far as I can tell it's not a crates.io issue. The download can fail multiple times then sporadically start working without a crates.io index update, and with identical .crate files being downloaded (checked with a proxy).

I added some logging to Cargo which was how I validated it was using the checksum of the yanked version:

diff --git a/src/cargo/sources/registry/download.rs b/src/cargo/sources/registry/download.rs
index cc39d7c11..6f35fb9e1 100644
--- a/src/cargo/sources/registry/download.rs
+++ b/src/cargo/sources/registry/download.rs
@@ -84,6 +84,10 @@ pub(super) fn finish_download(
 ) -> CargoResult<File> {
     // Verify what we just downloaded
     let actual = Sha256::new().update(data).finish_hex();
+
+    println!("actual sha256: {}", actual);
+    println!("expected sha256: {}", checksum);
+
     if actual != checksum {
         anyhow::bail!("failed to verify the checksum of `{}`", pkg)
     }

Version

cargo 1.65.0 (4bc8f24d3 2022-10-20)
release: 1.65.0
commit-hash: 4bc8f24d3e899462e43621aab981f6383a370365
commit-date: 2022-10-20
host: x86_64-unknown-linux-gnu
libgit2: 1.5.0 (sys:0.15.0 vendored)
libcurl: 7.83.1-DEV (sys:0.4.55+curl-7.83.1 vendored ssl:OpenSSL/1.1.1q)
os: Ubuntu 20.04 (focal) [64-bit]

[epage]

So ndk-sys has

• 0.4.0
• 0.4.0+25.0.8775105

Everything after + is build metadata. semver says

Build metadata MAY be denoted by appending a plus sign and a series of dot separated identifiers immediately following the patch or pre-release version. Identifiers MUST comprise only ASCII alphanumerics and hyphens [0-9A-Za-z-]. Identifiers MUST NOT be empty. Build metadata MUST be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence. Examples: 1.0.0-alpha+001, 1.0.0+20130313144700, 1.0.0-beta+exp.sha.5114f85, 1.0.0+21AF26D3—-117B344092BD.

Interesting, semver doesn't say the two versions are the same. I do find it surprising that crates.io allows you to upload two of the same "version" and I suspect this is what is confusing cargo.

[ehuss]

This is a known issue with crates.io (rust-lang/crates.io#1059). It should not be allowing publishes with versions that only differ in metadata. Fixing that is a little tricky, unfortunately.

[eminence]

There is some code in registry/index.rs that seems to deal with this situation. But I'm not familiar enough with cargo's inner workings to know if this code is insufficient, or if it just needs to be applied elsewhere.

[MarijnS95]

For completeness I described the same issue and posted the publishing+yanking steps for ndk-sys here: #7180 (comment).

[MarijnS95]

If I'm not mistaken the origin of this issue is now fixed, no longer allowing one to publish the same version with only metadata differences: rust-lang/crates.io#6518

However, this issue might linger for some time (e.g. if explicitly referencing the duplicate-but-yanked =0.4.0 version?).