Fix Cargo removing the sparse+ prefix from sparse URLs in .crates.toml

[arlosi]

The URL associated with a SourceId for a sparse registry includes the sparse+ prefix in the URL to differentiate from git (non-sparse) registries.

SourceId::from_url was not including the sparse+ prefix of sparse registry URLs on construction, which caused roundtrips of as_url and from_url to fail by losing the prefix.

Fixes #11751 by:

• Including the prefix in the URL
• Adding a test that verifies round-trip behavior for sparse SourceIds
• Modifying CanonicalUrl so it no longer considers sparse+ and non-sparse+ URLs to be equivalent

[rustbot]

r? @weihanglo

(rustbot has picked a reviewer for you, use r? to override)

[weihanglo]

Worth a beta backport?

[weihanglo]

It's not immediately clear what the expected and valid input/output URL is for each function.

From my understanding, SourceId::from_url including protos (git, registry, sparse, path). So do for_alt_registry and for_registry, but not for_git.

For SourceId::new, only SparseRegistry should have proto+ prefix.

Should it desire a doc or thorough unit tests for each kind and function?

[weihanglo]

Anyhow this looks correct to me. It is also safe from #11487 as we already have regression tests.

@rust-lang/cargo could anyone help me double check this?

[ehuss]

I'm not sure I can anticipate the consequences of this change. This seems like it is really subtle stuff. I don't understand why sparse is keeping the sparse+ prefix in the SourceId.url when no other source kinds do that. I looked at what it might take to change that, and it's not pretty. Can you explain why it is done this way? Is it feasible to not include the sparse+ prefix so that it behaves the same as all the other kinds? If not, can that be clarified as to why it is done differently?

Why was CanonicalUrl ignoring the sparse+ prefix in the first place?

To clarify my understanding, is the is_crates_io() function the key for cargo treating the crates.io git and sparse index the same? And that the PackageId skips rendering the registry when it is crates.io?

I'd like to see more comments explaining some of these subtleties.

---

Can you add a test that covers the example given in the issue? Perhaps something like this:

#[cargo_test]
fn sparse_install() {
    // Checks for an issue where uninstalling from something installed
    // from a sparse registry corrupted the source IDs of other entries.
    // See https://github.com/rust-lang/cargo/issues/11751
    let _registry = registry::RegistryBuilder::new().http_index().build();

    pkg("foo", "0.0.1");
    pkg("bar", "0.0.1");

    cargo_process("install foo --registry dummy-registry")
        .with_stderr(
            "\
[UPDATING] `dummy-registry` index
[DOWNLOADING] crates ...
[DOWNLOADED] foo v0.0.1 (registry `dummy-registry`)
[INSTALLING] foo v0.0.1 (registry `dummy-registry`)
[UPDATING] `dummy-registry` index
[COMPILING] foo v0.0.1 (registry `dummy-registry`)
[FINISHED] release [optimized] target(s) in [..]
[INSTALLING] [ROOT]/home/.cargo/bin/foo
[INSTALLED] package `foo v0.0.1 (registry `dummy-registry`)` (executable `foo`)
[WARNING] be sure to add `[..]` to your PATH to be able to run the installed binaries
",
        )
        .run();
    assert_has_installed_exe(cargo_home(), "foo");
    let assert_v1 = |expected| {
        let v1 = fs::read_to_string(paths::home().join(".cargo/.crates.toml")).unwrap();
        assert_match_exact(expected, &v1);
    };
    assert_v1(
        r#"[v1]
"foo 0.0.1 (sparse+http://127.0.0.1:[..]/index/)" = ["foo"]
"#,
    );
    cargo_process("install bar").run();
    assert_has_installed_exe(cargo_home(), "bar");
    assert_v1(
        r#"[v1]
"bar 0.0.1 (registry+https://github.com/rust-lang/crates.io-index)" = ["bar"]
"foo 0.0.1 (sparse+http://127.0.0.1:[..]/index/)" = ["foo"]
"#,
    );

    cargo_process("uninstall bar")
        .with_stderr("[REMOVING] [CWD]/home/.cargo/bin/bar[EXE]")
        .run();
    assert_has_not_installed_exe(cargo_home(), "bar");
    assert_v1(
        r#"[v1]
"foo 0.0.1 (sparse+http://127.0.0.1:[..]/index/)" = ["foo"]
"#,
    );
    cargo_process("uninstall foo")
        .with_stderr("[REMOVING] [CWD]/home/.cargo/bin/foo[EXE]")
        .run();
    assert_has_not_installed_exe(cargo_home(), "foo");
    assert_v1(
        r#"[v1]
"#,
    );
}

[arlosi]

@ehuss thanks for the extra test, I've added it. I also added another comment that attempts to explain why sparse+ URLs are different.

            // Sparse URLs are different because they store the kind prefix (sparse+)
            // in the URL. This is because the prefix is necessary to differentiate
            // from regular registries (git-based). The sparse+ prefix is included
            // everywhere, including user-facing locations such as the `config.toml`
            // file that defines the registry and whenever Cargo displays it to the user.

Why was CanonicalUrl ignoring the sparse+ prefix in the first place?

We were planning on doing an autodetection mechanism so the sparse+ prefix would be optional.

To clarify my understanding, is the is_crates_io() function the key for cargo treating the crates.io git and sparse index the same? And that the PackageId skips rendering the registry when it is crates.io?

The is_crates_io() function is the key for making it display the same. The key to making the sparse index of crates.io equivalent to the git index is source replacement, and the crates_io_maybe_sparse_http() function, which returns the sparse crates.io source id (if it's enabled).

[bors]

☔ The latest upstream changes (presumably #11771) made this pull request unmergeable. Please resolve the merge conflicts.

[ehuss]

I pushed a fix for Windows.

I still feel like the subtleties here are potentially fragile, but I don't have any particular ideas that don't involve massive changes.

@bors r+

@weihanglo Yea, I think this would be good to backport. Can someone take care of that?

(BTW, thanks @arlosi for the quick fix!)

[bors]

📌 Commit ab726e7 has been approved by ehuss

It is now in the queue for this repository.

[bors]

⌛ Testing commit ab726e7 with merge 964b730...

[bors]

☀️ Test successful - checks-actions
Approved by: ehuss
Pushing 964b730 to master...

[weihanglo]

Will do the backport part.

For consolidating the logic, maybe we should test against an exhaustive list of possible URLs here.