Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Omit checksum verification for local git dependencies #11188

Conversation

jarhodes314
Copy link

@jarhodes314 jarhodes314 commented Oct 7, 2022

What does this PR try to resolve?

Fixes #10939

With the current implementation of the local registry source in cargo, all dependencies in a local registry require a valid checksum to be present before compilation. Git dependencies, however, don't have a checksum and therefore cannot pass this verification step, meaning git dependencies cannot be used from a local registry at present. This change skips the verification when a checksum is not present. In #10939, I have detailed why this change can only be effected within cargo itself, as opposed
to a change in cargo local-registry.

How should we test and review this PR?

I have created an example crate in https://github.com/jarhodes314/rust-lr-git-deps and explained in the README how this fails with the version of Cargo I have installed locally, and how it is fixed when instead using Cargo built from this branch.

Additional information

To maintain backwards compatibility with cargo local-registry, all the checks are based on the checksum being an empty string for a git dependency, rather than null in the index. This mainly made testing my own changes easier. Since the git dependency functionality is entirely broken in existing versions of cargo local-registry, changing the index there to use null instead of "" in this case would be entirely possible if that is preferred from Cargo's side.

Edit: In order to keep existing tests passing, I have changed the local-registry code to use null in the index rather than "" as it does currently for git dependencies. I have updated the demo repository accordingly.

Further edit: Changing the local registry to use null requires a bunch of changes in cargo for relatively little benefit and also breaks backwards compatibility, so I reverted those changes and instead adjusted the failing test to provide a checksum.

@rust-highfive
Copy link

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @epage (or someone else) soon.

Please see the contribution instructions for more information.

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 7, 2022
src/cargo/sources/registry/index.rs Outdated Show resolved Hide resolved
crates/cargo-test-support/src/registry.rs Outdated Show resolved Hide resolved
src/cargo/sources/registry/mod.rs Outdated Show resolved Hide resolved
@bors
Copy link
Collaborator

bors commented Oct 7, 2022

☔ The latest upstream changes (presumably #11069) made this pull request unmergeable. Please resolve the merge conflicts.

@jarhodes314 jarhodes314 force-pushed the local-git-dependency-requires-checksum branch from ae6e5d0 to 0408e9f Compare October 10, 2022 10:08
@epage
Copy link
Contributor

epage commented Nov 5, 2022

As I'm unlikely to get to this soon

r? @weihanglo

@rustbot rustbot assigned weihanglo and unassigned epage Nov 5, 2022
src/cargo/sources/registry/local.rs Outdated Show resolved Hide resolved
tests/testsuite/local_registry.rs Show resolved Hide resolved
Copy link
Member

@weihanglo weihanglo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look nicer! Thanks for the update.

Comment on lines +150 to +153
let cksum = summary
.checksum()
.map(|s| s.to_string())
.filter(|s| !s.is_empty());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid this change? I feel like resolve graph should reflect what is inside the index, even it is an empty checksum = "". This also seems unnecessary in order to get cargo local-registry works.

Also, it is a bit risky to change what is included in a resolver graph, as it would get serialized into lockfile, which we want to keep as stable as possible.
(Though I currently have yet found any case breaking the compatibility)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really. The problem arises from the fact that the git dependency has no checksum, so when you generate a lockfile after adding a git dependency to a project, the relevant [[package]] section does not contain a checksum field. When updating the dependencies, cargo local-registry removes the source replacement configuration before running cargo::ops::resolve_ws, which verifies the checksums. The way round this would be to do similar filtering in Resolve::merge_from instead.

For example, after

if let Some(mine) = self.checksums.get(id) {
I added:

let mine = mine.as_ref().filter(|s| !s.is_empty());
let cksum = cksum.as_ref().filter(|s| !s.is_empty());

and this does appear to prevent the error from occurring.

re lockfile stability, my main expectation with the lockfile as a user is that enabling source replacement and running a build shouldn't cause the lockfile to change. The proposed alternative change does obviously cause the lockfile to change. My usual test for whether I have set up my local registry correctly is to verify cargo build --frozen works and immediately after syncing the local registry, that will not work.

As far as I'm aware, only cargo local-registry can generate empty checksums (as git dependencies, path dependencies and vendored dependencies all generate no checksum whatsoever) so there shouldn't be another case to break compatibility with.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am now convinced. Thank you for explaining it again and again!

As far as I'm aware, only cargo local-registry can generate empty checksums (as git dependencies, path dependencies and vendored dependencies all generate no checksum whatsoever) so there shouldn't be another case to break compatibility with.

That's also what I observed. I am going to do an FCP and see if we can reach a consensus from the team.

@weihanglo weihanglo added the T-cargo Team: Cargo label Nov 16, 2022
@weihanglo
Copy link
Member

@rfcbot merge

This PR is primarily for local registry source. It contains two changes I'd like to call out:

  • It skips verification of checksum for local registry, if given checksum from local registry is an empty string.
    This affects local registry only.
  • Resolve now only includes the checksum from a Summary only if a checksum is a non-empty string. That entails lockfiles isn't possible to contain checksum = "" anymore.
    This might impact other source kind, but I am not aware of any generating empty cksum.

I think these changes should be safe. From my observation, cargo local-registry might be the only public tool generating local registry sources. It has the ability to vendor git dependency but will create a line of index file with empty cksum {… "cksum":""}. Cargo usually skips checksum validation for git deps, so we make an assumption that an empty cksum from a local registry is always originated from a git dep, and we're happy to skip.

This is somewhat a special case for an external tool cargo local-registry, though.
(It seems to be a semi-official tool in the old days?)

@rfcbot
Copy link
Collaborator

rfcbot commented Nov 16, 2022

Team member @weihanglo has proposed to merge this. The next step is review by the rest of the tagged team members:

Concerns:

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

@rfcbot rfcbot added proposed-final-comment-period An FCP proposal has started, but not yet signed off. disposition-merge FCP with intent to merge labels Nov 16, 2022
@rfcbot rfcbot added final-comment-period FCP — a period for last comments before action is taken and removed proposed-final-comment-period An FCP proposal has started, but not yet signed off. labels Jan 10, 2023
@rfcbot
Copy link
Collaborator

rfcbot commented Jan 10, 2023

🔔 This is now entering its final comment period, as per the review above. 🔔

@ehuss
Copy link
Contributor

ehuss commented Jan 10, 2023

Just wanted to note that there was a concern on how this would interact with updating a git dependency (whose package version doesn't change). Cargo's src cache may not know that the package has changed, and thus cargo may work incorrectly (using the stale cache).

@weihanglo
Copy link
Member

weihanglo commented Jan 11, 2023

@rfcbot concern src-cache-not-update

As @ehuss pointed out #11188 (comment), With the current patch, Cargo's src cache is not aware of a git dependency has been updated.

Click to see steps to reproduce
  1. Add your own git dependency without any rev/branch specified.
  2. Run cargo generate-lockfile
  3. Run cargo local-registry -s Cargo.lock local-reg --git
  4. Add the follow configurations to .cargo/config.toml
    [source.local-registry]
    local-registry = "local-reg"
    
    [source.gitdep]
    git = "https://github.com/<username>/<pkg>.git"
    replace-with = "local-registry"
  5. Observe cargo build to work without any issue.
  6. Add a new function to your git dependency and push it to the remote.
  7. Comment out .cargo/config.toml
  8. Run cargo update and observe in Cargo.lock the hash of git dep URL changed
  9. Run cargo local-registry -s Cargo.lock local-reg --git to update local registry
  10. Uncomment .cargo/config.toml
  11. Edit your main package to call the newly added function in git dependency.
  12. Run cargo build and observe a failure on missing that function.

If my findings are correct, the situation of this patch is still half broken. I am sorry I don't have too much inputs about how to fix it at this moment.

@rfcbot rfcbot added the proposed-final-comment-period An FCP proposal has started, but not yet signed off. label Jan 11, 2023
@rfcbot rfcbot removed the final-comment-period FCP — a period for last comments before action is taken label Jan 11, 2023
@jonhoo
Copy link
Contributor

jonhoo commented Mar 7, 2023

I wanted to chime in on this from a slightly different angle given the context in #10071.

My understanding of the ask here is that you want it to be possible to:

  1. vendor a git dependency into a local registry while keeping Cargo.lock and have the build still work
  2. stop using the local registry vendor copy while keeping Cargo.lock and have the build still work
  3. update the git dependency vendor in a local registry while keeping Cargo.lock and have the build still work

The challenge here, I think, is that Cargo has no way to know that a particular local registry entry is "really" a git dependency. We could add that kind of metadata, but if we do, nothing stops that same annotation to anything in the local registry, meaning we're really discussing whether it should be possible to opt-out of checksumming for any package in a local registry.

For 1, the solution (I think) is to simply have Cargo allow the case where there is no checksum in Cargo.lock but the registry does have a checksum. That should be an easy fix.

For 2, I thought this case was already allowed since you're able to turn a foo = "1.0" dependency into a foo = { git = "" } dependency. Is that not the case?

For 3, this is the hard one. The approach taken in this PR runs into #10071 (which is ~= the concern currently raised), but more importantly it's going to mean that any entry in a local registry can be changed under Cargo, which is unlike how a "real" registry works. We (well, the Cargo team) will have to decide whether that's a feature they want Cargo local registries to have, even if the feature is partially obscured (such as having to add a "was a git dependency" marker). I have certainly run into cases where it'd be useful to be able to change something that's located in a local registry (and where I cannot use a directory registry instead), but I also acknowledge that it's a departure from the model local registries have traditionally had.

@ehuss
Copy link
Contributor

ehuss commented Jun 27, 2023

@jarhodes314 I wanted to check in and see if you are still interested in working on this. It looks like there were some concerns raised about how cargo makes assumptions about checksums not changing, and those could cause problems. Is that something you can look at?

@bors
Copy link
Collaborator

bors commented Feb 20, 2024

☔ The latest upstream changes (presumably #13409) made this pull request unmergeable. Please resolve the merge conflicts.

@weihanglo
Copy link
Member

Going to cancel the FCP and close this. We're still interested in seeing a reasonable solution to the original issue. However, as Jon has outlined in #11188 (comment) it may not be as easy as we thought before.

Thanks every for your participation.

@weihanglo weihanglo closed this Sep 20, 2024
@rfcbot rfcbot removed proposed-final-comment-period An FCP proposal has started, but not yet signed off. disposition-merge FCP with intent to merge labels Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-cargo Team: Cargo
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

git dependencies in a local-registry require a non-existent checksum
8 participants