A way for users to bulk upgrade across incompatible versions #12425

epage · 2023-08-01T01:58:57Z

Problem

A lot of incompatible upgrades have a small enough of breakages that a "upgrade all of my incompatible version requirements" would make it easier to go through this process.

This is currently exposed in the cargo upgrade command

Proposed Solution

See https://internals.rust-lang.org/t/feedback-on-cargo-upgrade-to-prepare-it-for-merging/17101/141

Notes

No response

The text was updated successfully, but these errors were encountered:

epage · 2023-08-01T02:03:00Z

Below is some background on what we are using to help come up with a design

Care Abouts

Priorities

Don't break behavior on cargo update
Don't write out incompatible Cargo.lock and Cargo.toml
Focus is on end-users solving common problems and not on being a general programmatic CLI that is meant to cover every case
Be predictable and understandable
- Can someone unfamiliar with Rust, reading a blog post, predict what different command invocations will do?
- Preference for not having too similarly named commands
When higher priorities allow, avoid errors that make users go "if you know exactly what I was asking for then why didn't you just do it?"; those are a sign of issues with the UX.
Don't be hassle when dealing with intentionally held back dependencies

Primary use cases:

Want to have simple workflow for "upgrade incompatible dependencies only".
Want to have simple workflow for bulk lock to latest compatible (already exists as cargo update)
(medium priority) Selective modify one dependency's version requirement to latest compatible, latest incompatible
(lower priority) Want to have simple-ish workflow for bulk upgrade to latest compatible
(lower priority) bulk upgrade all dependencies (could just be two command invocations)

Secondary use cases are:

Selective modify version requirement to user-provided value
Upgrade explicitly pinned version requirements

Some open questions we had

How do we tell when a renamed dependency like tokio_03 is pinned or not?
- We could just assume all renamed are pinned
- We could add a dependency field but I'm a bit leery of adding that kind of bookkeeping to the manifest
- We could force users to --exclude these dependencies but that might be a bit of a pain to always remember to do
- We could only skip renamed if multiple dependencies exist that point to the same package

Context

Currently, cargo update is focused solely on Cargo.lock editing

Spans entire dependency tree
Multiple versions of a package may exist, referenced by name@version
Deals with exact versions and not version ranges
Only affects you and not your dependents

Version requirement editing is different in that

Workspace members only
May want differences between members
Supports alternative names for packages
Affects dependents

And as a reminder of the CLI:

cargo update -p foo  # select a non-ambiguous `foo` and update it
cargo update -p foo@ver  # selects a specific `foo` to update
cargo update -p foo --aggressive  # also update dependencies
cargo update -p foo --precise <ver>  # select a specific version
cargo update --locked  # fail if the lockfile will change

Note: cargo add --locked will also fail if the manifest will change

Some design tools we can consider include

Renaming a command, making the old name an alias
- Even if there isn't a culture shift to use the new name, cargo <cmd> --help and cargo --list will point people to the new name
Versions without the build metadata field is a subset of version requirement syntax, we may be able to do some mixing of them
- Precedence: using the same foo@ver syntax for versions and version requirements
Minimal-version resolution being the default mode would make Cargo.lock mostly align with Cargo.toml, making it easier to conflate the two commands (whether merging them or keeping separate but de-emphasizing update)

Interjection

Through this, I realized that the core of my concern with our previous attempts at a single command is that it feels like we are shoehorning new behaviors into cargo update rather than making the behavior cohesive.

If I see a cargo update --incompatible on a blog post, can I predict what will happen if you do cargo update? No, because --save is needed to match behavior
We were trying to make --package be both for package IDs and dependency names to make some of the cargo upgrade workflows work
We were trying to overload --precise to allow control over version requirements

I also realized that my Windows Updates vs Windows Upgrades analogy for cargo update and cargo upgrade breaks down a little because cargo upgrade can do "upgrades" that are on the level of cargo update (say we call it cargo upgrade --compatible). The difference is in the target audience (yourself vs your dependents)

epage · 2023-08-01T02:04:45Z

Proposal: `cargo update` only changes version requirements as a side effect

The primary role of cargo update has been to update your active dependencies (ie Cargo.lock). We do not plan to change that role but give the user control to force it to update in situations that were previously unsupported, particularly updating the Cargo.toml if need be.

Behavior:

By default, only "safe" updates are allowed (today's behavior)
cargo update --incompatible / cargo update -i will force (ie update version reqs) update unpinned, incompatible versions (no other dependencies)
- Yes, this could potentially be called --breaking or something else. The name depends on what expresses the concept clearly especially in light of any pinning behavior we have
- Keeping -i unused would keep the door for a --interactive to be added
cargo update -p foo --precise ver will force the update to happen, only erroring if we can't (don't own relevant version reqs, is pinned), even if its incompatible but unpinned. Version requirement is only changed on incompatible.
- Maybe the error on pinned could be relaxed

Somewhere between deferred and rejected (speaking for myself): Support in cargo update for writing to the manifest for non-breaking changes, like bulk compatible upgrades of version requirements (ie a -save flag) which was one of our lower priority workflows. A --save flag is more about updating versions for your dependents, which while important for having valid lower-bounds on version requirements, doesn't fit with the existing model of cargo update. Maybe in the future we can find a way to express this in cargo update that fits with how it works or maybe another command can take on this role. We just aren't wanting to distract our efforts for handling most of the use cases to handle this one

While this tells a cohesive story, a part of me is somewhat concerned that this goes beyond the name update.

Potential related `cargo update` improvements

"Rename" --aggressive to --recursive (make --aggressive an alias for the new name) (fix(update): Clarify meaning of --aggressive as --recursive #12544)
Specialize the parsing of foo@x.y.z so that foo@x will work so long as its unambiguous (which it should be), much like cargo install foo@x.y.z supports auto-selecting the y and z fields (picks latest) (feat(pkgid): Allow incomplete versions when unambigious #12614)
Consider applying some of the output formatting from cargo upgrade into cargo update (feat(update): Tell users when they are still behind #13372)
- Provides some of the cargo outdated experience (Make cargo outdated part of cargo itself #4309)
- Is an alternative to warnings (cargo update should print warnings for incompatible candidates #7167)
Add a positional package argument to cargo update, removing the need for --package (fix(update): Make -p more convenient by being positional #12545)
- Should be safe because there are no trailing var args for a wrapped child process and no (and little chance of) flags that accept 0..=1 values

Alternatives

These are alternatives I had considered that help give an idea of what I mean by fitting into cargo.

cargo update always modifies Cargo.toml

This would be a breaking change
This would get in the way of people intentionally keeping separate versions from version requirements

We deprecate cargo update and a cargo upgrade always updates both files

This would get in the way of people intentionally keeping separate versions from version requirements

We migrate to minimal-version resolution by default

cargo update becomes less useful and we move it out of the spot light.
A cargo upgrade is added that is focused on editing version requirements

Separate commands for Cargo.lock (update) and Cargo.toml (upgrade)

Names don't clarify the role each fills
Much like Debian has apt dist-upgrade, maybe it could be cargo req-update?

Misc Notes

I don't see us making a distinction between default operator and ^ as we document them as being the same thing and sometimes people use ^ just because
We are erring on the side of needing cargo update && cargo update --incompatible vs cargo update --incompatible doing both as there are people who do want them separated and running two commands, while annoying, allows us to cover more use cases.

fix(update): Tweak CLI behavior ### What does this PR try to resolve? When looking at `cargo update` for #12425, I noticed that the two flags related to `--package` were not next to it or each other. I figured grouping them like that would make things easier to browse. When looking into that, I noticed that the two flags conflict and figured we'd provide a better error message if we did that through clap. ### How should we test and review this PR? Looking per commit will help show the behavior changes. ### Additional information I wanted to scope this to being simple, non-controversial, low effort, incremental improvements with this change so I did not look into the history of `--aggressive` not requiring `--package` like `--precise` does and figure out if there is any consistency we can be working towards.

epage · 2023-08-08T17:03:20Z

This proposal has been up here and on internals for a bit now without any major concerns raised.

@rfcbot fcp merge

rfcbot · 2023-08-08T17:03:21Z

Team member @epage has proposed to merge this. The next step is review by the rest of the tagged team members:

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

djc · 2023-08-08T19:02:39Z

Having been fairly involved in this discussion via the internals thread, I'm happy to see this move forward in a direction that I wholeheartedly support. @epage thanks for pushing this forward!

rfcbot · 2023-08-14T20:11:14Z

🔔 This is now entering its final comment period, as per the review above. 🔔

weihanglo · 2023-08-16T12:20:59Z

Can --incompatible be used in conjunction with --package?
Just recall #11974 and not sure about the implication of cargo update --incompat -p <pkgid>.

epage · 2023-08-16T12:51:16Z

Can --incompatible be used in conjunction with --package?
Just recall #11974 and not sure about the implication of cargo update --incompat -p <pkgid>.

Yes, we'd update the version requirement, if an incompatible version exists, and then run the normal code.

When working on cargo-upgrade, I found the meaning of `--aggressive` confusing and named it `--recursive` there. Renaming this in `cargo update` (with a backwards compatible alias) was referenced in rust-lang#12425.

MoSal · 2023-08-23T15:43:50Z

cargo upgrade can do "upgrades" that are on the level of cargo update (say we call it cargo upgrade --compatible). The difference is in the target audience (yourself vs your dependents)

This is probably stating the obvious. But if a dependency is imprecise, current cargo upgrade may have no effect where current cargo update would.

Let's say a dependency is serde_derive = "1" in Cargo.toml. And it's version = "1.0.183" in Cargo.lock. Then current cargo upgrade would change nothing locally, and wouldn't force dependants to get the latest version.

The separation of the tools and the clarity on each one's remit makes this, if not immediately predictable, at least easily understandable. But if this will no longer be the case, then this distinction, or a change from that behavior, should be made clear.

For what it's worth, I don't like this behavior, and I'll implement --force-precise for personal use. Especially since I only used imprecise deps in old projects.

epage · 2023-08-23T15:48:05Z

FYI on zulip I brought up the idea of a pedantic machine-applicable (ie --fixable) lint to flag imprecise dependencies.

Generally, cargo avoids positional arguments. Mostly for the commands that might forward arguments to another command, like `cargo test`. It also allows some flexibility in turning flags into options. For `cargo add` and `cargo remove`, we decided to accept positionals because the motivations didn't seem to apply as much (similar to `cargo install`). This applies the pattern to `cargo update` as well which is in the same category of commands as `cargo add` and `cargo remove`. Switching to a positional for `cargo update` (while keeping `-p` for backwards compatibility) was referenced in rust-lang#12425.

Generally, cargo avoids positional arguments. Mostly for the commands that might forward arguments to another command, like `cargo test`. It also allows some flexibility in turning flags into options. For `cargo add` and `cargo remove`, we decided to accept positionals because the motivations didn't seem to apply as much (similar to `cargo install`). This applies the pattern to `cargo update` as well which is in the same category of commands as `cargo add` and `cargo remove`. As for `--help` formatting, I'm mixed on whether `[SPEC]...` should be at the top like other positionals or should be relegated to "Package selection". I went with the latter mostly to make it easier to visualize the less common choice. Switching to a positional for `cargo update` (while keeping `-p` for backwards compatibility) was referenced in rust-lang#12425.

When working on cargo-upgrade, I found the meaning of `--aggressive` confusing and named it `--recursive` there. Renaming this in `cargo update` (with a backwards compatible alias) was referenced in rust-lang#12425.

Generally, cargo avoids positional arguments. Mostly for the commands that might forward arguments to another command, like `cargo test`. It also allows some flexibility in turning flags into options. For `cargo add` and `cargo remove`, we decided to accept positionals because the motivations didn't seem to apply as much (similar to `cargo install`). This applies the pattern to `cargo update` as well which is in the same category of commands as `cargo add` and `cargo remove`. As for `--help` formatting, I'm mixed on whether `[SPEC]...` should be at the top like other positionals or should be relegated to "Package selection". I went with the latter mostly to make it easier to visualize the less common choice. Switching to a positional for `cargo update` (while keeping `-p` for backwards compatibility) was referenced in rust-lang#12425.

Prepare for partial-version package specs ### What does this PR try to resolve? These are refactorings, test expansions, and cleanups I saw as I was preparing to implement support for `foo@0.1` as proposed in #12425. I figured these changes stand on their own so I separated them out. One further change I considered was that `foo@0` will suggest `foo` in a "did you mean" message. This is a big off *but* most likely any fix for this would be undone by the work to support `foo@0.1`, so I held off on it. ### How should we test and review this PR? Each change is broken down into an individual commit

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

feat(pkgid): Allow incomplete versions when unambigious ### What does this PR try to resolve? This was proposed in #12425 to help sand off some of the rough edges around `cargo update` for its wider use it would be getting. Its easy to accidentally get duplicate copies packages in a repo and a pain to have to specify the full version when `cargo update -p foo@1` is sufficient to describe it. Other effects - profile overrides also supports this since we already allow a spec to match multiple items - `cargo clean -p foo@...` already ignored the version, so now we also parse and ignore the partial version - `cargo tree --prune` will now accept partial versions and will match all of them Parts not effected: - Replacements - Two of the cases were found and we treat it as if the version isn't present which will error, so I think that is correct ### How should we test and review this PR? This extracts `PartialVersion` from `RustVersion` where `RustVersion` is a more specialized variant, not allowing prerelease or build. This works by adopting `PartialVersion` into `PackageIdSpec`. For `PackageIdSpec::query`, this will "just work". ### Additional information

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

feat(pkgid): Allow incomplete versions when unambigious ### What does this PR try to resolve? This was proposed in #12425 to help sand off some of the rough edges around `cargo update` for its wider use it would be getting. Its easy to accidentally get duplicate copies packages in a repo and a pain to have to specify the full version when `cargo update -p foo@1` is sufficient to describe it. Other effects - profile overrides also supports this since we already allow a spec to match multiple items - `cargo clean -p foo@...` already ignored the version, so now we also parse and ignore the partial version - `cargo tree --prune` will now accept partial versions and will match all of them Parts not effected: - Replacements - Two of the cases were found and we treat it as if the version isn't present which will error, so I think that is correct ### How should we test and review this PR? This extracts `PartialVersion` from `RustVersion` where `RustVersion` is a more specialized variant, not allowing prerelease or build. This works by adopting `PartialVersion` into `PackageIdSpec`. For `PackageIdSpec::query`, this will "just work". ### Additional information

feat(update): Tell users when they are still behind ### What does this PR try to resolve? Part of this is an offshoot of #12425 which is about pulling some of `cargo upgrade`s behavior into `cargo update`. One of the "'Potential related `cargo update` improvements" is informing the user when they are behind. Part of this is to help close the gap of users being behind on their dependencies unaware. This is commonly raised when discussing an MSRV-aware resolver (see rust-lang/rfcs#3537) but breaking changes are just as big of a deal so I'm starting this now. See also #7167, #4309 Compared to `cargo upgrade` / `cargo outdated`, I'm taking a fairly conservative approach and tweaking the existing output as a starting point / MVP. We can experiment with a richer or easier-to-consume way of expressing this over time. I view us telling people they aren't on the latest as a warning, so I made that text yellow. `clap $ cargo update --dry-run` ![image](https://github.com/rust-lang/cargo/assets/60961/4bf151e3-6b57-4073-8822-9140dd731d5e) `clap $ cargo update --dry-run --verbose` ![image](https://github.com/rust-lang/cargo/assets/60961/fbf802fb-3a6a-4e8b-a6ec-4ce49fb505f6) ### How should we test and review this PR? This sets up the minimal implementation and slowly adds bits at a time, with a test first that demonstrates it. ### Additional information I'm expecting that the `cargo upgrade` integration will extend the notes to say something like "X dependencies may be updated with `--breaking`"

loynoir · 2024-03-25T00:10:39Z

Keeping -i unused would keep the door for a --interactive to be added

Really hope --interactive.

yarn is one of the nodejs package managers, and it supports upgrade-interactive.

yarn 1

https://classic.yarnpkg.com/lang/en/docs/cli/upgrade-interactive/

yarn 4

https://yarnpkg.com/cli/upgrade-interactive

Not similar to cargo update, but similar to cargo-edit's cargo upgrade, but have a terminal ui.

For example, which supports

 Press <up>/<down> to select packages.            Press <enter> to install.
 Press <left>/<right> to select versions.         Press <ctrl+c> to abort.

? Pick the packages you want to upgrade.          Current          Range            Latest

 > @types/node --------------------------------- ◉ 16 ----------- ◯ 16.18.91 ----- ◯ 20.11.30 -----
   typescript ---------------------------------- ◉ 3.7 ---------- ◯ 3.7.7 -------- ◯ 5.4.3 --------

epage added A-new-subcommand Area: new subcommand Command-update C-feature-request Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted` S-triage Status: This issue is waiting on initial triage. labels Aug 1, 2023

epage mentioned this issue Aug 1, 2023

Merge cargo-upgrade into cargo killercup/cargo-edit#685

Open

11 tasks

epage mentioned this issue Aug 1, 2023

fix(update): Tweak CLI behavior #12428

Merged

epage added the T-cargo Team: Cargo label Aug 8, 2023

rfcbot added proposed-final-comment-period An FCP proposal has started, but not yet signed off. disposition-merge FCP with intent to merge labels Aug 8, 2023

rfcbot added the final-comment-period FCP — a period for last comments before action is taken label Aug 14, 2023

rfcbot removed the proposed-final-comment-period An FCP proposal has started, but not yet signed off. label Aug 14, 2023

epage mentioned this issue Aug 23, 2023

fix(update): Clarify meaning of --aggressive as --recursive #12544

Merged

epage mentioned this issue Aug 23, 2023

fix(update): Make -p more convenient by being positional #12545

Merged

epage added a commit to epage/cargo that referenced this issue Sep 1, 2023

feat(spec): Allow partial versions when unambigious

68d99fc

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

epage mentioned this issue Sep 1, 2023

feat(pkgid): Allow incomplete versions when unambigious #12614

Merged

epage added a commit to epage/cargo that referenced this issue Sep 1, 2023

feat(spec): Allow partial versions when unambigious

80c1381

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

epage added a commit to epage/cargo that referenced this issue Sep 1, 2023

feat(spec): Allow partial versions when unambigious

673e47b

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

epage added a commit to epage/cargo that referenced this issue Sep 1, 2023

feat(spec): Allow partial versions when unambigious

dc9f202

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

epage added a commit to epage/cargo that referenced this issue Sep 5, 2023

feat(spec): Allow partial versions when unambigious

53f80bf

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

epage added a commit to epage/cargo that referenced this issue Sep 7, 2023

feat(spec): Allow partial versions when unambigious

3e34dfb

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

epage added a commit to epage/cargo that referenced this issue Sep 7, 2023

feat(spec): Allow partial versions when unambigious

c61dc75

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

epage added a commit to epage/cargo that referenced this issue Sep 8, 2023

feat(spec): Allow partial versions when unambigious

0f06cd8

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

epage added a commit to epage/cargo that referenced this issue Sep 11, 2023

feat(spec): Allow partial versions when unambigious

1f18d45

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

epage added a commit to epage/cargo that referenced this issue Sep 14, 2023

feat(spec): Allow partial versions when unambigious

6311324

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

epage added a commit to epage/cargo that referenced this issue Sep 16, 2023

feat(spec): Allow partial versions when unambigious

82f9bd3

This was proposed in rust-lang#12425 to help improve usability of the existing `cargo update` when dealing with the added workflows.

epage mentioned this issue Sep 20, 2023

RFC: Precise Pre-release cargo update rust-lang/rfcs#3493

Merged

epage mentioned this issue Sep 29, 2023

move up looking at index summary enum #12749

Merged

This was referenced Oct 18, 2023

Make the precise field of a source an Enum #12849

Merged

Add flag for upgrade report as structured data (JSON) killercup/cargo-edit#877

Closed

epage added S-accepted Status: Issue or feature is accepted, and has a team member available to help mentor or review and removed S-triage Status: This issue is waiting on initial triage. labels Oct 23, 2023

epage self-assigned this Oct 23, 2023

This was referenced Nov 14, 2023

Preliminary support for sparse indices killercup/cargo-edit#870

Draft

allow using non-cargo command directly killercup/cargo-edit#878

Closed

epage mentioned this issue Dec 13, 2023

Add flag to allow Cargo to create a lock file that depends on yanked crates #4225

Open

ehuss removed the to-announce label Jan 14, 2024

epage mentioned this issue Jan 31, 2024

feat(update): Tell users when they are still behind #13372

Merged

loynoir mentioned this issue Mar 25, 2024

Error: The repo at path ... is unusable due to having an invalid HEAD reference killercup/cargo-edit#879

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A way for users to bulk upgrade across incompatible versions #12425

A way for users to bulk upgrade across incompatible versions #12425

epage commented Aug 1, 2023

epage commented Aug 1, 2023

epage commented Aug 1, 2023 •

edited

epage commented Aug 8, 2023

rfcbot commented Aug 8, 2023 •

edited by weihanglo

djc commented Aug 8, 2023

rfcbot commented Aug 14, 2023

weihanglo commented Aug 16, 2023

epage commented Aug 16, 2023

MoSal commented Aug 23, 2023

epage commented Aug 23, 2023

loynoir commented Mar 25, 2024

A way for users to bulk upgrade across incompatible versions #12425

A way for users to bulk upgrade across incompatible versions #12425

Comments

epage commented Aug 1, 2023

Problem

Proposed Solution

Notes

epage commented Aug 1, 2023

Care Abouts

Context

Interjection

epage commented Aug 1, 2023 • edited

Proposal: cargo update only changes version requirements as a side effect

Potential related cargo update improvements

Alternatives

Misc Notes

epage commented Aug 8, 2023

rfcbot commented Aug 8, 2023 • edited by weihanglo

djc commented Aug 8, 2023

rfcbot commented Aug 14, 2023

weihanglo commented Aug 16, 2023

epage commented Aug 16, 2023

MoSal commented Aug 23, 2023

epage commented Aug 23, 2023

loynoir commented Mar 25, 2024

epage commented Aug 1, 2023 •

edited

Proposal: `cargo update` only changes version requirements as a side effect

Potential related `cargo update` improvements

rfcbot commented Aug 8, 2023 •

edited by weihanglo