Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn the new lock file format on by default #7579

Merged
merged 3 commits into from Nov 19, 2019

Conversation

@alexcrichton
Copy link
Member

alexcrichton commented Nov 11, 2019

This commit enables the support added in #7070 by default. This means
that gradually over time all Cargo.lock files will be migrated to the
new format. Cargo shipped with Rust 1.38.0 supports this new lock file
format, so any project using Rust 1.38.0 or above will converge quickly
onto the new lock file format and continue working.

The main benefit of the new format is to be more friendly to git merge
conflicts. Information is deduplicated throughout the lock file to avoid
verbose depedencies lists and the checksum data is all listed inline
with [[package]]. This has been deployed with rust-lang/rust for some
time now and it subjectively at least seems to have greatly reduced the
amount of bouncing that happens for touching Cargo.lock.

@rust-highfive

This comment has been minimized.

Copy link

rust-highfive commented Nov 11, 2019

r? @ehuss

(rust_highfive has picked a reviewer for you, use r? to override)

@ehuss

This comment has been minimized.

Copy link
Contributor

ehuss commented Nov 11, 2019

So I imagine this will be problematic for projects that have extra long backwards compatibility. Is it correct that this is a forced upgrade for everyone?

What about having it so that it only applies to new projects for a while? That is, it only uses the new format if Cargo.lock is missing. That might allow more people to start using it now without disrupting projects with longer compatibility concerns.

@alexcrichton

This comment has been minimized.

Copy link
Member Author

alexcrichton commented Nov 12, 2019

Yeah that's true that this would be an upgrade for everyone, but it notably only affects projects which span many toolchain versions and are also actively developed on all those versions. While that seems pretty rare to me, I think you've got a good idea to start out by just phasing in with new lock files, so I'll see if I can whip that up.

alexcrichton added 2 commits Nov 8, 2019
This commit enables the support added in #7070 by default. This means
that gradually over time all `Cargo.lock` files will be migrated to the
new format. Cargo shipped with Rust 1.38.0 supports this new lock file
format, so any project using Rust 1.38.0 or above will converge quickly
onto the new lock file format and continue working.

The main benefit of the new format is to be more friendly to git merge
conflicts. Information is deduplicated throughout the lock file to avoid
verbose `depedencies` lists and the `checksum` data is all listed inline
with `[[package]]`. This has been deployed with rust-lang/rust for some
time now and it subjectively at least seems to have greatly reduced the
amount of bouncing that happens for touching `Cargo.lock`.
This commit adds support to Cargo and refactors the lockfile versioning
slightly. The goal here is that Cargo continually has two thresholds of
lockfile formats to create:

* One is used for new lock files at all times
* The other is used to update old lock files

The logic for these two thresholds is appropriately updated throughout
Cargo, and then this commit also preserves the previous update where new
lock files will get the new format, but old lockfiles will continue to
retain the old format by default.
@alexcrichton alexcrichton force-pushed the alexcrichton:lockfile-fmt branch from 5789ec8 to 013c1af Nov 12, 2019
@alexcrichton

This comment has been minimized.

Copy link
Member Author

alexcrichton commented Nov 12, 2019

Ok updated!

@ehuss

This comment has been minimized.

Copy link
Contributor

ehuss commented Nov 13, 2019

@alexcrichton the macos test failure looks legitimate. I haven't been able to fully dig into it, but it looks like it is rebuilding when it shouldn't. I can maybe try to dig more tonight.

@ehuss

This comment has been minimized.

Copy link
Contributor

ehuss commented Nov 14, 2019

@alexcrichton Here's a repro (any platform):

  1. cargo new foo
  2. Add a build.rs with fn main() {}
  3. cargo build
  4. cargo build again. The second one will rebuild when it should be fresh.

The issue is that the Cargo.lock is getting rewritten with an extra blank line. I believe this is because cargo new creates a v2 lock, then cargo build reverts it to v1 (because there are no dependencies) which adds the newline. The build script is necessary to make Cargo notice the timestamp change on Cargo.lock.

I'm a bit surprised this didn't trip up any other tests.

@alexcrichton

This comment has been minimized.

Copy link
Member Author

alexcrichton commented Nov 18, 2019

Ok I've added a new test and pushed up what I believe should be a fix for this, but it's a pretty targeted fix and I don't have a ton of confidence in it. Next time we change formats I think we should do so via #7144 since that'll make this much easier to deal with.

@ehuss

This comment has been minimized.

Copy link
Contributor

ehuss commented Nov 19, 2019

Seems good enough to me.

@bors r+

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Nov 19, 2019

📌 Commit c37a46c has been approved by ehuss

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Nov 19, 2019

⌛️ Testing commit c37a46c with merge dba478b...

bors added a commit that referenced this pull request Nov 19, 2019
Turn the new lock file format on by default

This commit enables the support added in #7070 by default. This means
that gradually over time all `Cargo.lock` files will be migrated to the
new format. Cargo shipped with Rust 1.38.0 supports this new lock file
format, so any project using Rust 1.38.0 or above will converge quickly
onto the new lock file format and continue working.

The main benefit of the new format is to be more friendly to git merge
conflicts. Information is deduplicated throughout the lock file to avoid
verbose `depedencies` lists and the `checksum` data is all listed inline
with `[[package]]`. This has been deployed with rust-lang/rust for some
time now and it subjectively at least seems to have greatly reduced the
amount of bouncing that happens for touching `Cargo.lock`.
@bors

This comment has been minimized.

Copy link
Contributor

bors commented Nov 19, 2019

☀️ Test successful - checks-azure
Approved by: ehuss
Pushing dba478b to master...

@bors bors merged commit c37a46c into rust-lang:master Nov 19, 2019
9 of 12 checks passed
9 of 12 checks passed
rust-lang.cargo Build #20191118.4 failed
Details
rust-lang.cargo #20191118.4 failed
Details
rust-lang.cargo (Windows x86_64-msvc) Windows x86_64-msvc failed
Details
homu Test successful
Details
rust-lang.cargo (Linux beta) Linux beta succeeded
Details
rust-lang.cargo (Linux nightly) Linux nightly succeeded
Details
rust-lang.cargo (Linux stable) Linux stable succeeded
Details
rust-lang.cargo (build_std) build_std succeeded
Details
rust-lang.cargo (docs) docs succeeded
Details
rust-lang.cargo (macOS) macOS succeeded
Details
rust-lang.cargo (resolver) resolver succeeded
Details
rust-lang.cargo (rustfmt) rustfmt succeeded
Details
@bors bors deleted the alexcrichton:lockfile-fmt branch Nov 19, 2019
bors added a commit to rust-lang/rust that referenced this pull request Nov 25, 2019
Update cargo, rls, books.

## nomicon

1 commits in 58e36e0e08dec5a379ac568827c058e25990d6cd..041c46e692a2592853aeca132c8dfe8eb5a79a9e
2019-10-30 08:14:24 -0500 to 2019-11-20 16:46:45 +0100
- Update unsafe-code-guidelines link (rust-lang/nomicon#175)

## cargo

15 commits in 8280633db680dec5bfe1de25156d1a1d53e6d190..750cb1482e4d0e74822cded7ab8b3c677ed8b041
2019-11-11 23:17:05 +0000 to 2019-11-23 23:06:36 +0000
- Some random comments and docstrings. (rust-lang/cargo#7625)
- Add value OUT_DIR to build-script-executed JSON message (rust-lang/cargo#7622)
- Update documentation for custom target dependencies. (rust-lang/cargo#7623)
- Document private items for binary crates by default (rust-lang/cargo#7593)
- Extend documentation on security concerns of crate names in a registry. (rust-lang/cargo#7616)
- Stabilize install-upgrade. (rust-lang/cargo#7560)
- Turn the new lock file format on by default (rust-lang/cargo#7579)
- bump im-rc version (rust-lang/cargo#7609)
- Ignore file lock errors if unsupported, on Windows (rust-lang/cargo#7602)
- Add hack for fwdansi change. (rust-lang/cargo#7607)
- Document Cargo's JSON output. (rust-lang/cargo#7595)
- Remove "cargo login" from user input when asking for login token. (rust-lang/cargo#7588)
- Fix all Clippy suggestions (but not add it to CI 🙃) (rust-lang/cargo#7574)
- Add kind/platform info to `cargo metadata` (rust-lang/cargo#7132)
- Update core-foundation requirement from 0.6.0 to 0.7.0 (rust-lang/cargo#7585)

## reference

2 commits in 45558c464fb458affbcdcb34323946da45c8a117..9e843aeb4df083522c7277179bbaa25d0507731c
2019-11-08 14:47:35 +0100 to 2019-11-24 17:44:04 +0100
- Minor never type additions. (rust-lang/reference#723)
- Update associated-items.md.  "it"->is (rust-lang/reference#721)

## book

3 commits in e79dd62aa63396714278d484d91d48826737f47f..81ebaa2a3f88d4d106516c489682e64cacba4f60
2019-10-30 07:33:12 -0500 to 2019-11-15 08:30:04 -0800
- small fix ch04-03 & code block typo ch07-02 (rust-lang/book#2138)
- Adapt content of Chapter 16.3 in order to be consistent with improved compiler message (rust-lang/book#1779)
- [Rust 1.35] Remove FnBox and use builtin impl FnOnce for Box<FnOnce()> instead. (rust-lang/book#1906)

## rls

3 commits in 5db91c7b94ca81eead6b25bcf6196b869a44ece0..9ec2b8cb57c87517bcb506ac302eae339ffa2025
2019-10-30 16:04:39 +0100 to 2019-11-24 23:16:11 +0100
- Fix test for latest nightly. (rust-lang/rls#1595)
- doc: contributing: Remove outdated LSP extension (rust-lang/rls#1594)
- Update cargo. (rust-lang/rls#1591)

## rust-by-example

1 commits in dcee312c66267eb5a2f6f1561354003950e29105..4835e025826729827a94fdeb7cb85fed288d08bb
2019-10-31 11:26:53 -0300 to 2019-11-14 09:20:43 -0300
- crates: fix suggested value for --crate-type flag (rust-lang/rust-by-example#1292)

## edition-guide

1 commits in f553fb26c60c4623ea88a1cfe731eafe0643ce34..6601cab4666596494a569f94aa63b7b3230e9769
2019-10-30 08:27:42 -0500 to 2019-11-22 12:08:58 -0500
- Remove final nursery reference
Copy link
Contributor

mathstuf left a comment

Is there some way to continue using the old format? I have CI testing -Z minimal-versions, but I test that configuration with my MSRV, which is older than 1.38, so some -Z flag to get the older format would be appreciated.

Can migrate to an issue if that is better.

@@ -59,9 +59,12 @@ pub struct Resolve {
///
/// It's theorized that we can add more here over time to track larger changes
/// to the `Cargo.lock` format, but we've yet to see how that strategy pans out.
#[derive(PartialEq, Clone, Debug)]
#[derive(PartialEq, Eq, Clone, Copy, Debug, PartialOrd, Ord)]
pub enum ResolveVersion {

This comment has been minimized.

Copy link
@mathstuf

mathstuf Nov 27, 2019

Contributor

The doccomment for this still says V1 is the default.

This comment has been minimized.

Copy link
@ehuss
@ehuss

This comment has been minimized.

Copy link
Contributor

ehuss commented Nov 27, 2019

Is there some way to continue using the old format?

If you have the old format, it will continue to be used. If you deleted the lock file, you can use an older version of Cargo to recreate it.

@mathstuf

This comment has been minimized.

Copy link
Contributor

mathstuf commented Nov 27, 2019

No, I can't, because I generate the file with nightly to get -Z minimal-versions support. But, I want to test with my MSRV. Now the Cargo.lock is unusable for this setup.

@mathstuf

This comment has been minimized.

Copy link
Contributor

mathstuf commented Nov 27, 2019

I guess to clarify, this is a library, so there's no Cargo.lock committed.

@mathstuf

This comment has been minimized.

Copy link
Contributor

mathstuf commented Nov 27, 2019

And if a Cargo.lock is committed, I found it hard to justify testing -Z minimal-versions anyways; anyone using it isn't getting the version creep that -Z minimal-versions will help to avoid in the first place.

@ehuss

This comment has been minimized.

Copy link
Contributor

ehuss commented Nov 27, 2019

Unfortunately we are recommending that people discontinue using minimal-versions, as we don't believe the design is going to work out.

@mathstuf

This comment has been minimized.

Copy link
Contributor

mathstuf commented Nov 27, 2019

Ok…why was that not announced on the tracking issue? (#5657) In any case, having it available today allows folks to go and find the projects that won't support the shallow minimal-versions so that the ecosystem is better when it lands in stable. And if that effort is wide and effective enough, maybe the deep version will be viable at some point. I do have reservations that things like unicode-rs/unicode-normalization@cd63ded will crop up with a shallow variant forcing a creep of MSRV across the ecosystem.

}
/// It's important that this trails behind `default_for_new_lockfiles` for
/// quite some time. This gives projects a quite large window to update in
/// where we don't force updates, so if projects span many versions of Cargo

This comment has been minimized.

Copy link
@est31

est31 Nov 27, 2019

Contributor

Could there be a config param for this? Somewhere in Cargo.toml maybe, or alternatively .cargo/config?

This comment has been minimized.

Copy link
@mathstuf

mathstuf Nov 27, 2019

Contributor

A flag to generate-lockfile would work nicely too.

@MikailBag MikailBag mentioned this pull request Dec 1, 2019
uqs pushed a commit to freebsd/freebsd-ports that referenced this pull request Dec 5, 2019
The new format [1,2] dropped the [metadata] table.  As a consequence
our cargo-crates.awk script no longer outputs CARGO_CRATES.  We can
get the crate list from the various [[package]] tables instead.
This should work with the new as well as the old format.

[1] rust-lang/cargo#7070
[2] rust-lang/cargo#7579

PR:		242416
Reported by:	jbeich


git-svn-id: svn+ssh://svn.freebsd.org/ports/head@519063 35697150-7ecd-e111-bb59-0022644237b5
uqs pushed a commit to freebsd/freebsd-ports that referenced this pull request Dec 5, 2019
The new format [1,2] dropped the [metadata] table.  As a consequence
our cargo-crates.awk script no longer outputs CARGO_CRATES.  We can
get the crate list from the various [[package]] tables instead.
This should work with the new as well as the old format.

[1] rust-lang/cargo#7070
[2] rust-lang/cargo#7579

PR:		242416
Reported by:	jbeich
Jehops pushed a commit to Jehops/freebsd-ports that referenced this pull request Dec 5, 2019
The new format [1,2] dropped the [metadata] table.  As a consequence
our cargo-crates.awk script no longer outputs CARGO_CRATES.  We can
get the crate list from the various [[package]] tables instead.
This should work with the new as well as the old format.

[1] rust-lang/cargo#7070
[2] rust-lang/cargo#7579

PR:		242416
Reported by:	jbeich


git-svn-id: svn+ssh://svn.freebsd.org/ports/head@519063 35697150-7ecd-e111-bb59-0022644237b5
mat813 pushed a commit to mat813/freebsd-ports that referenced this pull request Dec 6, 2019
The new format [1,2] dropped the [metadata] table.  As a consequence
our cargo-crates.awk script no longer outputs CARGO_CRATES.  We can
get the crate list from the various [[package]] tables instead.
This should work with the new as well as the old format.

[1] rust-lang/cargo#7070
[2] rust-lang/cargo#7579

PR:		242416
Reported by:	jbeich


git-svn-id: https://svn.freebsd.org/ports/head@519063 35697150-7ecd-e111-bb59-0022644237b5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.