rustc: 1.67.1 -> 1.68.1 #220373

RaitoBezarius · 2023-03-09T18:56:00Z

Description of changes

https://github.com/rust-lang/rust/blob/master/RELEASES.md

Things done

figsoda

Some hashes would probably need to be updated, see #217084

Cargo will introduce a change that will break our FODs for packages that use Git dependencies.

RaitoBezarius · 2023-03-09T19:14:19Z

Some hashes would probably need to be updated, see #217084

Cargo will introduce a change that will break our FODs for packages that use Git dependencies.

Making a draft until I figure out all these FODs.

figsoda · 2023-03-09T19:19:52Z

There was a treewide hash change like this and @danieldk had a script, maybe this will help speeding up the process

We can also merge #217084 before this and convert all the hashes to cargoLock, but this will require people to come to a consensus, which is not the case right now: #217084 (comment)

yu-re-ka · 2023-03-09T19:27:54Z

If we can somehow extract the Cargo.lock files from each package, we can determine if it needs updating based on if it has any git dependencies

winterqt · 2023-03-09T19:47:00Z

If we can somehow extract the Cargo.lock files from each package, we can determine if it needs updating based on if it has any git dependencies

Steal my logic from #217084 for that, it should work for 99% of packages.

RaitoBezarius · 2023-03-09T21:08:00Z

I will not have time to perform the treewide before maybe this weekend or more days, so anyone who really want to have this change can go ahead. I'm also in favor of waiting a proper resolution for #217084 (e.g. a merge).

oxalica · 2023-03-17T13:31:39Z

Could we enable fresh stabilized sparse-registry for fetchCargoTarball? This can dramatically improve the fetching speed if we need a treewide hash update anyway.

RaitoBezarius · 2023-03-17T16:13:03Z

Could we enable fresh stabilized sparse-registry for fetchCargoTarball? This can dramatically improve the fetching speed if we need a treewide hash update anyway.

Do you see this in this PR?

oxalica · 2023-03-17T22:42:16Z

Could we enable fresh stabilized sparse-registry for fetchCargoTarball? This can dramatically improve the fetching speed if we need a treewide hash update anyway.

Do you see this in this PR?

No? I mean:

diff --git a/pkgs/build-support/rust/fetch-cargo-tarball/default.nix b/pkgs/build-support/rust/fetch-cargo-tarball/default.nix
index 36ab9316974..6c4f8e3053a 100644
--- a/pkgs/build-support/rust/fetch-cargo-tarball/default.nix
+++ b/pkgs/build-support/rust/fetch-cargo-tarball/default.nix
@@ -62,8 +62,13 @@ in stdenv.mkDerivation ({
     export CARGO_HOME=$(mktemp -d cargo-home.XXX)
     CARGO_CONFIG=$(mktemp cargo-config.XXXX)
 
-    if [[ -n "$NIX_CRATES_INDEX" ]]; then
     cat >$CARGO_HOME/config.toml <<EOF
+    [registries.crates-io]
+    protocol = 'sparse'
+    EOF
+
+    if [[ -n "$NIX_CRATES_INDEX" ]]; then
+    cat >>$CARGO_HOME/config.toml <<EOF
     [source.crates-io]
     replace-with = 'mirror'
     [source.mirror]

For nix build -f . cargo-asm.cargoDeps -vL --rebuild, sparse registry reduces building (fetching) time from 119.5s to 14.3s for me, which is about 8x speed.

figsoda · 2023-03-17T23:17:57Z

or we can set the environment variable CARGO_REGISTRIES_CRATES_IO_PROTOCOL

zowoq · 2023-03-22T00:53:58Z

or we can set the environment variable CARGO_REGISTRIES_CRATES_IO_PROTOCOL

Opened a PR for this: #222444

figsoda · 2023-03-23T16:21:59Z

1.68.1 is released

RaitoBezarius · 2023-03-23T16:25:18Z

Upgrading.

RaitoBezarius · 2023-03-23T16:40:32Z

Done and tested.

yu-re-ka · 2023-03-26T11:37:04Z

Built firefox on x86_64-linux

yu-re-ka · 2023-03-26T11:55:21Z

@ofborg build fd

yu-re-ka · 2023-03-26T12:18:38Z

Built thunderbird on aarch64-linux

zowoq · 2023-03-26T13:09:56Z

Build passthru.tests on aarch64-darwin, x86-64_darwin.

RaitoBezarius · 2023-03-26T13:28:33Z

Should we add release notes for this upgrade?

RaitoBezarius · 2023-03-26T13:29:37Z

Could we enable fresh stabilized sparse-registry for fetchCargoTarball? This can dramatically improve the fetching speed if we need a treewide hash update anyway.

Do you see this in this PR?

No? I mean:

diff --git a/pkgs/build-support/rust/fetch-cargo-tarball/default.nix b/pkgs/build-support/rust/fetch-cargo-tarball/default.nix
index 36ab9316974..6c4f8e3053a 100644
--- a/pkgs/build-support/rust/fetch-cargo-tarball/default.nix
+++ b/pkgs/build-support/rust/fetch-cargo-tarball/default.nix
@@ -62,8 +62,13 @@ in stdenv.mkDerivation ({
     export CARGO_HOME=$(mktemp -d cargo-home.XXX)
     CARGO_CONFIG=$(mktemp cargo-config.XXXX)
 
-    if [[ -n "$NIX_CRATES_INDEX" ]]; then
     cat >$CARGO_HOME/config.toml <<EOF
+    [registries.crates-io]
+    protocol = 'sparse'
+    EOF
+
+    if [[ -n "$NIX_CRATES_INDEX" ]]; then
+    cat >>$CARGO_HOME/config.toml <<EOF
     [source.crates-io]
     replace-with = 'mirror'
     [source.mirror]

For nix build -f . cargo-asm.cargoDeps -vL --rebuild, sparse registry reduces building (fetching) time from 119.5s to 14.3s for me, which is about 8x speed.

Apologies, I meant: do you think I should do this change in this PR? I apologize if it has been taken aggressively. (English language barrier yadayada).

oxalica · 2023-03-26T15:29:57Z

I meant: do you think I should do this change in this PR?

I mean, yes. You can cherry pick the smaller env var patch from #222444. It also causes no rebuild since it affects only FODs.

RaitoBezarius · 2023-03-26T15:59:31Z

I meant: do you think I should do this change in this PR?

I mean, yes. You can cherry pick the smaller env var patch from #222444. It also causes no rebuild since it affects only FODs.

Done, thank you and sorry again for realizing too late I misunderstood your message.

vcunat · 2023-04-11T07:19:21Z

Reproducible ICE is interesting? https://hydra.nixos.org/build/214913529

figsoda · 2023-04-11T17:40:10Z

turns out rust-lang/rust#107688 was not in 1.68, opened #225778 to fix that

RaitoBezarius · 2023-04-11T17:55:34Z

Ugh, sorry for that.

ghost · 2023-04-17T04:40:32Z

Edit: I think I found the problem. ~~Testing a fix.~~ Fix here: #226593

rustc can no longer be cross compiled (build!=host) as of this commit according to git bisect.

When building the `rustc`'s vendored libraries, they are unable to find `std::`, which I find quite baffling. Any ideas?

Below is a sampling of the error messages (there are hundreds like this) from nix build -f . -L pkgsCross.aarch64-multiplatform.rustc:

rustc-aarch64-unknown-linux-gnu> error[E0463]: can't find crate for `std`
rustc-aarch64-unknown-linux-gnu>  --> /build/rustc-1.68.0-src/vendor/serde_json/build.rs:1:5
rustc-aarch64-unknown-linux-gnu>   |
rustc-aarch64-unknown-linux-gnu> 1 | use std::env;
rustc-aarch64-unknown-linux-gnu>   |     ^^^ can't find crate
rustc-aarch64-unknown-linux-gnu> error[E0463]: can't find crate for `std`
rustc-aarch64-unknown-linux-gnu>  --> /build/rustc-1.68.0-src/vendor/serde_json/build.rs:2:5
rustc-aarch64-unknown-linux-gnu>   |
rustc-aarch64-unknown-linux-gnu> 2 | use std::process::Command;
rustc-aarch64-unknown-linux-gnu>   |     ^^^ can't find crate
rustc-aarch64-unknown-linux-gnu> error[E0463]: can't find crate for `std`
rustc-aarch64-unknown-linux-gnu>  --> /build/rustc-1.68.0-src/vendor/serde_json/build.rs:3:5
rustc-aarch64-unknown-linux-gnu>   |
rustc-aarch64-unknown-linux-gnu> 3 | use std::str::{self, FromStr};
rustc-aarch64-unknown-linux-gnu>   |     ^^^ can't find crate
rustc-aarch64-unknown-linux-gnu> error: cannot find macro `println` in this scope
rustc-aarch64-unknown-linux-gnu>   --> /build/rustc-1.68.0-src/vendor/serde_json/build.rs:38:9
rustc-aarch64-unknown-linux-gnu>    |
rustc-aarch64-unknown-linux-gnu> 38 |         println!("cargo:rustc-cfg=no_btreemap_retain");
rustc-aarch64-unknown-linux-gnu>    |         ^^^^^^^
rustc-aarch64-unknown-linux-gnu> error: cannot find macro `println` in this scope
rustc-aarch64-unknown-linux-gnu>   --> /build/rustc-1.68.0-src/vendor/serde_json/build.rs:32:9
rustc-aarch64-unknown-linux-gnu>    |
rustc-aarch64-unknown-linux-gnu> 32 |         println!("cargo:rustc-cfg=no_btreemap_remove_entry");
rustc-aarch64-unknown-linux-gnu>    |         ^^^^^^^
rustc-aarch64-unknown-linux-gnu> error: cannot find macro `println` in this scope
rustc-aarch64-unknown-linux-gnu>   --> /build/rustc-1.68.0-src/vendor/serde_json/build.rs:26:9
rustc-aarch64-unknown-linux-gnu>    |
rustc-aarch64-unknown-linux-gnu> 26 |         println!("cargo:rustc-cfg=no_btreemap_get_key_value");
rustc-aarch64-unknown-linux-gnu>    |         ^^^^^^^
rustc-aarch64-unknown-linux-gnu> error: cannot find macro `println` in this scope
rustc-aarch64-unknown-linux-gnu>   --> /build/rustc-1.68.0-src/vendor/serde_json/build.rs:14:13
rustc-aarch64-unknown-linux-gnu>    |
rustc-aarch64-unknown-linux-gnu> 14 |             println!("cargo:rustc-cfg=limb_width_32");
rustc-aarch64-unknown-linux-gnu>    |             ^^^^^^^
rustc-aarch64-unknown-linux-gnu> error: cannot find macro `println` in this scope
rustc-aarch64-unknown-linux-gnu>   --> /build/rustc-1.68.0-src/vendor/serde_json/build.rs:11:13
rustc-aarch64-unknown-linux-gnu>    |
rustc-aarch64-unknown-linux-gnu> 11 |             println!("cargo:rustc-cfg=limb_width_64");
rustc-aarch64-unknown-linux-gnu>    |             ^^^^^^^
rustc-aarch64-unknown-linux-gnu> error[E0531]: cannot find tuple struct or tuple variant `Some` in this scope
rustc-aarch64-unknown-linux-gnu>   --> /build/rustc-1.68.0-src/vendor/serde_json/build.rs:19:9
rustc-aarch64-unknown-linux-gnu>    |
rustc-aarch64-unknown-linux-gnu> 19 |         Some(minor) => minor,
rustc-aarch64-unknown-linux-gnu>    |         ^^^^ not found in this scope

lblasc · 2023-04-19T09:20:33Z

Binaries generated by crate2nix (buildRustCrate) are much bigger after the last staging iteration, most likely do to rust bump (still checking), debug and release are affected.

I've tested on some internal crate.

Commit before merge bf839a4b2031557eb43e77d39d343b7ecc84c692:

150M    result/bin

After merge:

1.2G    result/bin

RaitoBezarius · 2023-04-19T23:00:13Z

Can you try by disabling auditable? (cargo-auditable)

figsoda · 2023-04-19T23:17:06Z

buildRustCrate doesn't have an auditable option

lblasc · 2023-04-20T11:08:17Z

It is not connected with rust version bump, after reverting to a 1.67.1 on master binaries are still much bigger.

RaitoBezarius · 2023-04-20T11:30:50Z

It is not connected with rust version bump, after reverting to a 1.67.1 on master binaries are still much bigger.

Can you do a bisect?

ghost · 2023-04-20T19:42:32Z

@lblasc, unless you are 100% sure that it was this PR that caused your problem, you need to open a new issue for your problem (you can reference it here). Otherwise the discussion keeps getting moved from one PR to another, pinging the wrong people.

I've tested on some internal crate.

Please post a complete reproducer in the new issue when you open it.

Note: before #210843 buildRustCrate was unconditionally stripping all builds, including debug builds. That was fixed by #210843. Maybe you're doing a debug build? If so, either switch to doing a release build or else strip manually.

dzmitry-lahoda · 2023-07-31T14:31:04Z

Interesting, it worked all with git deps before this PR. What was reason?
All reasonably big projects work with git deps. And why to break what was warking before? would be awesome there would be warning before breaking people using working nix.

winterqt · 2023-07-31T16:30:24Z

@dzmitry-lahoda You're about 6 months late to this discussion, but we talked about this issue (as well as the best way to handle it) for months. This isn't a breakage on our part, but Cargo's. See #217084, #221716, and the minimal discussion in these comments (starting here) for more information.

If you're working with a local project that doesn't have the limitations of Nixpkgs, just use importCargoLock or another solution that parses your Cargo.lock, like you probably should have been doing anyways.

I'd appreciate it if you did a tad bit of research (such as even reading the comments on this PR) before assuming that we didn't put in the work to solve this issue, or that we never noticed it.

dzmitry-lahoda · 2023-08-01T10:25:05Z

sorry, thank you for hard work

https://opencollective.com/nixos/contributions/678610

rustc: 1.67.1 -> 1.68.0

214274d

RaitoBezarius requested review from Mic92, zowoq, winterqt and figsoda as code owners March 9, 2023 18:56

github-actions bot added the 6.topic: rust label Mar 9, 2023

figsoda requested changes Mar 9, 2023

View reviewed changes

RaitoBezarius marked this pull request as draft March 9, 2023 19:14

ofborg bot added 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Mar 9, 2023

RaitoBezarius mentioned this pull request Mar 10, 2023

Don't use crane for production nix-community/lanzaboote#98

Closed

06kellyjac mentioned this pull request Mar 16, 2023

deno: 1.31.1 -> 1.32.3 #221462

Merged

12 tasks

zowoq mentioned this pull request Mar 22, 2023

rustPlatform.fetchCargoTarball: use sparse protocol for crates.io #222444

Closed

12 tasks

rustc: 1.68.0 -> 1.68.1

4efa843

RaitoBezarius changed the title ~~rustc: 1.67.1 -> 1.68.0~~ rustc: 1.67.1 -> 1.68.1 Mar 23, 2023

rustfmt: fix build for 1.68

3e46b7a

ofborg bot requested review from basvandijk and globin March 26, 2023 12:26

yu-re-ka approved these changes Mar 26, 2023

View reviewed changes

RaitoBezarius marked this pull request as ready for review March 26, 2023 13:25

rustPlatform.fetchCargoTarball: use sparse protocol for crates.io

fd10b2b

zowoq merged commit 9861cf4 into NixOS:staging Mar 26, 2023

ghost mentioned this pull request Apr 17, 2023

rustc: fix >=1.68 host!=build #226593

Merged

2 tasks

vcunat mentioned this pull request Apr 19, 2023

staging-next 2023-04-05 #224806

Merged

3 tasks

rustc: 1.67.1 -> 1.68.1 #220373

rustc: 1.67.1 -> 1.68.1 #220373

Conversation

RaitoBezarius commented Mar 9, 2023 • edited Loading

Description of changes

Things done

figsoda left a comment

Choose a reason for hiding this comment

RaitoBezarius commented Mar 9, 2023

figsoda commented Mar 9, 2023 • edited Loading

yu-re-ka commented Mar 9, 2023

winterqt commented Mar 9, 2023

RaitoBezarius commented Mar 9, 2023

oxalica commented Mar 17, 2023

RaitoBezarius commented Mar 17, 2023

oxalica commented Mar 17, 2023

figsoda commented Mar 17, 2023

zowoq commented Mar 22, 2023

figsoda commented Mar 23, 2023

RaitoBezarius commented Mar 23, 2023

RaitoBezarius commented Mar 23, 2023 • edited Loading

yu-re-ka commented Mar 26, 2023

yu-re-ka commented Mar 26, 2023

yu-re-ka commented Mar 26, 2023

zowoq commented Mar 26, 2023

RaitoBezarius commented Mar 26, 2023

RaitoBezarius commented Mar 26, 2023

oxalica commented Mar 26, 2023

RaitoBezarius commented Mar 26, 2023

vcunat commented Apr 11, 2023

figsoda commented Apr 11, 2023

RaitoBezarius commented Apr 11, 2023

ghost commented Apr 17, 2023 • edited by ghost Loading

lblasc commented Apr 19, 2023

RaitoBezarius commented Apr 19, 2023

figsoda commented Apr 19, 2023

lblasc commented Apr 20, 2023

RaitoBezarius commented Apr 20, 2023

ghost commented Apr 20, 2023 • edited by ghost Loading

dzmitry-lahoda commented Jul 31, 2023

winterqt commented Jul 31, 2023

dzmitry-lahoda commented Aug 1, 2023

RaitoBezarius commented Mar 9, 2023 •

edited

Loading

figsoda commented Mar 9, 2023 •

edited

Loading

RaitoBezarius commented Mar 23, 2023 •

edited

Loading

ghost commented Apr 17, 2023 •

edited by ghost

Loading

ghost commented Apr 20, 2023 •

edited by ghost

Loading