forked from NixOS/nixpkgs
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fetchcargo: use flat tar.gz file for vendored src instead of recursiv…
…e hash dir This has several advantages: 1. It takes up less space on disk in-between builds in the nix store. 2. It uses less space in the binary cache for vendor derivation packages. 3. It uses less network traffic downloading from the binary cache. 4. It plays nicely with hashed mirrors like tarballs.nixos.org, which only substitute --flat hashes on single files (not recursive directory hashes). 5. It's consistent with how simple `fetchurl` src derivations work. 6. It provides a stronger abstraction between input src-package and output package, e.g., it's harder to accidentally depend on the src derivation at runtime by referencing something like `${src}/etc/index.html`. Likewise, in the store it's harder to get confused with something that is just there as a build-time dependency vs. a runtime dependency, since the build-time src dependencies are tarred up. Disadvantages are: 1. It takes slightly longer to untar at the start of a build. As currently implemented, this attaches the compacted vendor.tar.gz feature as a rider on `verifyCargoDeps`, since both of them are relatively newly implemented behavior that change the `cargoSha256`. If this PR is accepted, I will push forward the remaining rust packages with a series of treewide PRs to update the `cargoSha256`s.
- Loading branch information
Showing
13 changed files
with
190 additions
and
38 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
# Updated fetchCargo behavior | ||
|
||
Changes to the `fetchcargo.nix` behavior that cause changes to the `cargoSha256` | ||
are somewhat disruptive, so historically we've added conditionals to provide | ||
backwards compatibility. We've now accumulated enough of these that it makes | ||
sense to do a clean sweep updating hashes, and delete the conditionals in the | ||
fetcher to simplify maintenance and implementation complexity. These | ||
conditionals are: | ||
|
||
1. When cargo vendors dependencies, it generates a config. Previously, we were | ||
hard-coding our own config, but this fails if there are git dependencies. We | ||
have conditional logic to sometimes copy the vendored cargo config in, and | ||
sometimes not. | ||
|
||
2. When a user updates the src package, they may forget to update the | ||
`cargoSha256`. We have an opt-in conditional flag to add the `Cargo.lock` | ||
into the vendor dir for inspection and compare at build-time, but it defaults | ||
to false. | ||
|
||
3. We were previously vendoring into a directory with a recursive hash, but | ||
would like to vendor into a compressed tar.gz file instead, for the reasons | ||
specified in the git commit message adding this feature. | ||
|
||
|
||
## Migration plan | ||
|
||
1. (DONE in this PR) Implement `fetchCargoTarball` as a separate, clean fetcher | ||
implementation along-side `fetchcargo`. Rename `verifyCargoDeps` (default | ||
false) to `legacyCargoFetcher` (default true), which switches the fetcher | ||
implementation used. Replace `verifyCargoDeps = true;` with | ||
`legacyCargoFetcher = false;` in Rust applications. | ||
|
||
2. Send a treewide Rust PR that sets `legacyCargoFetcher = true;` in all Rust | ||
applications not using this (which is ~200 of them), with a note to | ||
maintainers to delete if updating the package. Change the default in | ||
`buildRustPackage` to false. | ||
|
||
3. Go through all Rust src packages deleting the `legacyCargoFetcher = false;` | ||
line and re-computing the `cargoSha256`, merging as we go. | ||
|
||
4. Delete the `fetchcargo.nix` implementation entirely and also remove: | ||
- All overrides in application-level packages | ||
- The `fetchcargo-default-config.toml` and conditionals around using it when | ||
no `$CARGO_CONFIG` exists | ||
- This README.md file |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,81 @@ | ||
{ stdenv, cacert, git, cargo, python3 }: | ||
let cargo-vendor-normalise = stdenv.mkDerivation { | ||
name = "cargo-vendor-normalise"; | ||
src = ./cargo-vendor-normalise.py; | ||
nativeBuildInputs = [ python3.pkgs.wrapPython ]; | ||
dontUnpack = true; | ||
installPhase = "install -D $src $out/bin/cargo-vendor-normalise"; | ||
pythonPath = [ python3.pkgs.toml ]; | ||
postFixup = "wrapPythonPrograms"; | ||
doInstallCheck = true; | ||
installCheckPhase = '' | ||
# check that ./fetchcargo-default-config.toml is a fix point | ||
reference=${./fetchcargo-default-config.toml} | ||
< $reference $out/bin/cargo-vendor-normalise > test; | ||
cmp test $reference | ||
''; | ||
preferLocalBuild = true; | ||
}; | ||
in | ||
{ name ? "cargo-deps" | ||
, src ? null | ||
, srcs ? [] | ||
, patches ? [] | ||
, sourceRoot | ||
, sha256 | ||
, cargoUpdateHook ? "" | ||
, ... | ||
} @ args: | ||
stdenv.mkDerivation ({ | ||
name = "${name}-vendor.tar.gz"; | ||
nativeBuildInputs = [ cacert git cargo-vendor-normalise cargo ]; | ||
|
||
phases = "unpackPhase patchPhase buildPhase installPhase"; | ||
|
||
buildPhase = '' | ||
# Ensure deterministic Cargo vendor builds | ||
export SOURCE_DATE_EPOCH=1 | ||
if [[ ! -f Cargo.lock ]]; then | ||
echo | ||
echo "ERROR: The Cargo.lock file doesn't exist" | ||
echo | ||
echo "Cargo.lock is needed to make sure that cargoSha256 doesn't change" | ||
echo "when the registry is updated." | ||
echo | ||
exit 1 | ||
fi | ||
# Keep the original around for copyLockfile | ||
cp Cargo.lock Cargo.lock.orig | ||
export CARGO_HOME=$(mktemp -d cargo-home.XXX) | ||
CARGO_CONFIG=$(mktemp cargo-config.XXXX) | ||
${cargoUpdateHook} | ||
cargo vendor $name | cargo-vendor-normalise > $CARGO_CONFIG | ||
# Add the Cargo.lock to allow hash invalidation | ||
cp Cargo.lock.orig $name/Cargo.lock | ||
# Packages with git dependencies generate non-default cargo configs, so | ||
# always install it rather than trying to write a standard default template. | ||
install -D $CARGO_CONFIG $name/.cargo/config; | ||
''; | ||
|
||
# Build a reproducible tar, per instructions at https://reproducible-builds.org/docs/archives/ | ||
installPhase = '' | ||
tar --owner=0 --group=0 --numeric-owner --format=gnu \ | ||
--sort=name --mtime="@$SOURCE_DATE_EPOCH" \ | ||
-czf $out $name | ||
''; | ||
|
||
outputHashAlgo = "sha256"; | ||
outputHash = sha256; | ||
|
||
impureEnvVars = stdenv.lib.fetchers.proxyImpureEnvVars; | ||
} // (builtins.removeAttrs args [ | ||
"name" "sha256" "cargoUpdateHook" | ||
])) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters