Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build-bazel-package: switch hash mode to “flat” #87314

Merged
merged 3 commits into from May 11, 2020

Conversation

@matthewbauer
Copy link
Member

matthewbauer commented May 9, 2020

flat hashes can be substituted through hashed-mirrors, while recursive
hashes can’t. This is especially important for Bazel since the bazel
fetch dependencies can come from multiple different methods (git,
http, ftp, etc.). To do this, we create tar archives from the
output/external directory, which is then extracted to build. All of
the Bazel hashes are all updated.

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Copy link
Contributor

bhipple left a comment

This looks great; working on building and testing now!

pkgs/build-support/build-bazel-package/default.nix Outdated Show resolved Hide resolved
@FRidh
Copy link
Member

FRidh commented May 9, 2020

Should this be mentioned in the change log?

matthewbauer added 2 commits May 9, 2020
flat hashes can be substituted through hashed-mirrors, while recursive
hashes can’t. This is especially important for Bazel since the bazel
fetch dependencies can come from multiple different methods (git,
http, ftp, etc.). To do this, we create tar archives from the
output/external directory, which is then extracted to build. All of
the Bazel hashes are all updated.
@matthewbauer matthewbauer force-pushed the matthewbauer:bazel-flat branch from 9b8ed3b to 1e19cc3 May 9, 2020
@ofborg ofborg bot requested a review from andrew-d May 9, 2020
@kalbasit
Copy link
Member

kalbasit commented May 9, 2020

flat hashes can be substituted through hashed-mirrors, while recursive
hashes can’t.

Can you elaborate more on this? I'm not sure that I know the difference aside from file vs directory.

@bhipple
Copy link
Contributor

bhipple commented May 9, 2020

Yes, happy to elaborate, as this is something I've spent a lot of time looking at! I've been meaning to write up some more documentation about hashed mirroring, its advantages, and its limitations, but alas somehow one never finds the time to write docs ... anyways.

Currently, we of course have cache.nixos.org, which is the full Hydra binary cache. It has both source packages and binary packages, since it has everything Hydra needs.

We also have tarballs.nixos.org, which just has source packages mirrored. One constraint is that it is just holds simple content-addressed files, like those grabbed by fetchurl. The structure looks like:

https://tarballs.nixos.org/<hash-type>/<hash>

where <hash-type> is something like sha256, and the hash is the actual hash of the file in the server at that location.

Concretely:

$ wget tarballs.nixos.org/sha256/0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

$ nix-hash --type sha256 --flat --base32 0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

In our nix expressions, we actually reference 0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i directly in the package expression, like so:

sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";

This makes it trivially simple to substitute in an alternate fetcher that's just a basic S3 bucket or webserver with a directory of tarballs addressed by their hashes in a straightforward manner, and the fetchers will look there first. You can see that what we have there does in fact have this hash:

$ nix-build -A hello.src
/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz

$ nix-hash --type sha256 --to-base32 $(sha256sum /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz | awk '{print $1}')
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

This is great for mirroring, because it's simple, secure, efficient, and easy. Moreover, even users who are not using the Hydra binary cache or who are building nix stores at different store prefixes can still use tarballs.nixos.org or mirror it directly onto their own enterprise FTP servers/Artifactor/Nexus/etc., without a hash change.

While it is certainly possible to setup a spoofing mechanism by which tarballs are addressed both by their direct hashes and by the hash that they would have if you were to unpack them and recursively hash the directory, NixPkgs currently has no real mechanism for doing this, and it's a little dubious, since you pollute your hashed mirror with things that are addressed not by their direct content but by a proxy of content. The Software Heritage has a notion of "cooking" a tarball where you ask for something and they see if they have something that could satisfy that with some mutation, but it's a more complicated process and a little harder for users to understand/verify themselves.


That said, there are many other advantages to using a flat tar.gz file for the source package, enough so that even if the above were solved [1] we'd still want to go this route regardless. I've recently updated all of the cargo fetchers this way in #79975.

All six points at the top of that PR description are just as relevant for Bazel (if not moreso). For instance, I have some remote builders on AWS, and right now they completely choke while they replicate the tensorflow Bazel dependency src dir back and forth! When it's compressed the source directories are 80-90% smaller, which saves a ton of network traffic and builder time, resulting in a much more pleasant experience.

[1] I'd argue solving the above is a case where the cure is worse than the disease, because you end up with more complicated infra that's harder to reason about, but reasonable minds may disagree.

Copy link
Contributor

andrew-d left a comment

Assuming the hashes are correct and builds complete, this looks like an improvement to me 👍

<literal>flat</literal> hash mode is now used for dependencies
instead of <literal>recursive</literal>. This is to better allow
using hashed mirrors where needed. As a result, these hashes
will have changedv.

This comment has been minimized.

Copy link
@bhipple

bhipple May 9, 2020

Contributor
Suggested change
will have changedv.
will have changed.
@uri-canva
Copy link
Contributor

uri-canva commented May 10, 2020

Note you'll have to manually trigger the builds for the derivations that use buildBazelPackage to make sure they all build correctly.

@@ -146,14 +147,15 @@ in stdenv.mkDerivation (fBuildAttrs // {
preConfigure = ''
mkdir -p "$bazelOut"
test "${bazel.name}" = "$(<$deps/.nix-bazel-version)" || {
(cd $bazelOut && tar xfz $deps)

This comment has been minimized.

Copy link
@bhipple

bhipple May 10, 2020

Contributor

Note you'll have to manually trigger the builds for the derivations that use buildBazelPackage to make sure they all build correctly.

Unlike normal FOD staleness scenarios, this one happens to be safe, because if you forget to change the FOD and end up with a stale hash, this line will fail in the builder, since your input will be a directory that can't be extracted with tar.

@matthewbauer
Copy link
Member Author

matthewbauer commented May 11, 2020

Note you'll have to manually trigger the builds for the derivations that use buildBazelPackage to make sure they all build correctly.

Yes - this is an annoying part of Nix's caching mechanism. I think we can do something to make this better by validating the hash afterwards. Something similar is done in buildRustPackage:

# After unpacking and applying patches, check that the Cargo.lock matches our
# src package. Note that we do this after the patchPhase, because the
# patchPhase may create the Cargo.lock if upstream has not shipped one.
postPatch = (args.postPatch or "") + stdenv.lib.optionalString validateCargoDeps ''
cargoDepsLockfile=$NIX_BUILD_TOP/$cargoDepsCopy/Cargo.lock
srcLockfile=$NIX_BUILD_TOP/$sourceRoot/Cargo.lock
echo "Validating consistency between $srcLockfile and $cargoDepsLockfile"
if ! diff $srcLockfile $cargoDepsLockfile; then
# If the diff failed, first double-check that the file exists, so we can
# give a friendlier error msg.
if ! [ -e $srcLockfile ]; then
echo "ERROR: Missing Cargo.lock from src. Expected to find it at: $srcLockfile"
exit 1
fi
if ! [ -e $cargoDepsLockfile ]; then
echo "ERROR: Missing lockfile from cargo vendor. Expected to find it at: $cargoDepsLockfile"
exit 1
fi
echo
echo "ERROR: cargoSha256 is out of date"
echo
echo "Cargo.lock is not the same in $cargoDepsCopy"
echo
echo "To fix the issue:"
echo '1. Use "0000000000000000000000000000000000000000000000000000" as the cargoSha256 value'
echo "2. Build the derivation and wait it to fail with a hash mismatch"
echo "3. Copy the 'got: sha256:' value back into the cargoSha256 field"
echo
exit 1
fi
'' + ''
unset cargoDepsCopy
'';

@matthewbauer matthewbauer force-pushed the matthewbauer:bazel-flat branch from a9929ab to fe48f63 May 11, 2020
@matthewbauer matthewbauer merged commit 4354503 into NixOS:master May 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
You can’t perform that action at this time.