Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup buildRustCrate expression #75563

merged 9 commits into from Jan 2, 2020


Copy link

@andir andir commented Dec 12, 2019

Motivation for this change

I am working on test support for buildRustCrate and as a prerequisite I started cleaning up the expression so it gets easier to grasp what is actually going on. Some reformatting, minor code changes, moved stuff around. The main part is probably the "simpler" buildPhase were we are no longer constructing a huge script from a huge file but have a file with common primitives that we are driving based on the current crate.

There is probably more stuff that we can cleanup and rework here but before this gets too big I'd like to land this.

I did successfully rebuild two of my larger private projects, carnix, toml2nix & the entire buildRustCrateTests set.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits
Notify maintainers

cc @

andir added 9 commits Dec 11, 2019
We can just use `lib` instead of `builtins` in all cases but the
`hashString` case. Also changed a few lines to make use of some optional
helpers from lib.
The expression is already long and confusing enough without the color
stuff sprinkled in. Moving it to a dedicated file makes sense.

I switched a bit of the color support code to pure Nix since there
wasn't much point in doing that in bash while we can just do it in Nix.
I already have a few changes in here that will trigger rebuilds so I
might as well do that substitution now.
That code had been in the derivation for a while but no explanation was
given why that is needed. It might be helpful to our future selfs to
document why things are done the way they are.
This should carry the function better then `makeDeps` as it isn't
producing deps but the rustc arguments required to link against those.
This should make it more obvious that we have three parts to it and not
just one long gibberish string that makes up all of it.
We can get rid of a bunch of workarounds that were in the build script
before by just passing on the `crateBin` attribute.

Before we converted the list of attributes to a string only to convert
it back in bash during the build phase. We can do the entire looping
through builds in Nix and thus need no conversion and parsing of
attributes over and over again.

The big part that still remains bash is the heuristic that cargo
introduced and that we can't do at eval time.
The previous lines were only different in the kind of dependencies but
otherwise exactly the same. It makes the entire thing a bit more
readable by moving this into a function that takes care of this.
This means we aren't rebuilding hat file for each crate we are building
and the buildPhase expression is a lot easier to comprehent.
Copy link
Member Author

@andir andir commented Dec 18, 2019

@GrahamcOfBorg test buildRustCrateTests

@andir andir merged commit 9f03cb8 into NixOS:master Jan 2, 2020
15 checks passed
15 checks passed
tests.buildRustCrateTests on aarch64-linux No attempt
tests.buildRustCrateTests on x86_64-linux No attempt
Evaluation Performance Report Evaluator Performance Report
grahamcofborg-eval ^.^!
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
grahamcofborg-eval-check-meta config.nix: checkMeta = true
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
grahamcofborg-eval-package-list nix-env -qa --json --file .
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
let p = toString (src + ("/" + f));
p == path || (lib.strings.hasPrefix (p + "/") path)
) includedFiles
) src;
exclude = excludedFiles: src: builtins.filterSource (path: type:
lib.lists.all (f:
exclude = excludedFiles: src: lib.filterSource (path: type:

This comment has been minimized.


xzfc Jan 6, 2020

lib.filterSource is not defined and I'm getting an error for my carnix-generated project.

error: attribute 'filterSource' missing, at /nix/store/cyzias1yfbljig7jzi6iagfqwqimw16g-unstable-20.03pre208171.2e8fc97dbfa/unstable/pkgs/build-support/rust/build-rust-crate/helpers.nix:21:33
# Generated by carnix 0.10.0: carnix generate-nix --src ./.
# ...
buildRustCrate {
   # ...
   src = exclude [ ".git" "target" ] ./.;
   # ...
# ...

This comment has been minimized.


andir Jan 6, 2020
Author Member

oh wow, sorry. I thought I fixed that since I ran into the exact same issue. I'll push a fix later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.