Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crystal: reduce closure size, more robust runtime #74682

Merged
merged 1 commit into from Dec 3, 2019

Conversation

@manveru
Copy link
Contributor

manveru commented Nov 29, 2019

Motivation for this change

Crystal depended on the previous version after installation, as well as llvm, causing up to 3-4 versions of crystals to be installed at once.
The fix is to set the CRYSTAL_LIBRARY_PATH during build, to avoid reusing the one of the previous version which would be baked into the binary and cause Nix to deem it a dependency.

This also adds changes to the CRYSTAL_LIBRARY_PATH of the wrapped crystal binary, to provide needed dependencies for the standard library to avoid common issues that would require a separate nix-shell or derivation. It increases closure size a little, but should overall provide a much nicer user experience and for me it's worth it.

Also adding a snapshot of the upcoming 0.32 version, which has a new dependency on readline to showcase how we need to make the generic more configurable. It's not exposed on top-level for now, and shouldn't cause any build on its own. But I'm using it because of some new features and will bump it next week when the final release is out.

Another change is that I'm using lib.fix instead of let. This allows overrides to work as expected again, while keeping the ability to pass itself to buildCrystalPackage.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @peterhoeg @david50407

--suffix CRYSTAL_PATH : lib:$out/lib/crystal \
--suffix LIBRARY_PATH : ${lib.makeLibraryPath buildInputs}
--suffix CRYSTAL_LIBRARY_PATH : ${lib.makeLibraryPath (compiler.buildInputs ++ [

This comment has been minimized.

Copy link
@peterhoeg

peterhoeg Nov 30, 2019

Member

how about a:

(compilter.buildInputs ++ (map lib.getLib [ pcre zlib .... ])

to avoid having to hardcode the output?

This comment has been minimized.

Copy link
@manveru

manveru Nov 30, 2019

Author Contributor

Ah, I didn't know about this. I tried the make*Path functions, but none really fit well.

This comment has been minimized.

Copy link
@manveru

manveru Nov 30, 2019

Author Contributor

OK, had to work around buildInputs turning into a list of dev outputs, so I moved that list outside. It seems a lot cleaner now.

generic = { version, sha256, binary, doCheck ? true }:
let compiler = stdenv.mkDerivation rec {
generic = ({ version, sha256, binary, doCheck ? true, extraBuildInputs ? [] }:
lib.fix (compiler: stdenv.mkDerivation {

This comment has been minimized.

Copy link
@peterhoeg

peterhoeg Nov 30, 2019

Member

Do you have a reference to documentation for lib.fix?

This comment has been minimized.

Copy link
@manveru
@@ -95,16 +96,27 @@ let
"--single-module" # needed for deterministic builds
];

# This makes sure we don't keep depending on the previous version of
# crystal used to build this one.
CRYSTAL_LIBRARY_PATH = "${placeholder "out"}/lib/crystal";

This comment has been minimized.

Copy link
@peterhoeg

peterhoeg Nov 30, 2019

Member

Nice catch!!

@manveru manveru force-pushed the manveru:crystal-improvments branch from d373eec to bf4b82c Nov 30, 2019
@ofborg ofborg bot requested a review from peterhoeg Nov 30, 2019
@manveru manveru force-pushed the manveru:crystal-improvments branch from bf4b82c to aa2c03f Nov 30, 2019
--suffix CRYSTAL_PATH : lib:$out/lib/crystal \
--suffix LIBRARY_PATH : ${lib.makeLibraryPath buildInputs}
--suffix CRYSTAL_LIBRARY_PATH : ${
lib.makeLibraryPath (commonBuildInputs extraBuildInputs)

This comment has been minimized.

Copy link
@peterhoeg

peterhoeg Dec 1, 2019

Member

commonBuildInputs already contains extraBuildInputs.

This comment has been minimized.

Copy link
@manveru

manveru Dec 1, 2019

Author Contributor

yes, as an argument, that comes from here.

@grahamc grahamc merged commit cd2719f into NixOS:master Dec 3, 2019
15 checks passed
15 checks passed
crystal on aarch64-linux No attempt
Details
Evaluation Performance Report Evaluator Performance Report
Details
crystal on x86_64-linux Success
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@manveru manveru deleted the manveru:crystal-improvments branch Dec 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.