Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wip] Statically built Nix #56281

Open
wants to merge 10 commits into
base: master
from

Conversation

@matthewbauer
Copy link
Member

matthewbauer commented Feb 24, 2019

Motivation for this change

This makes it possible to build Nix statically through pkgsStatic.nix. Needs a couple of patches to nix but otherwise works smoothly.

If you'd like to try it, you can just check out this branch and run:

nix-build -A pkgsStatic.nix

and wait a little bit. Alternatively, you can use my prebuilt binary at https://matthewbauer.us/nix. It is built for x86_64-linux and can be used on any system like this:

$ curl https://matthewbauer.us/nix | sh -s ...

For instance, on a linux system without nix installed, you can do this:

$ curl https://matthewbauer.us/nix | sh -s run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable hello -c hello
Hello World!

There are many uses cases where nix is unavailable and the above command can be extremely useful. It's a little slow still, but can be sped up by reusing the same --store value (such as $HOME/.cache/nix/store). Here is the time:

$ time (curl https://matthewbauer.us/nix | sh -s run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable hello -c hello)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5596k  100 5596k    0     0  2807k      0  0:00:01  0:00:01 --:--:-- 2807k
[2 copied (24.0 MiB), 35.5 MiB DL]
Hello, world!
( curl https://matthewbauer.us/nix | sh -s run --store $(mktemp -d) -f  hello)  3.39s user 3.52s system 70% cpu 9.757 total
@ryantm

This comment has been minimized.

Copy link
Member

ryantm commented Feb 24, 2019

I tried following some of the examples in your blog post on my NixOS machine, and I get this error message:

[ryantm@laptop3:~]$ curl https://matthewbauer.us/nix | sh -s run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable hello -c hello
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5596k  100 5596k    0     0  1408k      0  0:00:03  0:00:03 --:--:-- 1408k
error: getting status of '/nix/store/p492jkrs1q0nddjqghw5d5s5w7kgifda-nix-2.2-x86_64-unknown-linux-musl': No such file or directory
[ryantm@laptop3:~]$ nix=$(mktemp); \
>   curl https://matthewbauer.us/nix > $nix && \
>   chmod +x $nix && \
>   $nix run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable \
>   ranger -c ranger
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5596k  100 5596k    0     0  5264k      0  0:00:01  0:00:01 --:--:-- 5269k
error: getting status of '/nix/store/p492jkrs1q0nddjqghw5d5s5w7kgifda-nix-2.2-x86_64-unknown-linux-musl': No such file or directory
@jb55

This comment has been minimized.

Copy link
Contributor

jb55 commented Feb 24, 2019

I'm getting the same error as @ryantm

@matthewbauer

This comment has been minimized.

Copy link
Member Author

matthewbauer commented Feb 24, 2019

Thanks for the heads up. I had forgotten about those extra files needed. The updated binary at https://matthewbauer.us/nix should have a working copy.

For future reference, this is how it was built:

nix-build -A pkgsStatic.nix
mkdir -p nix
cp ./result/bin/nix nix/nix
cp -r ./result/share nix/share
cd nix
chmod 755 share/ share/nix/ share/nix/corepkgs/
rm -rf share/nix/sandbox
chmod 644 nix share/nix/corepkgs/*
chmod a+x nix
tar cfz nix.tar.gz nix share/
arx tmpx ./nix.tar.gz // './nix "$@"' > ./nix.sh
cp ./nix.sh ~/matthewbauer.github.io/nix

If you don't want to download/extract anything each time, you should run:

mkdir -p $HOME/bin
cd $HOME/bin
curl https://matthewbauer.us/nix > nix.sh
sh nix.sh --extract
mv dat/* .
rm -rf dat run env
export PATH=$HOME/bin
nix run ...

Install size is ~11M.

@ryantm

This comment has been minimized.

Copy link
Member

ryantm commented Feb 24, 2019

Works now for me!


curl = super.curl.override {
gssSupport = false;
};

This comment has been minimized.

@bhipple

bhipple Feb 24, 2019

Contributor

I would imagine your target audience for this is users who are running and deploying on rootless "enterprise" environments such a RHEL or CentOS. It's worth noting that in many of these setups, kerberos is used extensively for all HTTP and git authentication, so if you could get it working in this static variant it'd probably be worthwhile.

This comment has been minimized.

@matthewbauer

matthewbauer Feb 24, 2019

Author Member

Yeah this is an issue. libkrb5 doesn't have good way to link statically, however:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=439039

kerberos heimdal might support it though.

};

nix = super.nix.overrideAttrs (_: {
NIX_LDFLAGS = "-lssl -lbrotlicommon -lssh2 -lz -lnghttp2 -lcrypto";

This comment has been minimized.

@bhipple

bhipple Feb 24, 2019

Contributor

I think the right way to do this would be patching the pkg-config files to include these libraries in the Requires.private section, as is done in:

$ nix-build -A brotli.dev && cat result-dev/lib/pkgconfig/libbrotlidec.pc
prefix=/nix/store/7i965pyrkqph3gwpfzswz35c6a1j1v5h-brotli-1.0.7
exec_prefix=/nix/store/7i965pyrkqph3gwpfzswz35c6a1j1v5h-brotli-1.0.7
libdir=/nix/store/iahrxydkznvf6z0q0hp0cp9nhrq1pkys-brotli-1.0.7-lib/lib
includedir=/nix/store/1qcykjlmxvvzgvvnyzsz7yii794m5rjv-brotli-1.0.7-dev/include

Name: libbrotlidec
URL: https://github.com/google/brotli
Description: Brotli decoder library
Version: 1.0.7
Libs: -L${libdir} -lbrotlidec
Requires.private: libbrotlicommon >= 1.0.2
Cflags: -I${includedir}

I haven't looked at this specific case too closely, but in general invoking pkgconfig --static <args> will also include the Requires.private sections for transitive link dependencies that are not needed in the dynamic variant.

For example, the libssl.pc has a Requires.private directive on libcrypto, which means you don't need to include -lcrypto on the linkline manually if you use pkgconfig --static ssl.

This comment has been minimized.

@matthewbauer

matthewbauer Feb 24, 2019

Author Member

I think the issue was that all of these need to be propagated in static. static linking requires knowing where all of the needed libraries are.

@matthewbauer matthewbauer changed the title Statically built Nix [wip] Statically built Nix Feb 24, 2019

@matthewbauer matthewbauer changed the base branch from master to staging Feb 24, 2019

@edolstra

This comment has been minimized.

Copy link
Member

edolstra commented Feb 25, 2019

In principle this looks fine to me, but it clutters Nixpkgs with numerous patches and special-case attributes that are going to be a pain in the ass to maintain. Who is going to be responsible for maintaining those?

@ryantm

This comment has been minimized.

Copy link
Member

ryantm commented Feb 25, 2019

Who is going to be responsible for maintaining those?

One possible part of the solution would be for @matthewbauer to add himself as a maintainer of these packages.

name = "boehm-gc-7.6.0-sys_select.patch";
url = "https://gitweb.gentoo.org/proj/musl.git/plain/dev-libs/boehm-gc/files/boehm-gc-7.6.0-sys_select.patch?id=85b6a600996bdd71162b357e9ba93d8559342432";
sha256 = "1gydwlklvci30f5dpp5ccw2p2qpph5y41r55wx9idamjlq66fbb3";
}) ] ++

This comment has been minimized.

@dtzWill

dtzWill Feb 25, 2019

Contributor

Is this still needed? It was explicitly removed in a recent-ish PR with the idea that this didn't hurt but didn't help, IIRC. Might be good to check.

@@ -44,6 +44,7 @@ rec {
then throw "Cannot build fully static binaries on Darwin/macOS"
else stdenv'.mkDerivation (args // {
NIX_CFLAGS_LINK = toString (args.NIX_CFLAGS_LINK or "") + " -static";
separateDebugInfo = false;

This comment has been minimized.

@dtzWill

dtzWill Feb 25, 2019

Contributor

This doesn't seem right, why must this be disabled for static adapter?

This comment has been minimized.

@matthewbauer

matthewbauer Feb 27, 2019

Author Member

Yeah maybe not... I was getting an issue that I thought was due to this but it might have been unrelated.

@matthewbauer

This comment has been minimized.

Copy link
Member Author

matthewbauer commented Feb 25, 2019

Yeah i can do that. Lots of these are because of ad hoc configure scripts (no autotools, cmake, or meson). Other times static libs are not being installed correctly into prefix. All of these can probably be corrected with upstream patches though.

@@ -13,6 +13,10 @@ stdenv.mkDerivation rec {

nativeBuildInputs = [ cmake ];

postPatch = stdenv.lib.optionalString stdenv.hostPlatform.isMusl ''
sed -i "24ireturn()" cmake/AwsTestHarness.cmake

This comment has been minimized.

@orivej

orivej Feb 25, 2019

Contributor

This is fragile. In general, a better approach is to generate a patch with diff -U1.

This breaks non-musl configure with:

CMake Error at tests/CMakeLists.txt:260 (target_compile_options):
  Cannot specify compile options for target "aws-c-common-tests" which is not
  built by this project.

Why does it work with musl?

You could try -DBUILD_TESTING=OFF instead.

This comment has been minimized.

@matthewbauer

matthewbauer Feb 25, 2019

Author Member

Yeah it’s super broken. BUILD_TESTING doesn’t work at least

This comment has been minimized.

@orivej

orivej Feb 25, 2019

Contributor

I see, BUILD_TESTING is not actually implemented. You could patch out add_subdirectory(tests) in the main CMakeLists.txt then.

@illegalprime

This comment has been minimized.

Copy link
Contributor

illegalprime commented Mar 2, 2019

This looks really cool, it would help people (me) run nix on ARM with limited memory / disk space (by downloading a static nix and loading some external disk space and setting the nix store inside there).
Is this the right derivation to build?

nix build -f . pkgsStatic.pkgsCross.aarch64-multiplatform.nix
@matthewbauer

This comment has been minimized.

Copy link
Member Author

matthewbauer commented Mar 2, 2019

I don't think that will work - it will build a static toolchain and then cross compile with it. Ideally, you would pass these args to Nixpkgs:

nix build -f. \
  --arg crossSystem '{ config = "aarch64-unknown-linux-musl"; }' \
  --arg crossOverlays '[ (import ./pkgs/top-level/static.nix) ]'

But I don't think crossOverlays is exposed yet.

@illegalprime

This comment has been minimized.

Copy link
Contributor

illegalprime commented Mar 5, 2019

Thanks for this PR @matthewbauer ! After 220 derivations & a couple of days on my small ARM board, I've got a statically compiled armv7l nix!

nix-2.2-armv7l.tar.gz

I did have to edit the bash derivation though:

diff --git a/pkgs/shells/bash/4.4.nix b/pkgs/shells/bash/4.4.nix
index e5e33c7..a61801a 100644
--- a/pkgs/shells/bash/4.4.nix
+++ b/pkgs/shells/bash/4.4.nix
@@ -28,7 +28,7 @@ stdenv.mkDerivation rec {
     sha256 = "1jyz6snd63xjn6skk7za6psgidsd53k05cr3lksqybi0q6936syq";
   };
 
-  hardeningDisable = [ "format" ];
+  hardeningDisable = [ "format" ] ++ optional (stdenv.hostPlatform.libc == "musl") "pie";
 
   outputs = [ "out" "dev" "man" "doc" "info" ];

now I can run it but nix seems to ignore the --store argument (it tries to find /nix), maybe it can't setup the chroot and defaults to the normal store, but I have this:

$ zcat /proc/config.gz | egrep 'USER_NS|SECCOMP'
CONFIG_USER_NS=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_SECCOMP=y

$ sudo mkdir -p /nix/store

$ ./nix run --store ~/my-nix nixpkgs.nix nixpkgs.bashInteractive
error: getting status of '/nix/store/3azwf31ib816jq403qrymzw02hrlj5fk-nix-2.2-armv7l-unknown-linux-musleabihf': No such file or directory

$ ./nix show-config | grep '/nix/store'
build-hook = /nix/store/3azwf31ib816jq403qrymzw02hrlj5fk-nix-2.2-armv7l-unknown-linux-musleabihf/libexec/nix/build-remote
sandbox-paths = /bin/sh=/nix/store/l31h4is1irdw1zfld2qf7l0l4w98ylrh-busybox-1.30.1-armv7l-unknown-linux-musleabihf/bin/busybox

i'll keep poking around, thought i'd post the built binary here if it helps anyone

@matthewbauer

This comment has been minimized.

Copy link
Member Author

matthewbauer commented Mar 5, 2019

You need to provide a Nix channel / file for it to use. So most likely you need something like this:

export NIX_DATA_DIR=$(nix-build '<nixpkgs>' -A nix)/share
./nix run --store ~/my-nix -f channel:nixpkgs-unstable nixpkgs.nix nixpkgs.bashInteractive

Both the --store and -f flags would be nice to provide defaults ofr.

NIX_DATA_DIR is needed to pick up some "booter" expressions in corepkgs. You can use the native Nix for this to avoid rebuliding.

@illegalprime

This comment has been minimized.

Copy link
Contributor

illegalprime commented Mar 5, 2019

@matthewbauer that worked! i'm currently building with a static nix in a custom nix-store 🎉 !
but in order to do that I did have to include these files in my root's /nix/store, even though there's no other nix files present on the system:

$ tree /nix/store
/nix/store
├── 3azwf31ib816jq403qrymzw02hrlj5fk-nix-2.2-armv7l-unknown-linux-musleabihf
│   └── libexec
│       └── nix
│           └── build-remote -> /home/debian/static-nix/bin/nix
└── l31h4is1irdw1zfld2qf7l0l4w98ylrh-busybox-1.30.1-armv7l-unknown-linux-musleabihf
    └── bin
        ├── ash
        ├── busybox
        └── sh

$ ls my-custom-nix-root/nix/store | wc -l
814

the files in my /nix/store also have incorrect permissions (they're writable) so it looks like it's just a hard-coded path in the static nix.

so, the static nix is fully portable except that it references those two paths.
maybe a unit test that checks that no buildInputs are runtime dependencies?

EDIT:
I think patching nix can solve it (might not be a great solution):

diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc
index 1c2c0871..34a08756 100644
--- a/src/libstore/globals.cc
+++ b/src/libstore/globals.cc
@@ -64,7 +64,7 @@ Settings::Settings()
     }
 
 #if defined(__linux__) && defined(SANDBOX_SHELL)
-    sandboxPaths = tokenizeString<StringSet>("/bin/sh=" SANDBOX_SHELL);
+    sandboxPaths = tokenizeString<StringSet>("/bin/sh=" + getEnv("NIX_SANDBOX_SHELL", SANDBOX_SHELL));
 #endif
 
     allowedImpureHostPrefixes = tokenizeString<StringSet>(DEFAULT_ALLOWED_IMPURE_PREFIXES);

and then using NIX_LIBEXEC_DIR to point to the relative libexecdir

@matthewbauer matthewbauer force-pushed the matthewbauer:static-nix branch from 1857278 to 4fb2125 Mar 13, 2019

@matthewbauer matthewbauer requested a review from Ericson2314 as a code owner Mar 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.