Support for binfmt_misc in build sandboxes

[Mic92]

Since we now get more and more software cross-compiled it will be useful
to also emulate non-native architectures to work-around unwilling buildsystems
and for testing. The kernel supports user-defined binfmt_misc magics, but
this can be only done as root and globally at the moment.
However there are promising patches for the linux kernel proposed that could solve this in future: https://patchwork.kernel.org/cover/10634807/
In short this would allow to register per build sandbox custom binary formats
to start qemu/wine emulators as needed. This feature will not require root privileges.

[Mic92]

cc @cleverca22, since you have the following snippet in your repo: https://github.com/cleverca22/nixos-configs/blob/18baf13c0fd1eac3516cd0a39481108077ac7654/qemu.nix

[shlevy]

I think this can only work if its reflected in the derivation that it's running under a particular binfmt_misc interpreter. In @cleverca22 's patch, the derivations pretend to be native versions of the target, which is definitely not a reasonable assumption.

[shlevy]

If that's covered and it doesn't break the sandbox, sounds like a great addition IMO.

[Mic92]

Yes could be some extension of derivation if the kernel would get binfmt namespace for that.

[Mic92]

This feature could also allow to run builds in qemu even for native execution if the binaries are not trusted. Overhead should be ok with KVM. I just read this does not improve security.

[matthewbauer]

The problem with binfmt_misc is that it requires Linux kernel configuration for it to work. There's no real way to force everyone to have it configured properly. I don't think there is anything similar in other kernels either. It also comes with a little too much "magic" in my opinion. A builder should only be able to execute from one architecture, adding more could lead to bad behavior (for instance using the cross compiler when the native compiler should be used).

I recently merged something that I think avoids the magic part of binfmt_misc. It is just a generic "emulator" that you can use to get a qemu-user for any platform. It can be used like this:

{ stdenv, buildPackages }:
stdenv.mkDerivation {
  buildCommand = ''
    ${stdenv.hostPlatform.emulator buildPackages} cross-binary ...
  '';
}

This makes the "magic" a little more explicit. It can use either qemu-user or wine depending on what stdenv.hostPlatform is.

[Mic92]

Does the emulator also work if a process forks and execute a different non-native binary? Regarding the kernel configuration we still have to wait if that is become the default, which is what I expect. All current implemented namespaces are enabled by default. We also will not be able to use it for some time in nixpkgs, but that does not mean it is not usable for anybody.
You can already pick the wrong compiler there is nothing that prevents you from that. I can also think of derivations that would only support a certain infrastructure, which allows people to actually test the result properly.
Making it configurable per derivation would also make it rather explicit.
I am not advocating we should use it everywhere in nixpkgs.
However if someone has to specify an emulator itself this requires patching again, which is something people want to avoid, when they want to get the job done.

[tomberek]

How does this interact with NixOS/nixops#574 ?

[Mic92]

Not at all.

[stale]

I marked this as stale due to inactivity. → More info

[Mic92]

We still don't have binfmt_misc namespaces unfortunately.

[stale]

I marked this as stale due to inactivity. → More info

[Mic92]

Those patches never made into the kernel.