remote builder (using qemu for aarch64)?

[bbigras]

I think many packages need to be compiled on my phone when I update. I takes a while and use my battery.

Anyone uses a remote builder? I wonder how to set up a remote builder using qemu to build the aarch64 packages for android on my x86_64 machine.

[t184256]

I don't, and I'd like to learn how to do it as well.

Note, that the performance likely won't be great with QEMU ARM emulation either. I wouldn't be surprised if faster IO due to the lack of proot would be the deciding factor, and not the CPU performance.

[bbigras]

I'm testing with some qemu overlay. Any way to make nix-on-droid pass --builders to nix-build? or define nix.buildMachines in some config?

EDIT: I was able to set the builder with ~/.config/nix/nix.conf

I might have a working setup:

I'll post back as soon as I see if the build failed or not.

[bbigras]

I think it's working but really really slow.

I'm using the qemu overlay from https://github.com/bqv/nixos (thanks to qy on the matrix channel).

qemux.nix: https://github.com/bqv/nixos/blob/live/profiles/misc/qemu.nix
overlay: https://github.com/bqv/nixos/tree/live/overlays/qemu

I'm creating a "builder" user on the builder machine.

builder host configuration.nix

let
  qemuOverlay = (import ./overlays/qemu);
in
{
  imports = [ ./qemu.nix ];

  boot.kernelModules = [ "kvm-intel" ];
  qemu-user.arm = true;
  boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

  nix = {
    trustedUsers = [ "bbigras" "builder" ];
  };

  users.users.builder = {
    createHome = true;
    isNormalUser = true;
    # I should probably add the ssh pub key here
  };
}

on Android:
~/.config/nix/nix.conf

builders-use-substitutes = true
builders = ssh://builder

~/.ssh/config

Host builder
    HostName <the ip>
    User builder
    IdentitiesOnly yes
    IdentityFile ~/.ssh/nix_remote

~/.ssh/nix_remote is an ssh key without a password. I added the pub key on the builder machine.

you need to ssh first to the host to make ssh accept the host key.

You can test with nix build '(with import <nixpkgs> { }; runCommand "foo" {} "uname -a > $out")' --builders 'ssh://builder' -j0

or nix-on-droid switch --max-jobs 0

[t184256]

Cool. Could you please add this to https://github.com/t184256/nix-on-droid/wiki, so that it ain't buried in a closed issue?

[bbigras]

Cool. Could you please add this to https://github.com/t184256/nix-on-droid/wiki, so that it ain't buried in a closed issue?

Yes no problem.

I have a question. Why bat needs to be built when running nix-shell -p bat on Android but it can be fetched from the cache when I build it manually on the builder machine? (on the 20.03 tag of nixpkgs).

❯ nix-build -A bat --argstr system aarch64-linux
these paths will be fetched (1.76 MiB download, 5.14 MiB unpacked):
  /nix/store/q97pn1hp62yik0s9ci3as5n0pabz8rsw-bat-0.12.1
copying path '/nix/store/q97pn1hp62yik0s9ci3as5n0pabz8rsw-bat-0.12.1' from 'https://cache.nixos.org'...

[t184256]

I'm afraid I don't know.

[Gerschtli]

Maybe its due to the bionic build env? I don't really understand that in detail but wasn't this the reason proot hash is different when compiled on the build host vs the phone? @t184255

[t184256]

No, it's not. Inside proot we don't use bionic.

[t184256]

@bbigras, is it possible to check if your builder would use the cache if you use same channel url? And will the hashes match?

[bbigras]

Sorry it took me a whole month to reply. I just added the stuff to the wiki.

is it possible to check if your builder would use the cache if you use same channel url? And will the hashes match?

Like this?

❯ cd ~/nixpkgs
❯ git checkout 20.03
❯ nix-build -I nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/nixos-20.03.tar.gz -A bat --argstr system aarch64-linux
these paths will be fetched (8.48 MiB download, 37.53 MiB unpacked):
  /nix/store/2sgk68v38wj5fc264b8nl2znz8j1ijf8-ncurses-6.1-20190112
  /nix/store/2vn35phq7d4q3q6pi6jipxy1h1x3346s-glibc-2.30
  /nix/store/lm6rckx5z0j3qr5y2a63figa61a6n6p4-less-551
  /nix/store/q97pn1hp62yik0s9ci3as5n0pabz8rsw-bat-0.12.1
  /nix/store/qa79fvrbxrgxx9pl1yqjmv82183z6n4s-bash-4.4-p23
copying path '/nix/store/2vn35phq7d4q3q6pi6jipxy1h1x3346s-glibc-2.30' from 'https://cache.nixos.org'...
copying path '/nix/store/qa79fvrbxrgxx9pl1yqjmv82183z6n4s-bash-4.4-p23' from 'https://cache.nixos.org'...
copying path '/nix/store/2sgk68v38wj5fc264b8nl2znz8j1ijf8-ncurses-6.1-20190112' from 'https://cache.nixos.org'...
copying path '/nix/store/lm6rckx5z0j3qr5y2a63figa61a6n6p4-less-551' from 'https://cache.nixos.org'...
copying path '/nix/store/q97pn1hp62yik0s9ci3as5n0pabz8rsw-bat-0.12.1' from 'https://cache.nixos.org'...
/nix/store/q97pn1hp62yik0s9ci3as5n0pabz8rsw-bat-0.12.1

[bbigras]

On Android bat seems to come from the cache right now. I'll check if it's the same hashes.

[bbigras]

the hash looks different on Android.

I'll try without ~/nixpkgs later. Meanwhile, don't hesitate if you want me to run some commands.

[bbigras]

If I run nix-env -f https://github.com/NixOS/nixpkgs/archive/48723f48ab92381f0afd50143f38e45cf3080405.tar.gz -i bat on my phone and nix-env -f https://github.com/NixOS/nixpkgs/archive/48723f48ab92381f0afd50143f38e45cf3080405.tar.gz -i bat --argstr system aarch64-linux it seems to fetch same thing from the cache.

Maybe the problem with bat is gone.

Is there any way to know which derivations need to be built?

[bbigras]

If we forget the remote builder for a sec.

If I want run nix-on-droid but only use cached stuff, no local build, can I use --max-jobs 0 or there's at least 1 derivation that need to be build locally. Like for my config.

[Gerschtli]

Try to run nix-build "<nix-on-droid/modules>" -A activationPackage and ctrl-c the command after the list of to be built packages is shown, if thats what you want.

[bbigras]

Thanks everyone.

My nix-on-droid doesn't seem to want to build stuff currently. Maybe I was on a different channel before I wiped to make openssh work. I'll open a new issue if I see it again in the future.

[573]

Hi @bbigras,

what you had set up there is exactly the use case me and sure many others aim for.

I wanted to try this as well. To better get a grip how things relate and since I do not own a machine with NixOS installed and hence have no configuration.nix file anywhere I thought I would do that in a vm with NixOS installed inside first, meaning what is the remote builder in your wiki example would be a qemu (aarch64-linux as on my phone) vm running NixOs. So far I have in a folder remote-builder-nix:

# EDIT: I removed the parts not in your configuration.nix, to no avail
# vm-config.nix from the link above modified
let
  qemuOverlay = (import ./overlays/qemu);
in
{
  imports = [
    ./qemu.nix
  ];

  config = {
    boot.kernelModules = [ "kvm-intel" ];

    qemu-user.aarch64 = true;
    boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

    nix = {
      trustedUsers = [ "573" "builder" ];
    };

    users.users.builder = {
      createHome = true;
      isNormalUser = true;
    };
  };
}

vm.nix as in my link above as well as the overlay dir and qemu.nix from your links given.

To build the vm I would now i. e. run:

$ nix-build '<nixpkgs/nixos>' -A vm -I nixos-config=vm.nix

but that gives me

error: attribute 'qemu-user-arm64' missing, at /home/573/remote-builder-nix/qemu.nix:11:22

How is that ?

EDIT: qemu-user.arm = true; above had to be qemu-user.aarch64 = true; for that error to disappear.

[573]

Building according to the steps given just works assumed you install the binfmt wrapper on the system powered by the remote machine, the system being whatever you like. In case of a non-NixOS remote you also safely would ignore the configuration.nix, qemu.nix and overlay changes.

Consider my previous comments describing a niché case where you tried to let that remote machine just be a virtual one (running in qemu).

FYI, I leave a tracking link here, where I discuss approaching a working template.

[573]

Sorry for the gruft added by the last two posts.
I added a wiki entry with my findings, spoiler it works now in a vm on the same metal/machine.

[t184256]

I find the page confusing, TBH. It elaborates on https://github.com/t184256/nix-on-droid/wiki/Use-a-remote-builder-with-qemu, and enhances it with a way to spawn such builder VMs on anything Nix-capable, right? Could you somehow make it more clear in the introduction?

Also, wow, let's offload compilation from Nix-on-Droid to WSL1. You, sir, win a unique setup award.

[573]

@t184256 thx I'm honored lol.

But yes indeed using this approach I'm able to compile packages for nix-on-droid the aarch64 architecture on a vm spawned as you said exactly. the vm running nixos but the metal spawning the vm just using nix which latter part is just conveniency.

EDIT: the point is that to achieve what the other article describes on a Windows-operated machine i. e. spawning qemu-user I need some virtualization approach i can't access certain ressources outside of a vpn due to disabled dns splitting and so I had to use this quasi dns-agnostic approach. side effect, finally there is a machine running NixOS for me as well.

I am trying to make the article more comprehensive with a better intro soon. Thanks for your input.

[t184256]

Thanks for following through! I definitely like the new version better.