Exec format error when trying to switch NixOS (16.09) architecture

[djvdorp]

Issue description

Since I found out that the i686-linux NixOS version (16.09) has quite a lot less pre-built binary available (thanks to this post), and this specific machine supports x86_64-linux, I decided to try to switch architecture.

When doing this, I am getting an error, likely because I am doing something wrong, but then what is the proper way to switch from i686 to x86_64?

The error seems to be thrown because it is trying to do something with x86_64 binaries already, while the system (and kernel) is still booted in i686 mode? The only information related to this I could find was this message.

Steps to reproduce

1. Change nixpkgs.system from i686-linux to x86_64-linux (see: djvdorp/nc110-nixos@ed7a572)
2. Run nixos-rebuild switch.
3. Get error: [..]/nix-instantiate: cannot execute binary file: Exec format error

Technical details

• System: 16.09.680.4e14fd5 (Flounder)
• Nix version: nix-env (Nix) 1.11.4
• Nixpkgs version: 16.09.741.354fd37

[djvdorp]

From the NixOS IRC channel, via @rbvermaa:

<rmrfroot> djvdorp: not sure about this, but i don't think you will be able to just switch architectures like that for the whole system.
<rmrfroot> djvdorp: since your kernel is 32-bit you will not be able to run a 64-bit executable on it.
<djvdorp> @rmrfroot I am also very curious if it is possible, maybe one needs a step in-between (like just building new initramfs, kernel etc) and booting 64-bit before proceeding? Like here: http://nixos.org/nixos/manual/index.html#sec-building-parts
<ikwildrpepper> djvdorp: you can probably use nixos-rebuild boot, and then boot
<ikwildrpepper> (see 'man nixos-rebuild)
<djvdorp> ikwildrpepper: thanks, going to check that out, I am very curious if it is even possible to just switch architectures like this
<ikwildrpepper> djvdorp: you can, but not without booting
<rmrfroot> ikwildrpepper: cool! do you know if there are any documentation about switching architecture like that?

[NeQuissimus]

While I have nothing much to add, I would suggest we document some of the info gathered here.

[djvdorp]

I just tried to do exactly what @rbvermaa suggested earlier (if I understand it correctly) but am hitting the same error again. Am I missing a step here? I have also read the man nixos-rebuild as suggested.

Steps to reproduce

1. Change nixpkgs.system from i686-linux to x86_64-linux (see: djvdorp/nc110-nixos@ed7a572)
2. Run nixos-rebuild boot.
3. Get error:

building Nix...
/nix/store/wrnaqa0r2znx9vfnam843bgssar3ilzy-nixos-rebuild/bin/nixos-rebuild: line 317: /tmp/nixos-rebuild.Wheacu/nix/bin/nix-instantiate: cannot execute binary file: Exec format error
building the system configuration...
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 3: use: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 4: use: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 5: use: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 6: use: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 7: use: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 8: use: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 9: use: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 10: use: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 12: binmode: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 14: my: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 15: my: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 16: my: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 17: my: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 18: my: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 19: my: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 20: my: command not found
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 22: syntax error near unexpected token `('
/tmp/nixos-rebuild.Wheacu/nix/bin/nix-build: line 22: `my @instArgs = ();'

1. Wanted to do reboot afterwards as suggested, but never got this far because of the error.

[djvdorp]

Not getting further yet, but also mentioning @Fuuzetsu here since he seems to have suggested this same approach in the past to another user with the same question @magnetophon.

I think @svanderburg might also have an idea on the matter, since his blogposts on Nix(OS) seem to contain very deep knowledge on the subjects covered there that seem a bit related.

[RonnyPfannschmidt]

when looking at the output, my uneducated guess is
that the new system is used to update grub, and of course breaks apart due to lacking a kernel it can run on

the indicator for me is the second line in the last output, an exec failed with the wrong format

[djvdorp]

Keeping track of useful information from today's NixOS IRC channel below:

<ikwildrpepper> djvdorp: nixos-rebuild boot, and then rebooting into the new configuration didn't work?
<djvdorp> ikwildrpepper: no, unfortunately not, I updated the issue with what you suggested + what the outcome was, but comes down to (what seems to be) the same error
<ikwildrpepper> djvdorp: what does this command return? echo "builtins.currentSystem" | nix-instantiate --eval-only -
<djvdorp> ikwildrpepper: will let you know right away when I get access to that computer again, don't have it near me now unfortunately :(
<ronny> djvdorp: i fear that the build process tries to use the new system (that can not run) for the system build
<ronny> djvdorp: i think a workaround is needed where you pull a 64 bit kernel into a 32 bit system, but i have absolutely no idea how to do that
<djvdorp> ronny: that is also the feeling I was getting, but I have no idea on how to pull a 64 bit kernel into my 32 bit system first as well, tried finding that out last night but did not succeed
<ronny> djvdorp: i wonder if you could just set boot.kernelPackages to an imported 64 bit kernel
<djvdorp> ronny: https://nixos.org/nixos/manual/#sec-kernel-config seems to insinuate we can, maybe "boot.kernelPackages = pkgs.linuxPackages_4_4.kernel.x86_64-linux;" would do so? I guess it is currently (implicitly) set to "boot.kernelPackages = pkgs.linuxPackages_4_4.kernel.i686-linux;"
<ronny> djvdorp: good find, please try that and see if you can make a i686 system with a 64 bit kernel
<djvdorp> ronny: that would be just doing that change, and reverting back to "nixpkgs.system = "i686-linux";", then nixos-rebuild boot, reboot, and after that trying to change nixpkgs.system to x86_64-linux again?
<ronny> djvdorp: correct
<ikwildrpepper> djvdorp: an alternative would be to just do nixos-install via a 64 bit ISO
<ikwildrpepper> although, I am not sure if that keeps the previous grub entries
<djvdorp> ikwildrpepper: that is absolutely true, but my curiosity is getting the better part of me and I am now mostly curious if it is possible so I can try to document it for others
<ikwildrpepper> djvdorp: :)
<djvdorp> ikwildrpepper: previous grub entries and such are not really a problem, since this computer is mainly my hobby machine to try out new distro's and software and solving these kind of puzzles :)
<djvdorp> ikwildrpepper: thanks to the power of NixOS and my config on github, it will be pretty trivial to do a complete re-install and get back the same system config :)

[djvdorp]

@rbvermaa / ikwildrpepper:

echo "builtins.currentSystem" | nix-instantiate --eval-only -
returns: 
"i686-linux"

@RonnyPfannschmidt / ronny:

setting "boot.kernelPackages = pkgs.linuxPackages_4_4.kernel.x86_64-linux;" in configuration.nix fails with:
attribute `x86_64-linux` missing at /etc/nixos/configuration.nix

This is probably because I am trying to specify the package architecture with a dot ('kernel.x86_64-linux') and there is no such package. I could just specify  "boot.kernelPackages = pkgs.linuxPackages_4_4.kernel"; but that would not change anything since this would still get the 4.4 kernel for i686-linux

[vcunat]

I'd personally try nixos-rebuild boot --no-build-nix, but it's possible it will blow in a later stage than evaluation with target nix (64-bit one). It might be easiest to just boot from a 64-bit ISO and run nixos-install.

[djvdorp]

@vcunat thank you for your input and suggestion, I have just tried this out and it does indeed blow in a later stage:

nixos-rebuild boot --no-build-nix --show-trace (which is equivalent to nixos-rebuild --fast)

root@nc110> nixos-rebuild switch --no-build-nix --show-trace                                                                                                /home/daniel
/nix/store/wrnaqa0r2znx9vfnam843bgssar3ilzy-nixos-rebuild/bin/nixos-rebuild: line 318: /nix/store/28wl3f34vfjpw0y5809bgr6382wqdscf-bash-4.3-p48/bin/bash: cannot execute binary file: Exec format error
building the system configuration...
[..]
these paths will be fetched (37.02 MiB download, 183.15 MiB unpacked):
  /nix/store/207fl2bc6n07lkqa35j4lclwrhwyci5y-zsh-5.2
  /nix/store/fpk8qvy8adjz0ydmcl6xg2b4qjd91362-NetworkManager-pptp-gnome-1.2.4
  /nix/store/g3dgg5bjkjs3mwnkkmlm4dm64z8iyzfn-NetworkManager-vpnc-gnome-1.2.4
  /nix/store/g5bz1qkxsrl2i5g6gw8m2jlkd9i0qpw1-NetworkManager-openvpn-gnome-1.2.6
  /nix/store/imrzrmgzwnav7sa7sa7zbrc1cyhnkwa9-vlc-2.2.4
  /nix/store/phc5w5hal2zgp26lshm6p37qxndjm39a-NetworkManager-l2tp-gnome-1.2.4
  /nix/store/vzzhlwfa3k6vsc7c2hcq4wv7isyk937s-midori-0.5.11
  /nix/store/wgr857irq27jv76nslfrwmvsmmpiyj24-webkitgtk-2.12.5
building path(s) ‘/nix/store/mrp2xn1djpjig6rwmd1ban7rh8crqzp7-NetworkManager.conf’
error: a ‘x86_64-linux’ is required to build ‘/nix/store/7g1ag9x6b88k84p8p4d7fpwb5ipch16h-NetworkManager.conf.drv’, but I am a ‘i686-linux’

At this point I am starting to be convinced that my only remaining option is going to be to just boot from a 64-bit ISO and run nixos-install, but I am thinking about how to do that without losing my configuration and homedirs (it is quite a while ago I did the initial nixos-install for this computer).

I was quite convinced this could/should be possible and would still be highly interested if there is any chance to make it work (mainly out of curiosity and for the science of it).

[vcunat]

You won't lose any of that.

[vcunat]

There's a problem that even config files are built with 64-bit tools (purity!), so you would at least need some x86_64-linux build slave.

[djvdorp]

@vcunat thanks again, I have just successfully made the switch from i686-linux to x86_64-linux by following these steps:

1. Download nixos-minimal-16.09.741.354fd37-x86_64-linux.iso and then dd it to my usb drive
2. Reboot system from usb drive, ensure configuration.nix now says nixpkgs.system = "x86_64-linux"; instead of nixpkgs.system = "i686-linux";
3. Mount the root file system under /mnt
4. Make sure internet works, either wireless or wifi
5. Run nixos-install, set the root password, reboot
6. System reboots into working x86_64 kernel while keeping the previous grub entries untouched (also homedir and such are untouched), great success!

cc @rbvermaa @RonnyPfannschmidt