[tracking] aarch64 build on Packet Type 2A

[vielmetti]

Issue description

A tracking issue to work on a release of NixOS to the Packet "Type 2A" Cavium ThunderX servers.

Steps to reproduce

Log in to Packet, http://packet.net . Try to provision a Type 2A server with NixOS, notice that NixOS is missing as an operating system choice.

Technical details

• System: Packet Type 2A https://www.packet.net/bare-metal/servers/type-2a/ (Cavium ThunderX dual socket 96-core ARMv8)

See also

Packet Community Slack #worksonarm channel

The issue on the Packet Trello board is https://trello.com/c/Rv5YQb7T/127-support-nixos-on-armv8

[dezgeg]

As for image building, here's my recollection of what needs to be done:

• Figure out how to run KVM virtual machines on those servers
• Figure out whether to use make-disk-image.nix or my sd-image.nix image building infra
  • make-disk-image.nix (I expect this is the better solution):
    • depends on KVM
    • also depends on lkl which has bunch of arch-specific stuff
    • can already build images that do legacy booting with GRUB on x86, presumably easy to convert to EFI grub
  • sd-image.nix
    • doesn't need KVM
    • doesn't do GRUB (currently)
    • grub-install (even for EFI) might not like being run without root
    • the GRUB config generator probably doesn't like being run in Nix builder context without major surgery
      - in theory could just hand-write an initial grub.cfg, but sounds like maintenance disaster
• Figure out what options to make the GRUB output its menu over the serial console for recovering (IIRC it didn't with what NixOS currently writes to grub.cfg)

[copumpkin]

We could probably kill the KVM dependency in make-disk-image.nix with a bit more effort. I think the OpenWRT project figured out how to convince grub to install to an image, and if necessary we could also submit a patch upstream. See here and here.

[copumpkin]

grub-install (even for EFI) might not like being run without root

TBC, grub-install is the only reason make-disk-image uses KVM. It's the only thing I wasn't able to do in a pure builder.

[dezgeg]

Note that I expect EFI GRUB installation to be easier as non-root than legacy GRUB installation, given how much 'simpler' it is. There you just write the files to a filesystem and you are done. In legacy mode, you write some stuff to the MBR, some raw sectors to the unpartitioned space between the MBR and the first filesystem, then some files to the filesystem again... with all those three parts somehow co-operating, which sounds like it would require determining whether the /boot filesystem is on the same device that you run grub-install on...

With that said, wow https://dev.openwrt.org/browser/trunk/target/linux/x86/image/Makefile?rev=49407#L69 indeed does look promising.

[bjornfor]

Another alternative is genimage. Buildroot uses it to create MBR and EFI bookable disk images. No root or VM needed.

[dezgeg]

Hmm, crap. LKL gives:

make: Entering directory '/tmp/nix-build-lkl-2017-08-09.drv-0/linux-083cdeece0577635d523244dcf0da86074e23e4e-src/tools/lkl'
Makefile:82: *** Unrecognized platform: elf64-littleaarch64.  Stop.

Edit: Ignore that, I tried to build and old lkl as I failed to upgrade nixpkgs. Latest LKL builds fine.

[grahamc]

PR here incorporates some netboot patches I've had kicking around: #32031

After that PR merges, we should recycle the branch for further testing, as https://hydra.nixos.org/jobset/nixos/unstable-aarch64 is setup already.

[dezgeg]

It's somehow related to the GIC virtualization options that need to be set on that hardware: https://bugzilla.redhat.com/show_bug.cgi?id=1375571.

[dezgeg]

Yep. And with gic-version=host it also works on machines with GICv2 (such as the Jetson TX1). I wonder who decided the default value for that option... Anyway, done in b2e315f

[grahamc]

Long done =)