Various changes and fixes with the general aim to support netbooting NixOS #2920

oxij · 2014-06-13T00:22:28Z

by serving NixOS images from a NixOS machine to anything supporting PXE.

The described concept works alright, but the main piece of code is far from being ready for the general public (ugly hacks), but I got tired of rebasing and stashing these changes.

So that `user` mount option would work allowing normal users to mount and umount stuff marked with it in `fileSystems.<name>.options`.

7c6f434c · 2014-06-13T06:19:27Z

pkgs/tools/networking/atftp/default.nix

+  patches = [ debianPatch ];
+
+  postInstall = ''
+    wrapProgram $out/sbin/atftpd --prefix LD_LIBRARY_PATH : ${stdenv.gcc.gcc}/lib${if stdenv.system == "x86_64-linux" then "64" else ""}


Isn't this in RPATH anyway? Maybe it should be fixed by patchelf, not by wrapping?

It should be, I suppose, but for some reason it is not. Expression for wine in nixpkgs has the same problem.

domenkozar · 2014-06-13T11:52:34Z

Which are the ugly hacks you pointed out?

oxij · 2014-06-13T23:06:13Z

@iElectric no hacks included here

domenkozar · 2014-06-25T15:08:41Z

I've cherry-picked de39787, although other changes seem to be fine. Would be nice if someone tries to boot NixOS with PXE boot.

@oxij could you provide some instructions?

oxij · 2014-06-25T21:02:56Z

@iElectric currently it needs a bit hacky kernel configuration and a huge nonmodular blob of nix code I'm not quite ready to share yet. But I'm working on it.

oxij · 2014-06-25T21:06:48Z

Also, @iElectric, could you then please cherry-pick 5e4d0f5 too? Isn't it minor enough?

offlinehacker · 2014-06-25T22:07:40Z

I have been booting nixos with pxe and it works ok, you need to use tarball
elease and do NFS and then configure pxeconfig. It's standard procedure.

On Wed, Jun 25, 2014 at 11:06 PM, Jan Malakhovski notifications@github.com
wrote:

Also, @iElectric https://github.com/iElectric, could you then please
cherry-pick 5e4d0f5
5e4d0f5
too? Isn't it minor enough?

—
Reply to this email directly or view it on GitHub
#2920 (comment).

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.11 (GNU/Linux)

mQENBFEY1PEBCADPOfERF2wo4qeoq9L1m2z4pKfWqNd4B6BsoFUWPNd7BXmY+9JG
jJddSkmYobWec7XjAFTBL0Xbttt+rK9SIED2dCOmU1FYMQElhGlM3PNA3kaiQFeV
ijgH318GCfZzDd0dWa5TN/IshVeWXwgngsIEmZTVf1VSeb3eO3B8Fxe3zsSLUq0b
71MmU5eLVP9pMkm5V5BTYp+lV70FIekKygkKq+uTDo1csWUatbs4Qvgv37Bymy2t
oTwOBXGoinQk5N/6asR1jWs3vKv0L0SruoZy/kEG/jXb4l2OZUP85EVMganYKouE
OchVmcmhBdWV+t3HK4r2ATfyEcMRzvzSflA1ABEBAAG0Jkpha2EgSHVkb2tsaW4g
PGpha2FodWRva2xpbkBnbWFpbC5jb20+iQE+BBMBAgAoBQJRGNTxAhsDBQkB4TOA
BgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRD6Zxi5hZclKnXNCACLOKa8abQp
eTWv9SXUwC7LVM5pP2mXcgn+Ipqr6YWBdLx4Iij0YlvUok9VeKvwTpUlT+cx++o3
wCM3AYrUyJE+zrtw49lInUmutz9seqJLU895oq+D+UuGoORrLBpEZrYR5f83uUmQ
E3Z1ZmWrNGXYtITWDtVZD/KauMF2nkPcmy/XaYXhd4WHD81DGNlKtGAHig6A3Phc
8Mr0A4yLDeRQJm8lCFEsxMJUNTupgY+ybbsMfVGx1gQvvGOTioV8CLCoRchUCCcm
YPArFg40KzIDSNjwdo9EVZDnlPx1hbOppfQydxP+JVnZsqoYmVY4UhIWi/NfOl3V
UMjl338INW1zuQENBFEY1PEBCADRSIfelOMjaTH7IfpMFFUc5Gys//njFnW9QAUg
wyfs2AFxUp6vKQ7nxXQiJXVhKTwe9iqo+oGxaHp4AeTjC7vXsfMuF5g5lfttbAo3
YEobEe6OG5so41nbwan6SyeIIQ2AmQqJBw8TKKMSec2qUN0Pw7iZRs0o9uJM/obG
DPsAsMOQgNLxJyMCP7X2jBtDXxkMFVHMmk50Tl3h3Fi9qWuNxgTXjs0tUvKkXiu2
Pco952jnm7HpCIKBek2pqR/UJXXb5qxy5G6Lc0qaMWZ5GKnSMTJmTY6Xl44EnaLK
zh0rqgF9qpoWck470ZbiGASMtB008hy2l0cyxUfvDaS3tY4hABEBAAGJASUEGAEC
AA8FAlEY1PECGwwFCQHhM4AACgkQ+mcYuYWXJSoT6AgAkvzvC0EGmeCR3cn9O3Gf
yG00Kqk9/1gJvlphis7AAce8iUgU+4xd94Vp0u8rghpdy88xKN5lF1W2YZQmmBaf
AVe6b7TOg6kxc3GKkVsWDxNyQKkpB46BwefIGaSljH7502X9aEWosrqWyJJNYCtt
QDit4BysX0Ww3Ka5Rx6ZFhC9ybPKoW2i8JwpyBaXDt7R2k+PC/ClBf9qzL+sb2es
zh/zCMVKNdm8KUITHU/5lgn2qZpUFZwiASPCMGGFP9u8g6UKeUTYTPD+GWaHIW63
RAgNIAffxx0M1r3P/2ipkAdI3NX/1iBKDQNG8Odsf+BswFKrNCnyUDdLPvJAhODS
gw==
=tmrm
-----END PGP PUBLIC KEY BLOCK-----

oxij · 2014-06-25T22:43:52Z

@offlinehacker Yep, but the goal here is a bit more ambitious (cf. see the first line in the opening comment of the thread).

domenkozar · 2014-06-26T09:13:42Z

@oxij cherry-picked, thanks!

7c6f434c · 2014-08-30T10:06:38Z

I guess we should cherry-pick the rest of non-obsolete changes and close…

Does anyone know what changes are still relevant?

oxij · 2014-08-31T18:18:12Z

@7c6f434c my humble opinion is than every non-applied change from this PR is still relevant

7c6f434c · 2014-09-01T06:17:53Z

What do we need from the Debian atftp patch?

7c6f434c · 2014-09-01T06:28:36Z

And what do we still need now? I tried to edit and commit most of the patches left…

oxij · 2014-09-01T14:07:51Z

@7c6f434c debian pretty much maintains atftpd with their patch, I'd say we need less bugs and IPv6 support. Everything else looks merged in.

7c6f434c · 2014-09-01T20:49:23Z

Please verify that everything is now applied into master and if so, close the pull request.

7c6f434c · 2014-09-03T07:47:32Z

@oxij forgot to put Cc mark and no idea how GitHub handles this

oxij · 2014-09-09T20:26:03Z

@7c6f434c Thanks. To do a proper testing I need to rebase a hefty branch that uses all these changes on top of current master, and that needs (much) more time (than expected) which I won't have at least till the end of the week. So, note that I will eventually reply and do not close this yet.

7c6f434c · 2014-10-09T07:38:22Z

@oxij Any news?

My theory is that I have merged everything from this pull request and that you will have more fixes (not included here yet) once you test.

Maybe close this instance and open a new request when you have more improvements?

oxij · 2014-10-12T19:17:37Z

@7c6f434c three weeks ago I wasn't able to build this, but current master seems to compile and even netboot.
Thanks again. Closing.

wmertens · 2014-10-13T04:58:47Z

Now we need a wiki page describing how to set this up :)

jagajaga · 2014-11-13T19:29:24Z

@oxij Can you shortly describe how to set this up? We'll create a wiki page after that. Or you can do it yourself.

domenkozar · 2014-11-13T20:17:35Z

I'd add this to NixOS manual to have information less fragmented.

jagajaga · 2014-11-13T20:36:59Z

🍻

wmertens · 2014-11-13T22:19:17Z

@oxij if you could just post a configuration file for the server and client
I could document it.

On Thu Nov 13 2014 at 9:37:06 PM Arseniy Seroka notifications@github.com
wrote:

🍻

—
Reply to this email directly or view it on GitHub
#2920 (comment).

oxij · 2014-11-15T00:28:47Z

@wmertens @jagajaga @iElectric a) it doesn't work perfectly; b) it's so ugly I'm embarrassed.

What I do is I generate a pxelinux and/or iPXE configuration files referencing a set of system profiles and nixos expressions for dhcp/atftp/lighttp daemons from types.loaOf types.attrs where each element contains nixos-config and some PXE settings.

PXE and nixos daemon configurations are more or less okay, but the main part that evaluates nixos-configs into netbootable system profiles is so abhorrent I'm seriously surprised that it works. The problem is that netbooting with network-mounted root needs an early network, which I currently provide by a custom kernel config (with all the networking drivers and nfs builtin and "ip=dhcp" parameter), initrd (fs root mounting hack similar to live-cd profile) and some minor hacks (like persistent dhcpcd). Even after all that this doesn't ensure a successful boot, because some network card drivers need firmware blobs which are to be supplied by userspace, but "ip=dhcp" forces the kernel to mount/run initrd only after the network is up.

In a way, I'm waiting for someone to make initrd configuration somewhat more modular so that one could generate a proper initrd with a service which would set the network up before /nix/store is mounted. Starting an init daemon from initrd would be perfect, but even a modular initrd/init script with an easy way to place packages onto initrd would be nice. I tried to implement this in a proper way myself, but after two weeks of non-stop hacking it turned out to be much more work than I was prepared to commit. Currently, I have a setup that has something like 80% chance of working, with 20% of effort (and I'm not happy with that).

What I can probably do with a clear conscience is to transform the PXE part into nixos service, and put everything else into an example configuration profile. Will that be okay?

lucabrunox · 2014-11-15T00:58:05Z

About an init in the initrd, you may be interested in my systemd in initrd
branch: https://github.com/lethalman/nixpkgs/commits/systemd

On Sat, Nov 15, 2014 at 1:28 AM, Jan Malakhovski notifications@github.com
wrote:

@wmertens https://github.com/wmertens @jagajaga
https://github.com/jagajaga @iElectric https://github.com/iElectric
a) it doesn't work perfectly; b) it's so ugly I'm embarrassed.

What I do is I generate a pxelinux and/or iPXE configuration files
referencing a set of system profiles and nixos expressions for
dhcp/atftp/lighttp daemons from types.loaOf types.attrs where each
element contains nixos-config and some PXE settings.

PXE and nixos daemon configurations are more or less okay, but the main
part that evaluates nixos-configs into netbootable system profiles is so
abhorrent I'm seriously surprised that it works. The problem is that
netbooting with network-mounted root needs an early network, which I
currently provide by a custom kernel config (with all the networking
drivers and nfs builtin and "ip=dhcp" parameter), initrd (fs root mounting
hack similar to live-cd profile) and some minor hacks (like persistent
dhcpcd). Even after all that this doesn't ensure a successful boot, because
some network card drivers need firmware blobs which are to be supplied by
userspace, but "ip=dhcp" forces the kernel to mount/run initrd only after
the network is up.

In a way, I'm waiting for someone to make initrd configuration somewhat
more modular so that one could generate a proper initrd with a service
which would set the network up before /nix/store is mounted. Starting an
init daemon from initrd would be perfect, but even a modular initrd/init
script with an easy way to place packages onto initrd would be nice. I
tried to implement this in a proper way myself, but after two weeks of
non-stop hacking it turned out to be much more work than I was prepared to
commit. Currently, I have a setup that has something like 80% chance of
working, with 20% of effort (and I'm not happy with that).

What I can probably do with a clear conscience is to transform the PXE
part into nixos service, and put everything else into an example
configuration profile. Will that be okay?

—
Reply to this email directly or view it on GitHub
#2920 (comment).

www.debian.org - The Universal Operating System

domenkozar · 2016-01-18T12:22:32Z

@oxij did you get PXE boot working with NixOS? Note there is https://github.com/sleinen/nixos-pxe-installer

oxij · 2016-01-18T13:20:56Z

Yep. Like two years ago.

You'd be surprised, but the biggest problem with my approach is that for
generic usability (installer is trivial) it needs a series of patches to
nixos most of which are either applied very slowly (#10996, e.g. that
one seems to be about GRUB, but I actually invented it to generate
pretty PXELINUX menus), need help to become backward compatible and are
stalled (#11529, this seems to be about simple refactoring, but the
actual point of that is that it allows one to easily duplicate
options.networking expressions so that one could define networking for
a whole cluster in a single pretty config), pretty much rejected
(#11484, I can't boot anything useful (except the installer and trivial
configs) without that) or actually rejected (#6969, I use
nixpkgs.config a lot, overrides are overrated). Considering the facts
that my branch has ~80 patches more (1/2 of those are actually pretty
useful to massive farming via PXE) and flaming in issues about my
changes actually takes more time that implementing those changes, you'd
understand why I'm not thrilled about pushing them. Hopefully, most of
the stuff I need gets merged a little by little or reimplemented by
someone else in a couple years, and then my services.netboot.menu
(which immediately needs at least 10, if not more, of those patches to
be barely functional) thing will be usable on vanilla nixos.

(Yep, I actually though about forking nixpkgs on this point, but I hate
merging, and git can't handle upstream rebases, so my hypothetical users
would need to checkout new branches, which isn't nice.)

I understand why @sleinen implemented his thing, installing via PXE is
awesome, but I wanted a generic thing, I implemented it and now I just
rebase it on top of upstream/master once in a while.

Also, my approach is somewhat more verbose than I'd like, e.g.:

  networking.firewall = {
    enable = true;
    allowPing = true;
    checkReversePath = false;
    allowedTCPPorts = [ 80 ] ++ [ 53 69 ] ++ [ 111 832 2049 ] ++ [ 389 ];
    allowedUDPPorts =           [ 53 69 ] ++ [ 111 832 2049 ];
    extraCommands = ''
      iptables -A nixos-fw -j nixos-fw-accept -p tcp -m multiport --dports 30000:64000
      iptables -A nixos-fw -j nixos-fw-accept -p udp -m multiport --dports 30000:64000
    '';
  };

  services.dnsmasq = {
    enable = true;
    resolveLocalQueries = true;
    extraConfig = ''
      domain = lan
      dhcp-fqdn
      dhcp-range = 192.168.1.100,192.168.1.254,255.255.255.0,192.168.1.255,6h
      dhcp-boot = lpxelinux.0,alice.lan,192.168.1.2
    '';
  };

  services.nfs.server = {
    enable = true;
    exports = ''
      /nix/store 192.168.1.0/255.255.255.0(ro,all_squash,async)
      /home/tv 192.168.1.0/255.255.255.0(rw,no_root_squash)
    '';
  };

  services.atftpd = {
    enable = true;
    root = "${config.services.netboot.menu.output}";
  };

  services.netboot.menu = {
    prefix = "http://192.168.1.1/boot/";

    default.entries = {
      "atv" = {
        label = "TV";
        key = "t";
        isDefault = true;
        config = {
          require = [ ./tv.nix ];
          services.netboot.client.nfsServer = "192.168.1.1";
        };
      };
      "dumb" = {
        label = "Cluster node";
        key = "h";
        config = {
          nixpkgs.system = "x86_64-linux";
          require = [
            ./node.nix
            ./localization/dvorak.nix
          ];
          services.netboot.client.nfsServer = "192.168.1.1";
        };
      };
    };
  };

Most of that crap is just the nature of PXE booting, but e.g. those
repeating

services.netboot.client.nfsServer = "192.168.1.1";

lines drive my crazy, especially when there's a ton of entries there,
but I can handle it, yes.

domenkozar · 2016-01-18T15:29:09Z

@oxij thank you. I'll review this in detail once I'm onto PXE boot and check on the level of effort.

oxij added 7 commits June 12, 2014 23:53

minor fix in documentation

5e4d0f5

nixos: change EDITOR and PAGER environment variables to mkDefaults

de39787

nixos: add support for mkhomedir in PAM

9710317

atftp: rewrite, fix and update

0718304

nixos: add atftpd service

8084a71

nixos: add dhcpcd.persistent option

4b01a2d

nixos: add setuid wrappers for some networked filesystems' helpers

3861349

So that `user` mount option would work allowing normal users to mount and umount stuff marked with it in `fileSystems.<name>.options`.

7c6f434c reviewed Jun 13, 2014
View reviewed changes

7c6f434c added the feedback-requested label Sep 3, 2014

oxij closed this Oct 12, 2014

domenkozar mentioned this pull request Jan 19, 2016

Prepare PXE booting snabblab/snabblab-nixos#9

Open

oxij deleted the various-changes branch August 29, 2017 11:17

SLNOBody mentioned this pull request Jan 19, 2018

kernel-less profile/module in nixpkgs/nixos? #33940

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various changes and fixes with the general aim to support netbooting NixOS #2920

Various changes and fixes with the general aim to support netbooting NixOS #2920

oxij commented Jun 13, 2014

7c6f434c Jun 13, 2014

oxij Jun 13, 2014

domenkozar commented Jun 13, 2014

oxij commented Jun 13, 2014

domenkozar commented Jun 25, 2014

oxij commented Jun 25, 2014

oxij commented Jun 25, 2014

offlinehacker commented Jun 25, 2014

oxij commented Jun 25, 2014

domenkozar commented Jun 26, 2014

7c6f434c commented Aug 30, 2014

oxij commented Aug 31, 2014

7c6f434c commented Sep 1, 2014

7c6f434c commented Sep 1, 2014

oxij commented Sep 1, 2014

7c6f434c commented Sep 1, 2014

7c6f434c commented Sep 3, 2014

oxij commented Sep 9, 2014

7c6f434c commented Oct 9, 2014

oxij commented Oct 12, 2014

wmertens commented Oct 13, 2014

jagajaga commented Nov 13, 2014

domenkozar commented Nov 13, 2014

jagajaga commented Nov 13, 2014

wmertens commented Nov 13, 2014

oxij commented Nov 15, 2014

lucabrunox commented Nov 15, 2014

domenkozar commented Jan 18, 2016

oxij commented Jan 18, 2016

domenkozar commented Jan 18, 2016

Various changes and fixes with the general aim to support netbooting NixOS #2920

Various changes and fixes with the general aim to support netbooting NixOS #2920

Conversation

oxij commented Jun 13, 2014

7c6f434c Jun 13, 2014

Choose a reason for hiding this comment

oxij Jun 13, 2014

Choose a reason for hiding this comment

domenkozar commented Jun 13, 2014

oxij commented Jun 13, 2014

domenkozar commented Jun 25, 2014

oxij commented Jun 25, 2014

oxij commented Jun 25, 2014

offlinehacker commented Jun 25, 2014

oxij commented Jun 25, 2014

domenkozar commented Jun 26, 2014

7c6f434c commented Aug 30, 2014

oxij commented Aug 31, 2014

7c6f434c commented Sep 1, 2014

7c6f434c commented Sep 1, 2014

oxij commented Sep 1, 2014

7c6f434c commented Sep 1, 2014

7c6f434c commented Sep 3, 2014

oxij commented Sep 9, 2014

7c6f434c commented Oct 9, 2014

oxij commented Oct 12, 2014

wmertens commented Oct 13, 2014

jagajaga commented Nov 13, 2014

domenkozar commented Nov 13, 2014

jagajaga commented Nov 13, 2014

wmertens commented Nov 13, 2014

oxij commented Nov 15, 2014

lucabrunox commented Nov 15, 2014

domenkozar commented Jan 18, 2016

oxij commented Jan 18, 2016

domenkozar commented Jan 18, 2016