Configure NixOS from EC2 user-data

[copumpkin]

[See #6662 for more discussion of how I got here and why I'm doing it this way]

This is my initial attempt at getting a working configure-from-user-data NixOS image working. The basic idea is to create an "unstable" NixOS image: its /etc/nixos/configuration.nix doesn't actually specify the way the machine is configured, but rather assumes that an /etc/nixos/amazon-init.nix exists, which is not bundled inside the image. Instead of bundling amazon-init.nix, the image bundles a postBootCommands script that downloads the EC2 user-data and writes it into /etc/nixos/amazon-init.nix.

The "unstable" NixOS image thus only configures itself from user-data on first boot, since when it calls nixos-rebuild switch on your personalized configuration, the custom postBootCommands will go away.

User-data should look something like:

### http://nixos.org/channels/nixos-unstable nixos

{
  # Insert your config here, and you'll probably not want me to log into your box so
  # change the user below, or don't put one there at all!

  security.sudo.wheelNeedsPassword = false;

  users.extraUsers.copumpkin = 
    { createHome      = true;
      home            = "/home/copumpkin";
      description     = "Dan P";
      extraGroups     = [ "wheel" ];
      useDefaultShell = true;
      openssh.authorizedKeys.keys = [
        "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkoril5uKjJohHvqz9Ys9R2rBH95MUb4Rxo5kcuRvEIwMranQ7xP5eU7rZqfv7elE1DLfMs19via+btUX3w8o4juYxzXjafnH6Mck5hYdvxNnErW6gsp0vGDQ0ruRCQx3UmOuC5Ld/wXY7iMQqOlxeLZF2dVCKP1+BSs37wLC7scXYu0U+wODprVpAsZIOwLP85w/uCNlC8wbvNDWG+Hx+XD/ml2ezQiNBRnh7Qo3QKgpUvVBO0d9z84g92D2H9IA+pEpJiWFcYKGEowKSVQVFCi5LoWRiz8XLKL+JeBt5mmmqjmJua6o8lXV7+nba//KCIkG+IWS4nwKQlpzZXc4H"
      ];
    };
}

One thing to note is the ### section at the top, which specifies the channels (I just strip off the ### and direct into ~/.nix-channels)

If you trust me and want to try it out, I'm hosting a public AMI (until I get sick of paying for storage) of the above with ID ami-1c477874.

Questions for anyone still reading:

• Is this the best way to do it? The postBootCommands with periodic timed check feels kind of hacky, but I also can't use a systemd service and activation scripts didn't work either.
• Is this the best format for user-data? I like it because it's simple, specifies the full machine (with channels, although I'm not necessarily sold on the ### convention), and is very clearly just nixos configuration. Unfortunately it's not compatible with the existing format nixops uses for its temporary host keys, but I'd rather make it use a clean nix configuration file than force this into that format and have to deal with escaping and other ugliness. I haven't used NixOps much so perhaps it'll be too painful to transition it, but if possible I'd like the user-data format to be clean.

Note that it's still a WIP, so I'll obviously take out the useless echo calls and such before merging 😄

cc @edolstra @shlevy @rbvermaa

[shlevy]

Actually looks pretty good and simple. Without hooking into the uevent system I'm not sure if there's a better way than polling to tell when networking is available, and unfortunately we can't use systemd's management.

[edolstra]

Why can't we use systemd's management? If it's about running nixos-rebuild from a service, I'm sure that can be accomplished.

[edolstra]

Why not write this to configuration.nix? That's more standard.

[copumpkin]

I wanted a simple place that would make it clear what came from user-data and what didn't. That way, configuration.nix comes pre-wired to import from amazon-init.nix but if you want to jump in and change it, you can tell it to stop importing amazon-init.nix, or use the existing file differently and so on. It also feels weird to auto-write to a location that people typically treat as human-managed. If someone were generating their own AMI, they wouldn't run the risk of getting their handmade configuration clobbered by my startup script.

[edolstra]

Yeah, I think it would be good to make this co-exist with the user data format used by NixOps (e.g., encode the configuration.nix data in some way), and only perform the nixos-rebuild if the user data contains a configuration.nix. That way, we don't need to create multiple AMIs.

It could be something like:

ec2-run-instances --user-data "NIXOS_CONFIGURATION: `base64 -w0 ./my-config.nix`"

[copumpkin]

@edolstra I pretty much documented all my attempts at using systemd on the other ticket and failed. If someone can get it working, that'd be nice, but @rbvermaa also said it felt wrong to have a systemd service doing this and said he preferred the current way.

About the format, I'm not super-opposed to that, but it feels like a historical accident would be making this less usable. Now I can't just go to the console and look at my NixOS configuration because it's base64-encoded because NixOps (which I don't even use) can't deal with it otherwise? It just seems like such a pity that we have this super-nice configuration format (nix itself) and are dropping it in favor of a hard-to-use simple key-value mapping that forces us to preprocess the input and not be able to read it later. This feels like a "let's rip off the bandaid and deal with a little bit of transition pain" kind of situation to me. For example, the configuration I included in the original ticket would not be human-readable right now if I followed that scheme.

If you insist on the NixOps format, how should I represent the channels?

[copumpkin]

@edolstra what if I devise some sort of real transition plan that would allow NixOps to continue to work unchanged for now, but also simultaneously supported the format I wanted (I haven't yet decided how to achieve that). Then there would be no real transition pain, but NixOps would switch over at some point to my proposed format without breaking anyone.

[edolstra]

Yeah, my only concern is that we can have a single AMI that supports both NixOps and configuration from user data. Note that nixos/modules/virtualisation/ec2-data.nix won't be bothered by a Nix expression in the metadata since it greps for some specific lines (e.g. starting with SSH_HOST_DSA_KEY:). But amazon-hvm-userdata-config.nix will barf on NixOps' metadata, I guess.

(It would probably be good to move amazon-hvm-userdata-config.nix to nixos/modules/virtualisation/ec2-data.nix so that we have only one place that deals with user data.)

[edolstra]

How about this:

• The script reads the user data, filters out any line starting with '#' or SSH_HOST_*:, and if the result is non-empty, writes the user data to /etc/nixos/configuration.nix (or wherever) and runs nixos-rebuild. If the result is empty, it does nothing.
• In the future, NixOps prefixes any user data with '#' (e.g. # SSH_HOST_DSA_KEY: ...) so it doesn't get confused with a Nix expression.

[copumpkin]

@edolstra I think that makes sense, thanks! Will update the PR in the next couple of days and report back if I run into any issues 😄

Did my reasoning for using a separate amazon-init.nix make sense? I don't feel nearly as strongly about that, so if you think it makes more sense to go straight to configuration.nix I'm happy to shove it in there.

[edolstra]

I have no strong feelings about amazon-init.nix either, but if the concern is clobbering the user's configuration file, you could just write configuration.nix only if it doesn't exist. So it would be created only on first boot, after which the user is free to edit it.

[cstrahan]

@copumpkin Are you still working on this? It sounds pretty slick. 😄

[copumpkin]

Yep, just haven't had a chance to finish it off yet!

[cstrahan]

@copumpkin I'm working on this right now.

[cstrahan]

A couple questions:

When rebooting and/or rerunning the fetch-ec2-data service, what should the behavior be?

• We probably don't want to clobber the user's /root/.nix-channels. I think we should probably check if the file exists, and only write to it once.
• I suppose overwriting /etc/nixos/amazon-init.nix shouldn't be a problem, though.
• Do we want to ensure that the nixos-rebuild switch only runs once? If so, how should we implement that - should we just check the existence of a file? (e.g. /root/.ec2-init-finished)

[cstrahan]

Here's a thought: we could ensure the whole thing happens only once by putting a file in the AMI (e.g. /root/.need-ec2-init), and then we just need to check that the file exists before doing a one-time rebuild from user-data, deleting the file thereafter. Any downsides to that?

[copumpkin]

It already does only happen once! When it reconfigures itself from userdata, it removes the reconfiguration script from the current configuration (unless you explicitly add it back, of course).

[copumpkin]

Re: clobbering /root/.nix-channels, I figured it would be fine because at that point no user will have ever touched the system. Once they touch the system, it won't get clobbered again due to the behavior I just mentioned.

[cstrahan]

It already does only happen once! When it reconfigures itself from userdata, it removes the reconfiguration script from the current configuration (unless you explicitly add it back, of course).

@copumpkin Ah, that makes sense - you could just overwrite the amazon-init.nix, of course :).

I see where you're copying over the configuration.nix, but I don't see where you're copying over the amazon-init.nix (which the former refers to), so I'm not really sure how this branch works, unless I'm missing something.

Here's what I have: master...cstrahan:nixos-userdata

[copumpkin]

The point is that the /etc/nixos/configuration.nix that I put inside the image isn't the configuration.nix that built the the image. That's what I meant by "unstable". If you took the image and did nothing but nixos-rebuild switch on it, it would change.

So instead, I build the image from an "external" configuration.nix that specifies that I want the machine to reconfigure itself on startup. Then after it does that, it reconfigures itself to not reconfigure itself anymore (because presumably the user doesn't ask for it to keep reconfiguring itself, although that's certainly a possibility too)

It's a little subtle and there's a phasing thing going on that makes it tricky to think about, I think.

[cstrahan]

Gotcha, just clicked a moment ago :).

[cstrahan]

@copumpkin Ok, I think my branch should work - I'll give it a test tomorrow.

[cstrahan]

@copumpkin How do you build an AMI?

[copumpkin]

@cstrahan there's a script somewhere in the nixpkgs source tree, but I've been using https://gist.github.com/copumpkin/6df9c50630ed5fc5abb5 with a self-signed cert. There's a few different ways to build an AMI but I'm using a fairly simple S3-based instance store one.

[cstrahan]

@copumpkin Thanks!

I don't know what else needs to change in our maintainer scripts and such to support building EBS-backed AMIs (which must be created from an existing instance), so I'll have to leave that to someone else.

/cc @edolstra

[copumpkin]

There are already scripts in in nixpkgs to automate the creation of EBS-backed AMIs as well, near the one I mentioned earlier. All the stuff is in here.

[copumpkin]

So what's the status on this? Is there anything left to do? I'd like to have three VM tests for it (to make sure NixOps-style userdata works, to make sure new-style userdata works, and that no userdata works), but would also like to get it into the 15.06 release. @domenkozar what's the cutoff date for getting things in there?

[copumpkin]

@cstrahan any progress? you have push access to my repo so if you want to update this PR, please do so 😄 assuming @edolstra approves of the new changes, I think the last thing left to do is a set of VM tests, which I can implement once you push your changes.

[copumpkin]

This is almost ready to be used. It depends on #8204 and #8013 (which also depends on #8204).

I now have two VM tests, built on top of the 169.254.169.254 user-data simulation functionality I put together in #8013. One tests that we still work properly with the NixOps user-data format, and the other ensures that we process the new-style configuration properly.

The main remaining thing I'd like to do is not force the user-data to contain a channel marker. To do that, I need to pre-populate the channel from the "host" building the image, so that it matches the actual content of the image. I expect not specifying a channel to be the common case, because it means minimal downloads to rebuild the system. Not having this means that one of the two tests is fairly slow, since it has to download a ton of NixOS stuff from the internet to reconfigure itself.

There's still a lot of stuff I'd clean up but we're getting close. Looking forward to feedback from anyone interested.

[copumpkin]

Since I have a VM test in place that makes sure that the NixOps-style userdata still works, I'm going to go ahead and merge this, treating the "configure from userdata" portion as a "beta". Beta in the sense that I'm still not sure what the best way to do it is, but I want to iterate on it a bit and it'll be easier to do once I have something concrete that's building and multiple people can experiment with.

[nyarly]

@copumpkin - it looks like as of 15.09, this is available in <nixpkgs/nixos/modules/virtualization/amazon-init.nix>, but the default AMIs don't have that in their stock configuration.nix. Am I reading correctly?

[paul-e-cooley]

Any word on when we might get a 15.xx AMI with this fix? Or alternatively, is it safe to just copy the commit results to the module and then perform a rebuild to initialize the fix?

[copumpkin]

I'll probably finish fixing the VM test for it this weekend and then I
think Rob said he'd be willing to regenerate the official AMIs. The test is
breaking because as part of the rebuild-from-config, it tries to pull from
the Internet and fails on Hydra because it's in a sandbox. Open to ideas on
fixing it!
On Fri, Jan 8, 2016 at 10:11 Paul E Cooley notifications@github.com wrote:

Any word on when we might get a 15.xx AMI with this fix? Or alternatively,
is it safe to just copy the commit results to the module and then perform a
rebuild to initialize the fix?

—
Reply to this email directly or view it on GitHub
#7370 (comment).

[paul-e-cooley]

Thanks, @copumpkin. I haven't even had a chance to look at the Hydra stuff yet. But plan to soon.

[aycanirican]

I wonder if we have network connectivity in postBoot. I got this in my log (in reverse order) and my user-data is not written to /etc/nix/configuration.nix

Mar 09 14:45:40 localhost stage-2-init: /nix/store/g9idnp7fzf06ir918wbhw6sz580i6gbw-nix-1.11.2/bin/nix-channel: unable to check ‘http://nixos.org/channels/nixos-unstable’
Mar 09 14:45:40 localhost stage-2-init: created 1 symlinks in user environment
Mar 09 14:45:40 localhost stage-2-init: attempting to fetch configuration from EC2 user data...

[nicknovitski]

I've observed the behavior @aycanirican describes, in 16.03. Would the script fit better in boot.initrd.network.postCommands?

[edolstra]

Yes, it's supposed to have network at that stage. In fact we even access the network in stage 1.

[nicknovitski]

Is it possible that the script needs something which gets set in $HOME/.nix-profile/etc/profile.d/nix.sh, but not in stage-2-init.sh? I've found year-old discussions saying that can cause a similar error.