Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NixOS AMI executes user data on restart #41826

Open
shmish111 opened this issue Jun 11, 2018 · 6 comments

Comments

@shmish111
Copy link

commented Jun 11, 2018

Issue description

USER_DATA is executed on restart of an EC2 instance, this is contrary to AWS documentation and general practice. It caused me some big problems as I assumed this wouldn't happen.

Steps to reproduce

  1. Start and EC2 instance with some configuration.nix user data
  2. nixos-rebuild the machine with some different configuration
  3. restart the machine

Expected outcome

User data is not executed and machine state remains as it was before reboot

Actual outcome

Machine configuration is rolled back to the user data version

Technical details

Please see "View and Update the Instance User Data" in https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html

@copumpkin

This comment has been minimized.

Copy link
Member

commented Jun 11, 2018

Whoops! Not going to have time to look into this for a few days at least, so if you want to take a stab at it, most of the logic for this is in here.

Easiest solution is probably just to touch /root/.initialized and then skip the rebuild if it already exists. We do have a nice VM test for this functionality so it should also be fairly easy to make sure it's doing the right thing.

@coretemp

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2018

Just use cloud-init, because then this logic doesn't need to be in NixOS anymore.

On this topic, I think we should also have recommendations as to how to use this feature if at all, because running nixos-rebuild can be a slow operation (not something you would want to do if you have 100s/1000s of machines).

@edolstra

This comment has been minimized.

Copy link
Member

commented Jun 13, 2018

Cloud-init is too bloated, see #39076 (comment).

@copumpkin

This comment has been minimized.

Copy link
Member

commented Jun 13, 2018

I've also written plugins for cloud-init (which we'd need here) and it's kind of a miserable and undocumented project. I was not impressed. And of course we'd need to wrap our user-data with yaml, reimplement most of their existing yaml support because it wouldn't work on our platform (you can list users and such, and we'd need to translate that to our declarative config because their default implementation is to just call useradd and the like).

@coretemp

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2018

Due to political considerations (Canonical creates cloud-init and likely cannot allocate people who could implement this with acceptable quality), I retract my suggestion for cloud-init.

@chris-martin

This comment has been minimized.

Copy link
Contributor

commented Oct 29, 2018

It caused me some big problems as I assumed this wouldn't happen.

INDEED

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.