Feature Request: Stop, don't destroy instance on user-data update

[jwaldrip]

when updating user-data, its not required to destroy the instance. In fact, this can be devastating to a etcd cluster. What would be acceptable is for the machines to be stopped, userdata updated, and then started.

[JeanMertz]

This is interesting. I don't think Terraform currently has a stop/start cycle, only destroy or don't destroy.

I can see a lot of value for this use-case (since we're also using user-data heavily), but I can also see this becoming complicated.

If I am not mistaken, on AWS an instance without an EBS backed storage device will simply reset the storage, so in that case the stop/start would have the same result as a destroy.

Here's the list of properties that AWS allows you to change in a stopped state:

You can modify the following attributes of an instance only when it is stopped:

• Instance type
• User data
• Kernel
• RAM disk

[sparkprime]

So this particular issue is not a problem with GCE as you can update metadata without forceNew. However there is a similar problem if you want to change zone / scopes without losing the data on your disk.

The way I go about doing this is to use a separate disk resource in any situation where the content of the disk is precious in some way. E.g. obviously for databases this is true, but typically in a node of a load balanced cluster the disk is expendable. Having a separate disk allows Terraform to recreate the instance without losing the disk (as the disk is not recreated). I believe this is equivalent to a reboot cycle.

[Pryz]

What about doing nothing when we update user_data ?

If you use Terraform and a configuration management solution, most of the time userdata are only for instance bootstrapping. So when you change userdata it's probably only for new instances not current ones.

[jwaldrip]

Disagree. I use CoreOS in production and manage everything with user data. Even existing instances.

[Pryz]

Ok, so what about having an argument to specify the strategy ? (Reboot, Recreate, None)

[sparkprime]

For the sake of comparison: For google compute instance metadata, we started off with updateable metadata (i.e. don't recreate the instance, but update the metadata so the instance can see the new values). This is actually very useful because you can do a hanging get on the metadata server which means you can run an agent in your VM that gets notified of changes to the metadata. This allows you to control the VM (e.g. roll out new application software) by updating the metadata. I don't know if AWS let's you do that but I assume it does as it's really useful.

However if you don't have such an agent, and the only thing you have to work with (if you want to do truly declarative node config) is the startup-script (userdata on AWS), then you want changing the startup script to recreate the instance so it will re-run the startup script. For lots of people, especially stateless servers, that is plenty good enough. The only cost is the time taken to recreate the instance (the agent is instantaneous).

So, I added a metadata_startup_script field in the google instance that is the same as metadata.startup-script except it is ForceNew. One is not allowed to specify both. The user can therefore choose whether to have the ForceNew behavior or not.

You could do the same thing for aws_instance: Have an updateable_user_data field that was not ForceNew to allow people to make changes to it without recreating the instance and run agents inside the VM to pick up these changes. And keep the current user_data field for the simple non-agent case.

[Pryz]

I like the idea of having two different user_data fields since it's for two totally different ways to manage instances.

[FergusNelson]

I think it would be nice to have stop and start command line options in general. The use case would be a staging or testing environment that is not needed all the time; rather than destroy and recreate the environment every time it is needed it would be nice to be able to do

terraform stop
terraform start

And have terraform start the machines in the correct order with respect to the dependency graph.

[c4milo]

As far as I understand the sole purpose of cloud-init or user-data scripts is to do early initialization of instances. From that perspective, it may not make sense to use it as a way to re-provision or re-configure instances since that's what tools like Puppet, Chef, Ansiable and Salt are for. Terraform was thought out as a way of creating and destroying infrastructure resources, and resource immutability is all over the place. Perhaps @mitchellh can shed some more light here.

[jwaldrip]

Sure, this is the case in some instances. But in our case, we use core-os
and use user-data to bootstrap the machines. We NEVER use puppet, chef,
salt etc, to instantiate the instance any further. Maybe this is bad
practice, but I know that user-data is subject to change. In addition we
cannot destroy our etcd instances to provision fresh user-data as we will
have a risk of losing quorom.

On Fri, Oct 23, 2015 at 11:13 AM, Camilo Aguilar notifications@github.com
wrote:

As far as I understand the sole purpose of cloud-init or user-data scripts
is to do early initialization of instances. From that perspective, it may
not make sense to use it as a way to re-provision or re-configure instances
since that's what tools like Puppet, Chef, Ansiable and Salt are for.
Terraform was thought out as a way of creating and destroying
infrastructure resources and resource immutability is all over the place.
Perhaps @mitchellh https://github.com/mitchellh can shed some more
light here.

—
Reply to this email directly or view it on GitHub
#1887 (comment)
.

Jason Waldrip
m: 646-460-5959
e: jason@waldrip.net
http://www.facebook.com/jason.waldrip
http://www.linkedin.com/in/jasonwaldrip

[sparkprime]

It's not bad practice, it works very well and is increasingly common.

[sparkprime]

Did you try using a separate disk resource? If you also use a statically assigned ip then I think that should be equivalent to a reboot?

[jmtsi]

I have the same use case as @jwaldrip. When using AWS's CloudFormation, it updates the user_data but doesn't re-create or reboot the instances. Amazon provides a cnf-hup script that detects user_data changes in instances and updates them accordingly, so it would be great to be able to only update the user_data with terraform and let AWS handle the updating.

[sparkprime]

Upgrading to "It's not bad practice, it works very well, is increasingly common, and is actively encouraged by AWS and Google" :)

[manojlds]

No movement on this?

[sparkprime]

I think it should be a fairly easy PR to add a new resource attribute for controlling userdata in a non force-new fashion. If someone were suitably motivated :)

[yogin]

I'd be very interested in a way to prevent instances from being recreated when there's a change to userdata. I understand some people use it for different purposes, and I like the idea of specifying a strategy (recreate, none, ...).

In our case, we only use the userdata to bootstrap new instances, later on, chef will take over and do the rest, but occasionally the userdata template might get a small update, and if I could avoid having to recreate instances, that'd be amazing!

[mcortinas]

No movement on this?

[modax]

I'm no terraform developer but I had a quick look at the current terraform code and it seems that the only possible solution without (big) terraform core changes could be adding another attribute like live_user_data (since ForceNew value cannot depend on the context as far as can tell). The code might still end up being messy due to two attributes modifying the same thing (with regard to diff calculation). Hence I'm not sure if terraform team would accept such a PR.

[sparkprime]

@modax As I said further up the thread, this is exactly what the google_compute_instance does and it works well

[echo-devnull]

Destroying and recreating instances when the user-data changes makes it so I can either:

• Not use user-data (And therefore cloud-config)
• Not use Terraform (Meaning I have to learn cloud-formation: ugh)

Since cloud-config is beeing used more and more to do the initial bootstrapping of an instance, I would think this is no longer an "enhancement" but more along the lines of "blocking"

[cheungpat]

If you don’t want your instance to re-create when user-data changes, you can try including this key in ignore_changes.

[br0ch0n]

I agree with @cheungpat that ignore_changes probably handles this ticket (once #5627 is done)

[gdubicki]

I confirm that what @cheungpat suggests is a good workaround. If you use GCE provider, then to ignore startup script changes use this code:

lifecycle {
  ignore_changes = ["metadata_startup_script"]
}

But it would be even better if you could set it once, globally - not in each resource of google_compute_instance type. Is that possible?

UPDATE: AFAIK it's not possible and you can't even use a variable to set ignore_changes. :( See #10730.

[kirkmadera]

This also applies to resizing AWS instances. We end up going into the AWS admin, stopping the instance, changing the instance type, then starting it again. Then we update terraform configs to match the new instance type and run terraform apply which syncs the terraform state to the new size.

Ideally we would just resize from Terraform. Resizing an AWS server also changes it's public and private ips. Running the resize from terraform would allow us to immediately change DNS as well if needed, rather than waiting for us to run terraform after the resize completes.

[stack72]

Dearest Friends, now that I have tackled the issue of changing instance_type and it will be part of terraform 0.8.8, I am going to start prototyping on this

Paul

[cnoffsin]

The use case for me here is I have to make type changes in AWS to a cloudera cluster.

The engineers shut it down gracefully so I could make the change.

I want to make the type change, but LEAVE them off. So that they can bring them up in the proper order.

[z0mbix]

Hi, why was this issue closed?

[sjourdan]

@z0mbix it's moved there as per the reference above the hashibot status

[stack72]

Hi @jwaldrip

@z0mbix hashicorp/terraform-provider-aws#23 - it was migrated

Paul

[z0mbix]

Sorry, I missed the move/reference

[zaphinath]

Is there an option for non coreos instances to load a cloud-config from a remote source like ignition so if we want we could change the file in some remote location and not force the termination of the instance?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.