Rolling-apply to instances, or individual instance application

[mwitkow]

I've been experimenting with applying changes to individual instances in Terraform a'la:

$ ~/bin/terraform plan -module-depth=2 -target=module.services.google_compute_instance.frontend_production_instances.3 -out increase.plan
There are warnings and/or errors related to your configuration. Please
fix these before continuing.

Errors:

  * Unexpected value for InstanceType field: "3"

The goal would be to implement a rolling update/reprovision for certain types of changes (e.g. changing machine size).

Is that something that you guys are planning or, or possibly would accept patches for?

[mwitkow]

Actually, it is entirely possible to apply to a single instance. Unfortunately, the syntax for addressing the -target is different from the addresses output by plan.

For example:

plan outputs:
module.eu1-staging.google_compute_instance.core_instances.1

however, for -target you need to pass
module.eu1-staging.google_compute_instance.core_instances[1]
or
module.eu1-staging.google_compute_instance.core_instances.pimary[1]

Still not sure whether Terraform could apply changes to at most X nodes in a graph of a given type in parallel.

[mwitkow]

It seems that Terraform Context has support for Parallelism option.

terraform/terraform/context.go

Line 98 in 9de673b

par := opts.Parallelism

Is this settable somehow on the command line? @mitchellh

[phinze]

Hi @mwitkow-io,

Parallelism is now exposed via the UI in #3365

We'd definitely like to explore support for rolling update behavior - there's some related conversation in #1552

[mwitkow]

@phinze thanks for coming back to me. The use case we have is a rolling update of instances that host an Etcd cluster.

The problem with parallelism is that it first destroys all the instances (granted, 1 at a time), and only then creates them again (again, 1 at a time). This means that there is no sufficient Etcd quorum left :(

What we thought parallelism would do would be apply complete (down+up) changes to resources of a given target.

In the mean time we wrote a wrapper around terraform that reads the plan paths and issues terraform per-resource with the explicit primary[id]. However, it's flimsy and would really prefer a more streamlined solution.

The discussion in #1552 seems to be purely around AWS autoscaling, and it isn't clear whether it'd be applicable to groups of normal instances (in our case GCE). Could you clarify?

[pmoust]

@mwitkow I am in the same boat. I resorted in building plumbing around our etcd cluster to ensure that quorum is always met.
Perhaps OT but, try to decouple your etcd cluster from your 'application' instances. It will help you maintain sanity.
My 2c.

[apparentlymart]

@mwitkow if you mark your instances as create_before_destroy, does setting the parallelism work better then? Still just a workaround of course, but maybe it helps.

For a similar problem (rolling upgrades to a Consul cluster) I've sadly been just creating an entirely new Terraform resource alongside the old, applying to create both of them, manually fixing up the cluster, and then removing the old ones from the Terraform config. It's clunky to have to work in multiple steps like that, so I'd love a better solution.

[mwitkow]

@pmoust thanks for the tip. It would have been viable if etcd was the only thing that needed to survive a rolling-restart. We also have API servers and monitoring instances that require at least one of them to be up during an update.

@apparentlymart, we'll take a look as to whether create_before_destroy helps.

@phinze, is a rolling update mechanism something that is considered a feature for Terraform?

[phinze]

@mwitkow we'd definitely love to figure out some sort of story for rolling updates in Terraform, it's just a matter of figuring out how to model the feature in a clean way that preserves our declarative model. That part is anything but straightforward.

Tagging this thread thinking since it's definitely something we think/talk about on a regular basis, the path is simply unclear.

[discordianfish]

@mwitkow Even if it would be possible to create a new, then delete the old instance you still need to make sure the cluster is in sync before terminating the old instance and proceed with the next one.
On AWS, all this is supported (blogged about it), but I wanted to give GCP a spin and now also wondering how to do that there.. Apparently the deployment-manager doesn't support it and apparently terraform can't help either..

[phinze]

Nice blog post @discordianfish!

Any rolling update facility we bake into Terraform wouldn't be ASG specific, so I'm going to merge this thread back down with #1552 and shift that issue to cover the general concept of rolling applies. 👍