Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't change AWS provider region with existing state #3454

Closed
papiveron opened this issue Oct 8, 2015 · 9 comments

Comments

Projects
None yet
6 participants
@papiveron
Copy link

commented Oct 8, 2015

Hi,

I'm having a little complicated issue with terraform about multi region.
I tried several ways without success.

In fact I've created a template to depoy an AWS VPC with several resources such as security groups, iam roles, instances, s3 buckets,....

But I have to create one VPC per region with same configuration (eu-west-1(Ireland) and us-west-2(Oregon)).

However there are aws resources wich are multi-region (means they are not limited to one region) like iam rôle, s3 bucket, cloudfront an so on.

Then, I ran terraform apply successfully first time in "eu-west-1"". Now I changed the region to "us-west-2", expecting terraform to create new resources smartly, leaving global/unchanged resources like iam roles, and routeR3 registrations..., but the terraform plan and terraform refresh** command fail, trying to access resources not present in the new region. Typically I got this :

Error refreshing state: 2 error(s) occurred:

* 1 error(s) occurred:

* aws_vpc_dhcp_options.support_vpc-dhcp: Error retrieving DHCP Options: InvalidDhcpOptionID.NotFound: The dhcpOption ID 'dopt-9e8e65fb' does not exist
        status code: 400, request id: []
* 1 error(s) occurred:

* aws_s3_bucket.support_ghost_bucket: error reading S3 bucket "s3.support.ghost-packages.eu-west-1.account_name": RequestError: send request failed
caused by: Head : 301 response missing Location header

For me terraform is trying to access the eu-west-1 dhcpOtion in us-west-2 while I thought i would create a new vpc in us-west-2 with its own attributes.

In the meanwhile it's trying to access (maybe to destroy) the s3 bucket present in Ireland (**eu-west-1) and got an 301 error.
But I would like there is to leave the Ireland S3 bucket unchanged and create new one in Oregon (us-west-2) named s3.support.ghost-packages.us-west-2.account_name as us-west-2 is the new value of aws_region variable.

I hope you're getting what I mean here and someone has already faced with that, and I would like the best way achieve I'm trying to achieve, without duplicate resources and mistakenly detroy resources.

Thanks.

@apparentlymart apparentlymart changed the title [Terraform]Difficulties with terraform state with AWS provider with muti region! Can't change AWS provider region with existing state Oct 9, 2015

@apparentlymart

This comment has been minimized.

Copy link
Member

commented Oct 9, 2015

It's a limitation of Terraform right now that it doesn't track the provider settings, like region, on the resources it creates. The Terraform state tracks only the resource ids and not the region they belong to, so the provider configuration is necessary to know the resource where the resources were created.

Therefore it is unfortunately not currently possible to change the AWS region once resources are created, and doing so is likely to result in undesirable behavior since it will (as you saw) attempt to refresh resources from the wrong region. To switch regions it's necessary either to first run terraform destroy to remove all of the existing resources, or to discard the state file so that Terraform will create new instances for all resources. There isn't a supported way to retain the resources that are not region-specific while remakng the ones that belong to a specific region.

This is a frustrating and counter-intuitive behavior that's tripped me up before too, so I'm going to leave this issue open as a prompt to discuss how Terraform could better support this use-case.

@apparentlymart apparentlymart added the core label Oct 9, 2015

@papiveron

This comment has been minimized.

Copy link
Author

commented Oct 9, 2015

Ok @apparentlymart,

Personally I understand the issue, and I could contribute in implementing the feature.

I'm going to reading the contributors, source code and let you know I think it can be done providers-wide.

Regards.

@papiveron

This comment has been minimized.

Copy link
Author

commented Oct 9, 2015

Hello @apparentlymart,

I've got an idea to achieve my need that I would like you to confirm or correct me.

This is my idea :

  • Create a "global" directory/module where I put code/configuration for global-region resources like iam and dns

  • Create a "eu-west-1" directory/module for eu-west-1 resources

  • Create a "us-west-2" directory/module for us-west-2 resources

  • In each module above, create a subdir/module containing a main.tf and its variables

  • And then when planning or applying configuration, use the -target parameter for each module, typically

    terraform plan -target=global -target=eu-west-1
    terrform apply
    terraform plan -target=global -target=us-west-2
    terrform apply
    

    My questions are then :

  • Is this idea sweet?

  • In case it's ok, how should I expose my modules and variables? Do I have only to define the three module above in my root main.tf, and expose variables in directories and sub directories?

  • With this configuration, how should I use output variables if the output is from a sub module? How should I resolve the output variables? For example in the module eu-west-1 and/or us-west-2 I have a sub module toto with and output variable named my_output. Should I use the syntax lile my_var = "${module.eu-west-1.toto.my_ouput}" or my_var = "${module.eu-west-1.my_ouput}" instead, or whatever other syntax?

I know it's not easy, please let me know if you don't understand what I mean.

Thanks a lot for you collaboration.

Regards.

@papiveron

This comment has been minimized.

Copy link
Author

commented Oct 9, 2015

I tried,

But in my case it's painful with dependencies management. And terraform doesn't really support nested variable, interpolation of modules' source paramater, .....

@lwander

This comment has been minimized.

Copy link
Contributor

commented Oct 9, 2015

@papiveron,

This is definitely something you can implement provider side. For the Google code I made region an optional attribute where applicable, which gets its value from the resource when available, otherwise the provider. See the code here, and the PR using it here. The nice thing about this approach is that it doesn't break existing infrastructure since the default behavior is left unchanged.

@papiveron

This comment has been minimized.

Copy link
Author

commented Oct 10, 2015

Hi @lwander,

Thanks, I'll have a look on it.

Regards.

@apparentlymart

This comment has been minimized.

Copy link
Member

commented Oct 10, 2015

@papiveron at work right now we do something like what you described, with a separate Terraform config per region and then one "global" config which creates non-region-specific things like Route53 zones.

The only difference is that we treat each as a completely separate Terraform config that has to be planned and applied separately, rather than using -target to select resources from a single config.

It's not the most ideal thing but it's been working reasonably well for us so far for things that don't change very often, like our shared network infrastructure (VPCs, etc) and DNS zones.

The idea of allowing the resources themselves to specify/override the region is one I had too. I think it can make sense, but should ideally be implemented in a way that results in minimum duplicated code between resources, and also when the default, provider-level region is used it should still get saved explicitly in the resource so that users can seamlessly switch between provider-level region and resource-level region, without Terraform wanting to recreate the resource.

@farridav

This comment has been minimized.

Copy link

commented Oct 23, 2015

I think I may be looking at a similar issue, trying to build things in different regions, for me the architecture I need to provision is exactly the same in each region.. after struggling with each team member appending the same -var arguments to each of their terraform commands, (then realizing you cannot re-use the script with a different region) I have looked at two ways of tackling this..

  1. Use a different state file for each region (I'm not yet sure if this is possible with remote state, and it doesn't help with common code)
  2. Turn my re-usable code into a module, and re-use that module once per region within my terraform (no more command line arguments, except access keys) and perhaps target items individually, like @papiveron suggests

Does this sound like a sensible approach? (I'm favoring 2 atm)

@radeksimko radeksimko added the bug label Feb 6, 2016

@mitchellh

This comment has been minimized.

Copy link
Member

commented Oct 26, 2016

This is likely a feature request rather than a bug, depending how you look at it. I consider this a feature request. This isn't a priority for us right now and because it been over a year I'm going to close this.

In a future world (enhancement), I can imagine Terraform detecting changed provider configs and giving you some sort of message. But the provider configs themselves have semantics that Terraform will need to understand: changing a region changes a target and thus a whole new state, but changing access keys might be just fine... Also: do we delete the old items? Do we leave them? Do we create a new state?

Due to the magnitude of such questions, I'd consider this a whole new feature.

The point of the state is to track the resources you created with the configured provider. By changing the target of that provider, Terraform will still believe the resources should be there. This is the current behavior and I believe it is correct.

Like I said, this can probably be improved upon but its a pretty large undertaking and I don't think the demand is quite there to prioritize this. Sorry!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.