Import resources into Terraform #581

Closed
piavlo opened this Issue Nov 19, 2014 · 56 comments

Projects

None yet
@piavlo
piavlo commented Nov 19, 2014

Use case , manage infrastructure environments both existing and created from scratch, by same terraform configs. For example in development environments we want to create everything from scratch and destroy everything then we finish working with it. In production we want to be able to add new resources to it with terraform but not conflict with resources that already exist but not managed by terrafrom yet.

For ex:

resource "aws_subnet" "b" {
    vpc_id = "${aws_vpc.terraform.id}"
    cidr_block = "${var.vpc_network_prefix}.${lookup(var.vpc_subnet_suffix, "b")}"
    map_public_ip_on_launch = true
    availability_zone = "${var.region}b"
    if defined(var.subnet_b_id) { load_by_id(${var.subnet_b_id}) }

}

if terraform has not yet created this resource it will check if subnet_b_id is defined and if resource with such id exists in provider api it will import this resource into terraform based on data in api instead of creating it.

@sethvargo sethvargo changed the title from import resources into terraform to Import resources into Terraform Nov 19, 2014
@sparkprime
Collaborator

I think this doesn't need language support, instead you just need the ability to import the existing subnet into the tfstate file (and give it the name "b" in the process). Then Terraform would reuse the existing one instead of creating a new one to fill the "b" slot. I.e. that conditional if defined(...) essentially already exists in the core terraform planning logic to check if a resource already exists in the tfstate.

@evaryont
evaryont commented Dec 9, 2014

Would the recommended way forward for this be then to create a script that traverses an AWS network, getting all Route53 records, VPC information, instance data, etc, and generating the appropriate tfstate JSON + example .tf files?

@armon
Member
armon commented Dec 9, 2014

Correct. Ideally the tool would be able to scan resources on various provides and with operator assistance map it to config logical names.

@carimura

+1 to this one for us. Keeps us from being able to launch new clusters of our service into an existing VPC. Any update on this one since Dec?

@phinze
Member
phinze commented Mar 2, 2015

No specific update on the feature itself, it's still on the roadmap.

But I wanted to jump in here to link an excellent article by @phrawzty on a strategy for dealing with existing resources with current versions of terraform:

https://www.dark.ca/2015/01/27/handling-extant-resources-in-terraform/

@clstokes
Member

+1, would love to see this (soon).

@AlexanderEkdahl
Contributor

An alternative approach could be something inspired by terraform taint where the user could execute something along the lines of terraform add [options] name id. For example terraform add aws_security_group.name sg-4bdc302e and it would be marked as existing but without attributes in the .tfstate-file. Upon the next refresh or apply the attributes would be populated.

@fatih fatih referenced this issue in koding/koding May 6, 2015
Merged

kloud: apply improvements for klient access #3635

@johnrengelman
Contributor

I like @AlexanderEkdahl suggestion. This would be awesome and we could keep people out of the state file - @mitchellh @catsby, thoughts? I'd be willing to do a POC for something like this.

@mitchellh
Member

Thats a pretty neat idea. Could help basic stuff... What do you think @phinze?

@phinze
Member
phinze commented May 12, 2015

Agreed - I think that would be a really valuable baby step to take. ๐Ÿ‘ฃ POC away @johnrengelman!

@johnrengelman
Contributor

Cool.

@johnrengelman
Contributor

I put together a very simple POC for discussion - #2022

@blalor
Contributor
blalor commented May 21, 2015

Separate from @johnrengelman's POC (where I commented, too), I feel like a missing part of this conversation is actually generating the Terraform config. I'd expect it to still be a relatively manual, iterative process, but I'd want terraform import <resource_id> <provider_id> to generate config appropriate for a .tf file, probably with a helper for the various _association resource types. Even outputting JSON a la .tfjson would be a huge help.

@ketzacoatl
Contributor

+1 to @blalor's comment, though I would advocate for taking this in separate.. the first being coverage for inport to begin with.. generating config has more nuances and work in general.

@meylor
meylor commented Jun 5, 2015

Others may find this tool helpful. I tested the EC2, ELB, security group, and VPC capabilities and it was able to extract existing AWS infrastructure into resources. https://github.com/dtan4/terraforming

@jwaldrip
Contributor
jwaldrip commented Jul 8, 2015

๐Ÿ‘ I would love to see this incorporated into terraform.

@nrcxcia
nrcxcia commented Jul 9, 2015

Would love to see this as well, this functionality would allow us to start using terraform at my company to manage existing infrastructure. +1

@ckelner
ckelner commented Jul 20, 2015

๐Ÿ‘

@danielcbright
Contributor

๐Ÿ‘

@smcavoy-travelbird

I'll join the party. This would be a huge feature.
(AWS first please)

@leopepe
leopepe commented Jul 22, 2015

๐Ÿ‘

@mikerahmati

I'd like to see this incorporated into terraform too ๐Ÿ‘

@antonbabenko
Contributor

yes, yes, yes ๐Ÿ‘ I would love to see this incorporated into terraform also, because it should have wider community support.

@dtan4 great work!

@JesperTerkelsen

๐Ÿ‘

@br0ch0n
Contributor
br0ch0n commented Aug 21, 2015

๐Ÿ‘ I'd think the terraform equivalent of puppet's 'resource' command would be good: https://docs.puppetlabs.com/references/3.8.latest/man/resource.html

@Deserved

๐Ÿ‘ It would help a lot!

@JacobASeverson
Contributor

๐Ÿ‘

@zstarer
zstarer commented Sep 29, 2015

๐Ÿ‘ for sure.

@josephholsten
Collaborator

could folks giving a ๐Ÿ‘ try out the patch in #2022 or #3345? Those implementations seem to be the way forward.

@realloc
realloc commented Oct 22, 2015

๐Ÿ‘

@preflightsiren

+1 do you think this would cover a similar idea I've had to allow for stub resources / read-only resources? essentially it would allow you define a resource that already exists (eg. vpc/subnets), to allow you to reference them in aws_instances for example, but without having terraform manage the resources and try to destroy/recreate them?

@ross
Contributor
ross commented Oct 27, 2015

+1 do you think this would cover a similar idea I've had to allow for stub resources / read-only resources? essentially it would allow you define a resource that already exists (eg. vpc/subnets), to allow you to reference them in aws_instances for example, but without having terraform manage the resources and try to destroy/recreate them?

Not as it stands, but something like it/built upon it could potentially allow that. I'd assume you'd need some sort of read-only meta property that would say you can refresh this thing, but don't ever try to change it. If you had that or something like it this would allow you to get it in to your state file initially. That said it's definitely out of scope for what this PR is trying to do.

@masterkorp

๐Ÿ‘

@kha0S
kha0S commented Oct 27, 2015

A way to import existing resources would be awesome ๐Ÿ‘

@rdark
rdark commented Dec 17, 2015

@kha0S - I've had some luck with https://github.com/dtan4/terraforming

@kha0S
kha0S commented Dec 23, 2015

@rdark Thanks. Will give it a try ๐Ÿ‘

@aldarund

+1

@rokka-n
rokka-n commented Jan 26, 2016

๐Ÿ‘

@JigarS91
JigarS91 commented Feb 1, 2016

I am trying to create new subnets in existing vpc.
I have added vpc id in variables.tf file as

variable "aws_vpc" {
default = "vpc id"
}

Getting below error:-
aws_subnet.private: Error creating subnet: InvalidVpcID.NotFound: The vpc ID "vpc id not found"
Not sure how to go ahead with this one. Need help !

@preflightsiren

@JigarS91 https://www.terraform.io/community.html is the place to ask these questions :)

@rizvfe00

๐Ÿ‘

@avdhoot
avdhoot commented Feb 29, 2016

+1 it will be good to have feature in terraform.

@cbroglie
Contributor

terraforming is working OK for me too. My process is essentially to export all resources into tfstate form, then rename the ones I'm importing, and delete the others. Would be great to have this functionality built into terraform.

@allandrick

This is a bit of a show stopper for my application - I don't want to edit the tfstate if I can avoid it... any update on whether this will be moving forward in one form or another?

@Pryz
Contributor
Pryz commented Mar 30, 2016

๐Ÿ‘

@robertwe
robertwe commented Apr 1, 2016

๐Ÿ‘

@TerribleDev

๐Ÿ‘

@LusciousPear

+1, this makes it much easier to recover from partial applies

@cyakimov
cyakimov commented May 3, 2016

๐Ÿ‘

@davidfic
davidfic commented May 3, 2016

๐Ÿ‘

@jvasallo
Contributor

๐Ÿ‘

@nitdana
nitdana commented May 17, 2016

+1

@manojlds
Contributor

Folks, github has had reactions for a while. Please stop the +1 and thumbs up.

@brandon-dacrib

seconded on a request for an update, clearly this is something that the community wants. If the response is that hey this will be included in version 0.6.20 then so be it.

@johnrengelman
Contributor

Looking at the work on the master branch, parts of this (or maybe all) will be in 0.7.

@mitchellh
Member

The basics of this are in master and will be part of 0.7. See the website subfolder for docs until it is released. More fancy features such as config generation will come, but 0.7 will be able to import resources into Terraform.

This is just the beginning, but we're finally getting there!

@mitchellh mitchellh closed this Jun 10, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment