Feature: Remote Modules

[mitchellh]

🚧 Work-in-progress.

This PR introduces a new feature to Terraform that we've internally called "remote modules." Remote modules are similar in spirit to normal modules: they are an abstraction to treat a set of resources like a black box, only accessing certain outputs from the module as a whole. The difference between normal and remote modules is that remote modules expect the resources they're containing to be created externally and beforehand, and are just accessing the resulting outputs, whereas normal modules create and manage the contained resources as part of the Terraform run.

Note: I'm not convinced remote modules is the best name, it just is what we've been calling it.

The practical use case: different teams can manage different infrastructures, but can share information about the infrastructures they're managing. Example: core IT can manage the VPC, security groups, subnets, etc. and an app team can consume those values to deploy their app within the shared VPC.

For implementation, this PR introduces a new provider and resource: terraform_state. This lets you access the outputs of any remote state (stored in any backend supported by Terraform's remote state system).

I think this feature is core enough that it may deem some syntactic sugar to promote it to a 1st class feature. I stress that this should probably just be sugar: internally we should just convert it back to a resource and treat it as such.

For now, here is how it looks like:

resource "terraform_state" "vpc" {
    backend = "atlas"
    config {
        path = "hashicorp/vpc-prod"
    }
}

resource "aws_instance" "foo" {
    # ...
    subnet_id = "${terraform_state.vpc.output.subnet_id}"
}

[phinze]

All the impl LGTM. So clean! 🚿 ✨

As for sugar, I agree - my best idea so far is to wire this into variables somehow. I think semantically it makes sense to put it there, you're wiring up "output" from one side to "input" on another.

Implementation-wise is probably more more difficult, but I like it mental-model-wise.

I'll see if I can cook up a candidate syntax for that idea, even though I'm not sure about it.

[radeksimko]

For implementation, this PR introduces a new provider and resource: terraform_state

👍 I like that. That makes Terraform much more transparent and may help solving quite a few other features in the tracker here.

[mitchellh]

@philk Yeah, I see what you're saying, but I think it is more analagous to a module moreso than a variable. A variable declaration is just a single variable. This is kind of like a "remote variable set" but I don't think that explains the practicality of it well enough. The more I think about it the more I'm still leaning towards a "remote module" since it is better analogous to a module which is a black box of functionality that gives you outputs.

[mitchellh]

@phinze ^ I meant you, not @philk, sorry.

[mitchellh]

@phinze You know what, for 0.4, I think we should just ship this as-is. We don't need to call it anything. We can just use documentation to show how its used in various places and just reference it as "accessing the results of an independent Terraform run". Thoughts?

[phinze]

@mitchellh sounds reasonable. One last concern before we land:

The way this is named, I can imagine people are going to expect it to be able to work with local state files as well. Two ways to go there:

• Say we'll support that either sooner or later
• Add _remote to the resource or provider name to conceptually attach it to remote state

[bobtfish]

👍 to the feature, but also +1 to wanting local state files with this for it to be what I want currently.

[bobtfish]

I've tried this out, and it doesn't seem to work like I expect when I terraform destroy. My terraform config looks like this:

https://github.com/bobtfish/terraform-example-vpc-infra/blob/master/eucentral1-demo/internal.tf

and the issue I see in the destroy plan is a cycle:

• Cycle: aws_autoscaling_group.consul, terraform_state.vpc, aws_launch_configuration.consul, aws_launch_configuration.consul (destroy), terraform_state.vpc (destroy)

[mitchellh]

@bobtfish I'm not seeing your terraform state in the link, but I don't think its caused by this in particular. I'd like to see more though. This thing is coming in after @phinze's feedback.

[mitchellh]

Docs are missing, but will add those after. I want to do a bigger section on the utility of referencing state in this way. Merging.

[bobtfish]

@mitchellh np, I'll raise an issue with committed configs next time I see the problem!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.