Configuring one provider with a dynamic attribute from another (was: depends_on for providers)

[dupuy]

This issue was inspired by this question on Google Groups.

I've got some Terraform code that doesn't work because the EC2 instance running the Docker daemon doesn't exist yet so I get "* Error pinging Docker server: Get http://${aws_instance.docker.public_ip}:2375/_ping: dial tcp: lookup ${aws_instance.docker.public_ip}: no such host" if I run plan or apply.

There are providers (docker and consul - theoretically also openstack but that's a stretch) that can be implemented with Terraform itself using other providers like AWS; if there are other resources in a Terraform deployment that use the (docker or consul) provider they cannot be provisioned or managed in any way until and unless the other resources that implement the docker server or consul cluster have been successfully provisioned.

If there were a depends_on clause for providers like docker and consul, this kind of dependency could be managed automatically. In the absence of this, it may be possible to add depends_on clauses for all the resources using the docker or consul provider, but that does not fully address the problem as Terraform will attempt (and fail, if they are not already provisioned) to discover the state of the docker/consul resources during the planning stage, long before it has completed the computation of dependencies. Multiple plan/apply runs may be able to resolve that specific problem, but having a depends_on clause for providers would allow everything to be managed in a single pass.

[bitglue]

fleet is another example of a service for which this is a problem.

That provider used to work: if the ip address is an empty string, then it would use a mock API that failed on everything. But that solution is no longer working in Terraform 0.6.3.

[apparentlymart]

I think this is describing the same issue I wrote up in #2976, in which case unfortunately the problem is a bit more subtle than supporting depends_on on the provider.

Terraform is actually already able to correctly handle provider instantiation in the dependency graph, correctly understanding that (in your example) the docker provider instantiation depends on the completion of the EC2 instance.

The key issue here is that providers need to be instantiated for all operations, not just apply. Thus when terraform plan is run, Terraform will run the plan for the AWS instance, noting that it needs to create it, and then it will try to instantiate the Docker provider to plan the docker_container resource, which of course it can't do because we won't know the AWS instance results until apply.

When I attempted to define this problem in #2976 I was focused on working with resources that don't really have a concept of "creation", like consul_keys or template_file, rather than things like aws_instance, etc. There really isn't a good way to make that EC2 instance and docker example work as long as we preserve Terraform's strict separation between plan and apply.

The workaround for this problem is to explicitly split the problem into two steps: make one Terraform config that creates the EC2 instance and produces the instance IP address as an output, publish the state from that configuration somewhere, and then use the terraform_remote_state resource to reference that from a separate downstream config that sets up the Docker resources.

Unfortunately if you follow my above advice, you will then run into the issue that I described in #2976: the terraform_remote_state resource also won't get instantiated during plan. That issue seems solvable, however; terraform_remote_state just reads some data from elsewhere and doesn't actually create anything, so it should be safe to refresh it during plan and get the data necessary to populate the provider configuration before the provider is instantiated.

[bitglue]

@apparentlymart: In that issue, you are describing a case "which is very handy when parts of the config are set dynamically from outside the configuration." And you propose to get around the issue by making those resources that represent the outside configuration "pre-refreshed", meaning you skip the create step and immediately go to the read step.

But I'm describing a case where parts of the config are set dynamically from inside the configuration. For example, I want to manipulate a Docker or Fleet service that exists on the EC2 instance I just made. Would pre-refreshing help in this case?

[apparentlymart]

@bitglue no, as my (rather verbose) comment described, having a "true" resource from one provider be used as input to another is not compatible with Terraform's model of separating plan from apply. The only way to solve that, without changing Terraform's architecture considerably, is to break the problem into two separate configurations and then use terraform_remote_state (which can potentially be pre-refreshed, but isn't yet) to pass resource data between the two.

[bitglue]

edited a bit to clarify the plan/apply stages

@apparentlymart I don't think it's impossible, because terraform-provider-fleet used to do it successfully. When the configuration is being planned for the first time, the provider doesn't actually need to do anything, which is why it can use a Fleet API object which fails all the time. None of the provider methods get called because it's not necessary: if there's no prior state, then the plan is trivially to create everything, and you don't need to call any provider methods to know that.

After the plan is made and it's time to apply, then the provider can be initialized after having created the EC2 instance, and now it has a proper endpoint and can actually work and create the fleet resources.

On subsequent planning runs, the public IP address of the EC2 instance is already known, so planning can happen as usual. Bonus points for refreshing the EC2 instance before initializing the fleet provider to do its refreshing.

I'd also think it's not the separation of plan and apply that's really the issue here, but more specifically refresh. You can always terraform plan -refresh=false and that will work even if the providers can't connect to anything, right? Assuming the state file is accurate, of course.

You can run into a little trouble if you delete the EC2 instance to which fleet was connecting but after the fleet resources have been created. Now plan can't work. But there are two ways to resolve that situation:

1. Give the fleet provider the IP address of another EC2 instance in the same fleet cluster which hasn't been deleted.
2. If the entire fleet cluster has been deleted and so there is no IP address you could give it, then all the fleet units have been deleted, too. So you can delete all the fleet units from the state file.
3. Assuming you have an accurate (enough) state file, create the plan without refreshing and apply it. Then the missing EC2 instance will exist again and things will be back to normal.

Granted, these resolutions require a little hackish manual action, but it's not a situation I ever hit in practice. I'm sure with a little refinement it could be made less hackish.

[apparentlymart]

@bitglue it sounds like you're saying that in principle the providers could tolerate their configurations being incomplete until they are asked to do something. That is certainly theoretically true... while today most of them verify their config during instantiation and fail hard if the config is incomplete (as you saw the Docker provider do), they could potentially just let an incomplete config pass and then have it fail if any later operation tries to do any operations.

So one thing we could prototype is to revise how helper/schema handles provider configuration errors: in Configure, rather than returning an error when the ConfigureFunc returns an error, it could simply remember the error as internal state and return nil. The Apply and Refresh functions would then be responsible for checking for that saved error and returning it, so that Diff (which, as you noted, does not depend on the instantiated client) can complete successfully.

Having a prototype of that would allow us to try out the different cases and see what it fixes and when/how it fails. As you said, it should resolve the initial creation case because at that point the Refresh method won't be called. The case I'm less sure about -- which is admittedly an edge case -- is when a provider starts of with a totally literal configuration, and then at some later point it's changed to depend on the outcome of another resource; in that case Terraform will try to refresh the resources belonging to that provider, which will presumably then fail.

[bitglue]

@apparentlymart That's more or less my thinking, yeah. Though from what I've observed trying to get terraform-fleet-provider working again, in many circumstances the failure happens before ConfigureFunc is even called, so we might need a different approach.

[apparentlymart]

@bitglue I guess the schema validation will catch cases where a field is required but yet empty, so you're right that what I described won't entirely fix it unless we make all provider arguments optional and handle them being missing inside the ConfigureFunc.

[mtekel]

This is issue for postgresql provider as well, when you e.g. want to create AWS RDS instance and then use the port in the provider configuration. This fails, as the provider initializes before RDS instance is created, the port number is returned as "" and that doesn't convert to int:

  * provider.postgresql: cannot parse '' as int: strconv.ParseInt: parsing "": invalid syntax

TF code:

 provider "postgresql" {
   host = "${aws_db_instance.myDB.address}"
   port = "${aws_db_instance.myDB.port}"
   username = "${aws_db_instance.myDB.username}"
   password = "abc"
 }

[mtekel]

Interestingly, in your case TF graph does show the provider dependency, yet provider runs in parallel still. This is esp. problematic on destroy, as RDS instance gets destroyed before postgresql provider has chance to destroy the resources it has created, leaving "undestructable" state file behind. See #5340

[BastienM]

Hello there,

I got a similar problem with the Docker provider when used inside a Openstack instance (graph).

# main.tf
module "openstack" {
    source            = "./openstack"
    user-name         = "${var.openstack_user-name}"
    tenant-name       = "${var.openstack_tenant-name}"
    user-password     = "${var.openstack_user-password}"
    auth-url          = "${var.openstack_auth-url}"
    dc-region         = "${var.openstack_dc-region}"
    key-name          = "${var.openstack_key-name}"
    key-path          = "${var.openstack_key-path}"
    instance-flavor   = "${var.openstack_instance-flavor}"
    instance-os       = "${var.openstack_instance-os}"
}

module "docker" {
    source                  = "./docker"
    dockerhub-organization  = "${var.docker_dockerhub-organization}"
    instance-public-ip      = "${module.openstack.instance_public_ip}"
}

$ terraform plan

Error running plan: 1 error(s) occurred:
* Error initializing Docker client: invalid endpoint

Even thought I'm using :

provider "docker" {
    host = "${var.instance-public-ip}:2375/"
}

Logically it should wait for the instance to be up but sadly the provider is still initialized at the very beginning ...

---

So as a workaround I splitted my project into modules (module.openstack & module.docker) and then execute them one at a time with the -target parameter.
Like this :

$ terraform apply -target=module.openstack && terraform apply -target=module.docker

It does the job but make the whole process quite annoying as we must always specify the modules in the good order for each steps (plan, apply, destroy ...).

So until we got an option such as depends_on, I don't see other ways to do.
Is there an update on this matter ?