Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

depends_on for providers #2430

Open
dupuy opened this issue Jun 23, 2015 · 74 comments
Open

depends_on for providers #2430

dupuy opened this issue Jun 23, 2015 · 74 comments

Comments

@dupuy
Copy link
Contributor

@dupuy dupuy commented Jun 23, 2015

This issue was inspired by this question on Google Groups.

I've got some Terraform code that doesn't work because the EC2 instance running the Docker daemon doesn't exist yet so I get "* Error pinging Docker server: Get http://${aws_instance.docker.public_ip}:2375/_ping: dial tcp: lookup ${aws_instance.docker.public_ip}: no such host" if I run plan or apply.

There are providers (docker and consul - theoretically also openstack but that's a stretch) that can be implemented with Terraform itself using other providers like AWS; if there are other resources in a Terraform deployment that use the (docker or consul) provider they cannot be provisioned or managed in any way until and unless the other resources that implement the docker server or consul cluster have been successfully provisioned.

If there were a depends_on clause for providers like docker and consul, this kind of dependency could be managed automatically. In the absence of this, it may be possible to add depends_on clauses for all the resources using the docker or consul provider, but that does not fully address the problem as Terraform will attempt (and fail, if they are not already provisioned) to discover the state of the docker/consul resources during the planning stage, long before it has completed the computation of dependencies. Multiple plan/apply runs may be able to resolve that specific problem, but having a depends_on clause for providers would allow everything to be managed in a single pass.

@bitglue
Copy link

@bitglue bitglue commented Oct 22, 2015

fleet is another example of a service for which this is a problem.

That provider used to work: if the ip address is an empty string, then it would use a mock API that failed on everything. But that solution is no longer working in Terraform 0.6.3.

@apparentlymart
Copy link
Member

@apparentlymart apparentlymart commented Oct 22, 2015

I think this is describing the same issue I wrote up in #2976, in which case unfortunately the problem is a bit more subtle than supporting depends_on on the provider.

Terraform is actually already able to correctly handle provider instantiation in the dependency graph, correctly understanding that (in your example) the docker provider instantiation depends on the completion of the EC2 instance.

The key issue here is that providers need to be instantiated for all operations, not just apply. Thus when terraform plan is run, Terraform will run the plan for the AWS instance, noting that it needs to create it, and then it will try to instantiate the Docker provider to plan the docker_container resource, which of course it can't do because we won't know the AWS instance results until apply.

When I attempted to define this problem in #2976 I was focused on working with resources that don't really have a concept of "creation", like consul_keys or template_file, rather than things like aws_instance, etc. There really isn't a good way to make that EC2 instance and docker example work as long as we preserve Terraform's strict separation between plan and apply.

The workaround for this problem is to explicitly split the problem into two steps: make one Terraform config that creates the EC2 instance and produces the instance IP address as an output, publish the state from that configuration somewhere, and then use the terraform_remote_state resource to reference that from a separate downstream config that sets up the Docker resources.

Unfortunately if you follow my above advice, you will then run into the issue that I described in #2976: the terraform_remote_state resource also won't get instantiated during plan. That issue seems solvable, however; terraform_remote_state just reads some data from elsewhere and doesn't actually create anything, so it should be safe to refresh it during plan and get the data necessary to populate the provider configuration before the provider is instantiated.

@bitglue
Copy link

@bitglue bitglue commented Oct 22, 2015

@apparentlymart: In that issue, you are describing a case "which is very handy when parts of the config are set dynamically from outside the configuration." And you propose to get around the issue by making those resources that represent the outside configuration "pre-refreshed", meaning you skip the create step and immediately go to the read step.

But I'm describing a case where parts of the config are set dynamically from inside the configuration. For example, I want to manipulate a Docker or Fleet service that exists on the EC2 instance I just made. Would pre-refreshing help in this case?

@apparentlymart
Copy link
Member

@apparentlymart apparentlymart commented Oct 22, 2015

@bitglue no, as my (rather verbose) comment described, having a "true" resource from one provider be used as input to another is not compatible with Terraform's model of separating plan from apply. The only way to solve that, without changing Terraform's architecture considerably, is to break the problem into two separate configurations and then use terraform_remote_state (which can potentially be pre-refreshed, but isn't yet) to pass resource data between the two.

@bitglue
Copy link

@bitglue bitglue commented Oct 23, 2015

edited a bit to clarify the plan/apply stages

@apparentlymart I don't think it's impossible, because terraform-provider-fleet used to do it successfully. When the configuration is being planned for the first time, the provider doesn't actually need to do anything, which is why it can use a Fleet API object which fails all the time. None of the provider methods get called because it's not necessary: if there's no prior state, then the plan is trivially to create everything, and you don't need to call any provider methods to know that.

After the plan is made and it's time to apply, then the provider can be initialized after having created the EC2 instance, and now it has a proper endpoint and can actually work and create the fleet resources.

On subsequent planning runs, the public IP address of the EC2 instance is already known, so planning can happen as usual. Bonus points for refreshing the EC2 instance before initializing the fleet provider to do its refreshing.

I'd also think it's not the separation of plan and apply that's really the issue here, but more specifically refresh. You can always terraform plan -refresh=false and that will work even if the providers can't connect to anything, right? Assuming the state file is accurate, of course.

You can run into a little trouble if you delete the EC2 instance to which fleet was connecting but after the fleet resources have been created. Now plan can't work. But there are two ways to resolve that situation:

  1. Give the fleet provider the IP address of another EC2 instance in the same fleet cluster which hasn't been deleted.
  2. If the entire fleet cluster has been deleted and so there is no IP address you could give it, then all the fleet units have been deleted, too. So you can delete all the fleet units from the state file.
  3. Assuming you have an accurate (enough) state file, create the plan without refreshing and apply it. Then the missing EC2 instance will exist again and things will be back to normal.

Granted, these resolutions require a little hackish manual action, but it's not a situation I ever hit in practice. I'm sure with a little refinement it could be made less hackish.

@apparentlymart
Copy link
Member

@apparentlymart apparentlymart commented Oct 23, 2015

@bitglue it sounds like you're saying that in principle the providers could tolerate their configurations being incomplete until they are asked to do something. That is certainly theoretically true... while today most of them verify their config during instantiation and fail hard if the config is incomplete (as you saw the Docker provider do), they could potentially just let an incomplete config pass and then have it fail if any later operation tries to do any operations.

So one thing we could prototype is to revise how helper/schema handles provider configuration errors: in Configure, rather than returning an error when the ConfigureFunc returns an error, it could simply remember the error as internal state and return nil. The Apply and Refresh functions would then be responsible for checking for that saved error and returning it, so that Diff (which, as you noted, does not depend on the instantiated client) can complete successfully.

Having a prototype of that would allow us to try out the different cases and see what it fixes and when/how it fails. As you said, it should resolve the initial creation case because at that point the Refresh method won't be called. The case I'm less sure about -- which is admittedly an edge case -- is when a provider starts of with a totally literal configuration, and then at some later point it's changed to depend on the outcome of another resource; in that case Terraform will try to refresh the resources belonging to that provider, which will presumably then fail.

@bitglue
Copy link

@bitglue bitglue commented Oct 23, 2015

@apparentlymart That's more or less my thinking, yeah. Though from what I've observed trying to get terraform-fleet-provider working again, in many circumstances the failure happens before ConfigureFunc is even called, so we might need a different approach.

@apparentlymart
Copy link
Member

@apparentlymart apparentlymart commented Oct 23, 2015

@bitglue I guess the schema validation will catch cases where a field is required but yet empty, so you're right that what I described won't entirely fix it unless we make all provider arguments optional and handle them being missing inside the ConfigureFunc.

@mtekel
Copy link

@mtekel mtekel commented Feb 25, 2016

This is issue for postgresql provider as well, when you e.g. want to create AWS RDS instance and then use the port in the provider configuration. This fails, as the provider initializes before RDS instance is created, the port number is returned as "" and that doesn't convert to int:

  * provider.postgresql: cannot parse '' as int: strconv.ParseInt: parsing "": invalid syntax

TF code:

 provider "postgresql" {
   host = "${aws_db_instance.myDB.address}"
   port = "${aws_db_instance.myDB.port}"
   username = "${aws_db_instance.myDB.username}"
   password = "abc"
 }
mtekel added a commit to alphagov/paas-cf that referenced this issue Feb 25, 2016
Don't specify port, as the RDS instance doesn't exist yet in the
moment of postgresQL provider initialization, which then breaks,
because port is returned as empty quotes, which doesn't convert to
string. See hashicorp/terraform#2430
@mtekel
Copy link

@mtekel mtekel commented Feb 26, 2016

Interestingly, in your case TF graph does show the provider dependency, yet provider runs in parallel still. This is esp. problematic on destroy, as RDS instance gets destroyed before postgresql provider has chance to destroy the resources it has created, leaving "undestructable" state file behind. See #5340

@BastienM
Copy link

@BastienM BastienM commented Mar 11, 2016

Hello there,

I got a similar problem with the Docker provider when used inside a Openstack instance (graph).

# main.tf
module "openstack" {
    source            = "./openstack"
    user-name         = "${var.openstack_user-name}"
    tenant-name       = "${var.openstack_tenant-name}"
    user-password     = "${var.openstack_user-password}"
    auth-url          = "${var.openstack_auth-url}"
    dc-region         = "${var.openstack_dc-region}"
    key-name          = "${var.openstack_key-name}"
    key-path          = "${var.openstack_key-path}"
    instance-flavor   = "${var.openstack_instance-flavor}"
    instance-os       = "${var.openstack_instance-os}"
}

module "docker" {
    source                  = "./docker"
    dockerhub-organization  = "${var.docker_dockerhub-organization}"
    instance-public-ip      = "${module.openstack.instance_public_ip}"
}
$ terraform plan

Error running plan: 1 error(s) occurred:
* Error initializing Docker client: invalid endpoint

Even thought I'm using :

provider "docker" {
    host = "${var.instance-public-ip}:2375/"
}

Logically it should wait for the instance to be up but sadly the provider is still initialized at the very beginning ...


So as a workaround I splitted my project into modules (module.openstack & module.docker) and then execute them one at a time with the -target parameter.
Like this :

$ terraform apply -target=module.openstack && terraform apply -target=module.docker

It does the job but make the whole process quite annoying as we must always specify the modules in the good order for each steps (plan, apply, destroy ...).

So until we got an option such as depends_on, I don't see other ways to do.
Is there an update on this matter ?

@closconsultancy
Copy link

@closconsultancy closconsultancy commented May 9, 2016

I've submitted a similar question to the google group on this:

https://groups.google.com/forum/#!topic/terraform-tool/OhDdMrSoWK8

The workaround of specifying the modules separately didn't seem to work for me. Weirdly the docker provider was spinning up a second EC2 instance?! I've also noticed that terraform destroy didn't seem to take notice of the target module. See below:

#terraform destroy -target=module.aws

Do you really want to destroy?
  Terraform will delete the following infrastructure:
    module.aws

module.aws.aws_instance.my_ec2_instance: Refreshing state... (ID: i-69b034e5)
.....
Error refreshing state: 1 error(s) occurred:

* Error initializing Docker client: invalid endpoint
@apparentlymart
Copy link
Member

@apparentlymart apparentlymart commented May 10, 2016

Issue #4149 was my later proposal to alter Terraform's workflow to better support the situation of having a single Terraform config work at multiple levels of abstraction (a VM and the app running on it, as in this case).

It's not an easy fix but it essentially formalizes the use of -target to apply a config in multiple steps and uses Terraform's knowledge of the dependency graph to do it automatically.

@dbcoliveira
Copy link

@dbcoliveira dbcoliveira commented Jun 2, 2016

@apparentlymart I don't fully understand why the dependency graph does actually count on the plan step. Intuitively I would guess that the dependency matrix would be applied at all stages (including instantiation on each step) . At least for all these cases would just avoid a bunch of problems.
By other words the dependency graph should indicate if the instantiation should wait or not for certain resource.
This kind of issues just smashes eventual capabilities of the tool. Its a bit silly that with a series of posix commands it can be fixed while programmatically (using TF logic) it can't.

@CloudSurgeon
Copy link

@CloudSurgeon CloudSurgeon commented Jan 25, 2017

So, if I understand this correctly, there is no way for me to tell a provider to not configure until after certain resources have been configured. For example, this won't work because custom_provider will already be initialized before my_machine is built:
provider "custom_provider" {
url = "${aws_instance.my_machine.public_ip}"
username = "admin"
password = "password"
}

The only option would be to run an apply with a -target option for my_machine first, the run the apply again after the dependency has been satisfied.

brianantonelli added a commit to Cox-Automotive/terraform-provider-alks that referenced this issue Mar 22, 2017
but this will never work until this issue is resolved
hashicorp/terraform#2430
@derFunk
Copy link

@derFunk derFunk commented Mar 29, 2017

+1 for depends_on for providers.
I want to be able to depend on having all actions from another provider applied first.

My use case:
I want to create another database and roles/schema inside this database in PostgreSQL.

To do so, I have to connect as the "root" user first, create the new role with appropriate permissions, and then connect again with the new user to create the database and schema in it.

So I need two providers with aliases, one with root and one for the application db.
The application postgresql provider depends on the finished actions from the root postgresql provider.

My workaround currently is to comment out the second part first, apply the first part, then comment in the second part again to apply it as well. :(

# ====================================================================
# Execute first
# ====================================================================

provider "postgresql" {
  alias           = "root"
  host            = "${var.db_pg_application_host}"
  port            = "${var.db_pg_application_port}"
  username        = "root"
  password        = "${lookup(var.rds_root_pws, "application")}"
  database        = "postgres"
}

resource "postgresql_role" "application" {
  provider        = "postgresql.root"
  name            = "application"
  login           = true
  create_database = true
  password        = "${lookup(var.rds_user_pws, "application")}"
}

# ====================================================================
# Execute second
# ====================================================================

provider "postgresql" {
  alias           = "application"
  host            = "${var.db_pg_application_host}"
  port            = "${var.db_pg_application_port}"
  username        = "application"
  password        = "${lookup(var.rds_user_pws, "application")}"
  database        = ""
}

resource "postgresql_database" "application" {
  provider = "postgresql.application"
  name     = "application"
  owner = "${postgresql_role.application.name}"
}

resource "postgresql_schema" "myschema" {
  provider = "postgresql.application"
  name     = "myschema"
  owner      = "${postgresql_role.application.name}"

  policy {
    create = true
    usage  = true
    role   = "${postgresql_role.application.name}"
  }

  policy {
    create = true
    usage  = true
    role   = "root"
  }
}
@apparentlymart
Copy link
Member

@apparentlymart apparentlymart commented Mar 29, 2017

@derFunk your use case there (deferring a particular provider and its resources until after its dependencies are ready) is a big part of what #4149 is about. (Just mentioning this here to create the issue link, so I can find this again later!)

@andylockran
Copy link

@andylockran andylockran commented Oct 17, 2017

I've managed to his this same issue with the postgres provider depending on an aws_db_instance outside my module. Is there a workaround available now?

@jjlakis
Copy link

@jjlakis jjlakis commented Feb 22, 2018

Any progress here?

@johnmarcou
Copy link

@johnmarcou johnmarcou commented Mar 6, 2018

Hi all.

I had a similar issue where I am using Terraform to:
1 - deploy an infrastructure (Kubernetes Typhoon)
2 - then to deploy resources on the fresh deployed infrastructure (Helm packages)

The helm provider was checking the connection file (kubeconfig) at terraform initialisation, so before the file was created itself (this occur during the step 1).
So the helm resources creation was crashing for sure, because the provider was unable to contact the infrastructure.

A double terraform apply works though, but, here is how I manage to make it working with a single terraform apply, forcing the helm provider to wait for the infrastructure to be online, and exporting the infrastructure config file in a temporary local_file:

resource "local_file" "kubeconfig" {
  # HACK: depends_on for the helm provider
  # Passing provider configuration value via a local_file
  depends_on = ["module.typhoon"]
  content    = "${module.typhoon.kubeconfig}"
  filename   = "./terraform.tfstate.helmprovider.kubeconfig"
}

provider "helm" {
  kubernetes {
    # HACK: depends_on via an another resource
    # config_path = "${module.typhoon.kubeconfig}", but via the dependency
    config_path = "${local_file.kubeconfig.filename}"
  }
}

resource "helm_release" "openvmtools" {
  count      = "${var.enable_addons ? 1 : 0}"
   # HACK: when destroy, don't delete the resource dependency before the resource
  depends_on = ["module.typhoon"]
  name       = "openvmtools"
  namespace  = "kube-system"
  chart      = "${path.module}/addons/charts/open-vm-tools"
}

NB: This hack works because the provider expect a file path as config value.

Hope it can help.

@ap1969
Copy link

@ap1969 ap1969 commented Apr 8, 2018

Hi,
Similar issue with Rancher. The rancher provider requires the URL for the rancher host, but if the plan is to create the rancher host and some other hosts to run the containerized services, it's then impossible to:

  1. create the rancher host, and
  2. have the other hosts register with the rancher hosts.

This is due to the rancher provider failing during the terraform plan step, as it can't reach the API.

@abdennour
Copy link

@abdennour abdennour commented Jul 21, 2020

After checking the majority of solutions, I can see that all your workaround are implemented according to the provider(s).

There is no a single of truth from Terraform side how to make it work.

I am struggling , not with helm provider, but with helmfile provider.
The helmfile provider does not require any argument.
as consequence, there is no workaround :(

However, I adopt the solution of BastienM .

  • separate the prerequisite resources in a new module (module_a)
  • refactor current module (module_b) and remove resources that have been migrated to the new module.
  • then :
terraform apply -target=module.module_a && terraform apply -target=module.module_b  
@containerpope
Copy link

@containerpope containerpope commented Jul 22, 2020

@abdennour the new terraform 13 beta already includes the needed feature: see here

Terraform 0.13 highlights include:

  • Module-centric workflows are getting a boost with the count, depends_on, and for_each features of the Terraform
    confirmation language.
@Jasper-Ben
Copy link

@Jasper-Ben Jasper-Ben commented Aug 5, 2020

@abdennour the new terraform 13 beta already includes the needed feature: see here

Terraform 0.13 highlights include:

  • Module-centric workflows are getting a boost with the count, depends_on, and for_each features of the Terraform
    confirmation language.

@mkjoerg does it already, though? I tried installing Terraform v0.13.0-beta3 and adding a depends_on argument to a provider block still throws the following error:

Error: Reserved argument name in provider block

  on main.tf line 53, in provider "kubernetes":
  53:     depends_on          = [

The provider argument name "depends_on" is reserved for use by Terraform in a
future version.

P.S.: I just noticed your comment was referring to the usage within modules. I'd hoped however, that this would work with providers as well 😞

@chrisjaimon2012
Copy link

@chrisjaimon2012 chrisjaimon2012 commented Aug 6, 2020

Similar issue, I'm trying to create an account using a provider, and then use another provider to deploy resources into it.
Is there any other way to get this working?

### master account's provider ###
provider "aws" {
  region = var.aws_region
}
resource "aws_organizations_account" "account" {
  name  = "my_new_account"
  email = "john@doe.org"
  iam_user_access_to_billing = "ALLOW"
  role_name                  = "org-access"
  parent_id                  = var.root_id
}

### member accounts provider ###
provider "aws" {
  region  = var.aws_region
  alias   = "infra"
  assume_role {
    role_arn = "arn:aws:iam::${aws_organizations_account.account.id}:role/org-access"
  }
}
resource "aws_s3_bucket" "b" {
  provider = aws.infra
  bucket = "my-tf-test-bucket"
  acl    = "private"
}
@MoritzKn
Copy link

@MoritzKn MoritzKn commented Aug 7, 2020

@chrisjaimon2012 unfortunately this use case is currently not properly supported. Current way to do this would be using two terraform projects. But it's a good example of why this is needed.

@gibsonje
Copy link

@gibsonje gibsonje commented Sep 25, 2020

This is well discussed so far but adding another scenario.

I create an AWS Managed Active Directory instance, then I want to create AD resources within it using the new active directory provider.

Same situation as creating an RDS instance and wanting to use a provider to manage resources related to that database.

Same situation as creating an EKS cluster and wanting to create kubernetes resources within it.

Some situations it makes more sense to split into a different folder. But sometimes I just want to get this done in one apply. Active Directory is one of those situations.

@Lucasjuv
Copy link

@Lucasjuv Lucasjuv commented Sep 29, 2020

Just adding another case here:

I have a terraform config that deploys an AKS instance and installs some helm charts on AKS. So far no problems but when I needed to use the kubernetes-alpha to deploy some CRDs it simply tries to connect to the AKS cluster before it being deployed. Gives me a GRPC Error.

@FromZeus
Copy link

@FromZeus FromZeus commented Oct 30, 2020

Just adding another case here:

I have a terraform config that deploys an AKS instance and installs some helm charts on AKS. So far no problems but when I needed to use the kubernetes-alpha to deploy some CRDs it simply tries to connect to the AKS cluster before it being deployed. Gives me a GRPC Error.

Good example, I'm facing that issue right now, trying to install cert-manager and then create cert-issuer with kubernetes-alpha, failing with Error: rpc error: code = Unknown desc = no matches for cert-manager.io/v1alpha2, Resource=Issuer due to the wrong order of resource execution.

@ademariag
Copy link

@ademariag ademariag commented Nov 4, 2020

@mtekel @derFunk FYI I was in despair but it appears lazy initialisation for terraform-provider-postgresql is coming soon hashicorp/terraform-provider-postgresql#199

@thenonameguy
Copy link

@thenonameguy thenonameguy commented Nov 19, 2020

The above PR for the postgresql provider got merged, but not released.
I needed this so I published it as a fork for the time being.
You can use it like this:

terraform {
  required_providers {
    postgresql = {
      # FIXME: check new official release for lazy connections: https://github.com/cyrilgdn/terraform-provider-postgresql
      source = "thenonameguy/postgresql"
      version = "2.0.1"
    }
  }
}

provider "postgresql" {
  # Configuration options
}

Thanks for Deep Impact for sponsoring this work.

Hoping for a more general solution eventually.

@godacre
Copy link

@godacre godacre commented Nov 29, 2020

Adding another case, although this scenario has already been briefly touched upon above:

If you have a simple TF project which looks to provision a new EKS cluster and then also tries to deploy k8s resources to the newly created cluster, this cannot be achieved through a single "terraform apply". Instead, as it is understood that providers are initialised during the plan phase, the example project would have to be split into two separate configurations, or layers. For anyone who has also come across this, I found the scenario and (a) solution detailed here: https://blog.logrocket.com/dirty-terraform-hacks/ - Break up dependent providers into staged Terraform runs.

From my experience of this scenario in an enterprise setting, I can say that having isolated TF state & configs for different "layers" of an app infrastructure can actually be quite handy. For example, if you are doing a lot of development work regarding TF deployed k8s resources and find yourself doing a lot of destroys & apply's, it is a lot easier to do this if all of your k8s resource config are separated from the rest of your infrastructure which enables those k8s deployments to exist (EKS, ASG, VPC etc..). It saves you having to specify --target as much and also protects the more static parts of your infra that you probably don't want to be dropping/spinning-up every 5 mins..

Opened a discussion @ https://discuss.hashicorp.com/t/eks-gke-aks-kubernetes-resources-provider-dependency/18217

@iTaybb
Copy link

@iTaybb iTaybb commented Jan 19, 2021

Would like to see this supported.

@patuzov
Copy link

@patuzov patuzov commented Jan 22, 2021

Just adding another case here:

I have a terraform config that deploys an AKS instance and installs some helm charts on AKS. So far no problems but when I needed to use the kubernetes-alpha to deploy some CRDs it simply tries to connect to the AKS cluster before it being deployed. Gives me a GRPC Error.

"So far no problems" - there is already a problem before you even use the kubernetes-alpha.

After creating the cluster and any resource in it, if you change a field in the cluster configuration that leads to recreate, you will get an error during refresh:

"http://localhost/api/v1/namespaces/ingress": dial tcp 127.0.0.1:80: connect: connection refused

Here is a minimalistic script to reproduce it: gist

Can someone explain what exactly is happening here? Somehow during initial create, it gets correctly the order of creating resources. But what happens at refresh? Why can't it find the cluster at refresh already?

@doctor-eval
Copy link

@doctor-eval doctor-eval commented Feb 24, 2021

I'm really new to Terraform - but, in the case of Postgresql and Docker providers, at least, the problem seems to be that the provider assumes that it's connecting to some static IP address. I'd suggest that docker and postgresql hosts should be resources, not part of the provider configuration. It actually seemed really weird to me that the configuration for these providers contained a hostname. I don't see why a PG (or Docker) host is different from any other resource.

I know I'm probably missing something super obvious, but in terms of the difference between plan and apply, given a putative "postgresql_db" which refers to a "postgresql_host" which references a "vultr_vps", clearly if the VPS isn't up, the host isn't up, and the database isn't up, so the plan needs to recreate the host and then the database.

Anyway I'm sorry if this has all been hashed out already. For the record, I want to deploy thousands of PG servers and Docker hosts; every VM I spin up needs one of each. In this context, it seems nuts that I'd have to create thousands of providers with unique aliases... it feels like the assumption is that we have one big, precious docker or PG host, but that's not how it works in my place.

@lukemarsden
Copy link

@lukemarsden lukemarsden commented Mar 16, 2021

I was able to workaround this by specifying depends_on for the output of one module, using that output as an input to another module, and specifying the provider configuration in the second module in terms of that input.

https://www.terraform.io/docs/language/values/outputs.html#depends_on-explicit-output-dependencies

@jbg
Copy link

@jbg jbg commented Mar 16, 2021

That works, but configuring providers inside modules referred to by other modules is deprecated since v0.11 so I guess it might stop working one day.

@lukemarsden
Copy link

@lukemarsden lukemarsden commented Mar 16, 2021

@jbg interesting, would you mind sharing a link to where this deprecation is documented please?

To be clear, I'm not configuring one module's provider from within another module. Each module just configures providers that only it uses. It's just that one module outputs a kubeconfig that is used by the second module as provider configuration, and the kubeconfig doesn't exist until the first module has done its provisioning.

In any case, it would be great if terraform provided a supported way to solve the "provision a thing, then provision another thing inside the thing you just provisioned" use case before deprecating the workarounds that we have to support this.

Doing teardown in the right order for this is another pain point, but I've got it working with judicious use of depends_on pointing to whole objects exported between modules to enforce the correct ordering for destroy.

@jbg
Copy link

@jbg jbg commented Mar 16, 2021

https://www.terraform.io/docs/language/modules/develop/providers.html

A module intended to be called by one or more other modules must not contain any provider blocks, with the exception of the special "proxy provider blocks" discussed under Passing Providers Explicitly below.

For backward compatibility with configurations targeting Terraform v0.10 and earlier Terraform does not produce an error for a provider block in a shared module if the module block only uses features available in Terraform v0.10, but that is a legacy usage pattern that is no longer recommended.

Since v0.11, providers are supposed to only be configured in the root module, and then passed (implicitly or explicitly) into modules that the root module calls.

AFAIK the "supported" ways to do the "provision a thing, then provision another thing inside the thing you just provisioned" workflow are to either apply a set of changes, then add the dependent changes and apply again (not ideal) or to split your terraform configuration into multiple terraform configurations that interact at arms length (which actually works quite well).

@lukemarsden
Copy link

@lukemarsden lukemarsden commented Mar 16, 2021

Thanks! How is the arms length interaction pattern meant to work, particularly with respect to parameter passing e.g using the output from one configuration, such as a kubeconfig, for the input to another? Is it up to the system driving terraform apply to do that passing or can they share a state file and reference another's output?

I really appreciate your prompt and helpful responses, thank you!

@jbg
Copy link

@jbg jbg commented Mar 16, 2021

https://www.terraform.io/docs/language/state/remote-state-data.html is one option - pull outputs from one root module as a data source in another root module.

@mabunixda
Copy link

@mabunixda mabunixda commented Mar 16, 2021

Use this within TFC or TFE you can use a similar solution

@j94305
Copy link

@j94305 j94305 commented Mar 16, 2021

This is a somewhat old thread but unfortunately, this is still an issue.
Meanwhile, we are resorting to Terragrunt to give the extra layer of defining process steps across Terraform scripts.
There should be a different way of modelling some things as resources instead of providers. This way, they can be the result of execution steps and dependencies can be respected accordingly. The model of providers is for resource-type objects (e.g., databases) too limited in the current Terraform framework.

@doctor-eval
Copy link

@doctor-eval doctor-eval commented Mar 16, 2021

AFAIK the "supported" ways to do the "provision a thing, then provision another thing inside the thing you just provisioned" workflow are to either apply a set of changes, then add the dependent changes and apply again (not ideal) or to split your terraform configuration into multiple terraform configurations that interact at arms length (which actually works quite well).

What I'm trying to do is provision a VPS that runs Docker, and then provision things on Docker. IMO the problem is in the modelling of the Docker provider - moreso than Terraform itself. There doesn't seem to be a good reason that the Docker endpoint isn't just a resource.

I seem to remember that the Postgresql provider works the same way - the provider needs a hostname. But in my environment the provider doesn't even exist until the VPS is created... why is the PG or Docker provider defined at the IP address level when the VPS they live on, isn't?

From the outside it looks like these providers have been built with a specific use-case in mind - "using terraform to manage docker containers" - rather than the general use case that Terraform is so good at - "using terraform to build infra".

@gothrek22
Copy link

@gothrek22 gothrek22 commented Apr 15, 2021

Another example up for consideration:

resource "null_resource" "kubeconfig" {
  provisioner "local-exec" {
    command = <<LOCAL_EXEC
export KUBECONFIG="${path.root}/kubeconfig"
aws eks update-kubeconfig eks-cluster
LOCAL_EXEC
  }
}

provider "kubernetes" {
  alias            = "eks"
  config_path      = "${path.root}/kubeconfig"
  depends_on = [
    null_resource.kubeconfig
  ]
}

provider "helm" {
  alias = "eks"
  kubernetes {
    config_path      = "${path.root}/kubeconfig"
  }

  depends_on = [
    null_resource.kubeconfig
  ]
}

Instead of repeating an almost the same provider definition when using both kubernetes and helm providers, allow making things a bit more dry by using a depends on between a resource and providers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet