Terragrunt fails to get outputs of dependency

[mkv-ts]

I get the following error when running "terragrunt apply" in module "bbb":

[terragrunt] 2020/09/08 19:46:04 /.../aaa/terragrunt.hcl is a dependency of /.../bbb/terragrunt.hcl but detected no outputs. Either the target module has not been applied yet, or the module has no outputs. If this is expected, set the skip_outputs flag to true on the dependency block.

There is however outputs. My code worked with terragrunt 0.23.36, but broke with 0.23.37. It doesn't work with 0.24.0 either.
Using terragrunt version 0.13.2 during all runs.

The setup is as follows:

# module "aaa":
include {
  path = find_in_parent_folders()
}

# terragrunt.hcl in parent folder
dependency "bbb" {
  config_path = "/.../bbb")
}

[yorinasub17]

If you want to use relative paths, you need to remove the prefix /. Otherwise, that gets interpretted as an absolute path.

[mkv-ts]

No, it's not relative paths - it's three dots in the example above, not two. I just removed my long path since it's not relevant.

Ah, I see there was one with only two - I'll update the post!

[yorinasub17]

Ah gotcha. Sorry for the confusion.

Can you share how you are configuring your remote state? Does accessing the remote state depend on any special processing like hooks and extra_arguments, or getting plugins? If so, does it work if you set the disable_dependency_optimization = true flag?

See the docs on dependency optimization for more info.

[mkv-ts]

The dependency "bbb" is actually used to when I define the remote state...

This is from the included terragrunt.hcl:

remote_state {
  backend = "s3"

  config = {
    bucket         = dependency.bbb.outputs.s3_bucket_name
    dynamodb_table = dependency.bbb.outputs.dynamodb_name
    ...
  }
}

Will check the dependency opt tomorrow! Thanks!

[yorinasub17]

Oh hmm that should work automatically... I'll need to investigate this to see what might have caused the regression.

For now, disabling the optimization should work.

[mkv-ts]

I tried adding the "disable_dependency_optimization = false" to all remote_state -blocks, no change!

[yorinasub17]

I spent some time digging into this, but I was unable to reproduce the particular issue you are running into.

Can you take a look at my test module and see if you can identify some differences between your setup and mine? https://github.com/gruntwork-io/terragrunt/tree/yori-investigate-dependency-regression/test/fixture-get-output/nested-optimization-dependency

Note how I am using outputs from deepdep to feed into the remote state block for dep.

[mkv-ts]

I fixed a test case and invited you @yorinasub17
I can't post this code in public...

[yorinasub17]

Thanks for providing a test case. I tried it out with tg 0.24.0, but I was unable to reproduce the issue. However, I noticed that you were using workspaces and I believe this may be the issue.

In particular, the dependency optimization will fail if the dependency is using a workspace for the state because it doesn't switch to that workspace.

So in your example, I was able to produce an issue if I use a different workspace for the S3 and dynamodb modules, but keep the default for the debug environment code that depends on it.

I was also able to successfully work around it when I added disable_dependency_optimization = true to https://github.com/mkv-ts/terragrunt-get-outputs-bug/blob/master/inits/main/terragrunt.hcl#L25 so that it used the preconfigured workspace in the cache.

As a side note, the dependency fetching will also naturally fail if you wipe the terragrunt cache, as terragrunt doesn't know to switch to that workspace when it reinitializes the cache. So if you are wiping the cache in between runs, that can also cause this issue.

[mkv-ts]

I was adding a docker image to the repo, to make sure we were running the same thing. But I couldn't recreate the issue in that new image (it still exists in our "real" docker image).

So this issue must be because of some other config we're doing somewhere... I will look into this further when I have time, and get back to you!

[brudnyhenry]

Hi,
Not sure what was the outcome of the investigation but I face the same issue.
The example structure I have looks like this

└── xxx

    ├── aks

    │   └── terragrunt.hcl

    └── aks-addon

        └── terragrunt.hcl

In my "aks-addon" module terragrunt.hcl I defined dependency

dependency "aks" {
  config_path = "../aks"
}

But when I try to start any terragrunt command inside aks-addon directory I get

[terragrunt] [/home/adamplaczek/xxx/aks] 2020/11/26 08:16:35 Running command: terraform init -get=false -get-plugins=false

[terragrunt] [/home/adamplaczek/xxx/aks] 2020/11/26 08:16:38 Running command: terraform output -json

[terragrunt] 2020/11/26 08:16:40 /home/adamplaczek/xxx/aks/terragrunt.hcl is a dependency of /home/adamplaczek/xxx/aks-addons/terragrunt.hcl but detected no outputs. Either the target module has not been applied yet, or the module has no outputs. If this is expected, set the skip_outputs flag to true on the dependency block.

But when I enter the dependency manually and start

terraform init -get=false -get-plugins=false
terraform output -json

I can see the output.

If I set disable_dependency_optimization = true on the remote_state block it works.
I'm using Azure

[khushil]

Sadly I get the same problem but for me the disable_dependency_optimization = true doesn't fix. Basically I can't delete the alb because it can't get outputs of it's security group. So I'm left with a manual delete which is odd.

[khushil]

As a side note, the dependency fetching will also naturally fail if you wipe the terragrunt cache, as terragrunt doesn't know to switch to that workspace when it reinitializes the cache. So if you are wiping the cache in between runs, that can also cause this issue.

I thought https://terragrunt.gruntwork.io/docs/features/caching/ meant we could safely get rid of these or are you talking about something else?

[JeremieCharest]

We triggered the issue when we migrated our remote state files (S3, Dynamo for locking) to another location. Before that, relative dependencies were working fine but now plan and apply are failing. Applying in the dependency folder generate outputs but the downstream module don't see them like it was used to do before.

The disable_dependency_optimization = true workaround unblocked us. We also tried to change relative path to absolute but it didn't helped.

terraform = 0.12.29
terragrunt = 0.26.2

[valdestron]

Still reproducing this with:
terraform 0.15.2
terragrunt 0.29.2

disable_dependency_optimization = true does not help

My modules was in plan stage.

I get rid of:
mock_outputs_allowed_terraform_commands = ["validate"]

and left only:
mock_outputs = {
....
}

Started to work.

// module
include {
  path = find_in_parent_folders("account.hcl")
}

locals {
  root_vars = read_terragrunt_config(find_in_parent_folders())
  region = local.root_vars.locals.default_region
}

terraform {
  source = "../../../../../modules//module"
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents  = <<-EOF
    provider "aws" {
      region = "${local.region}"
    }
  EOF
}

// deepmodule
include {
  path = find_in_parent_folders("account.hcl")
}

locals {
  root_vars = read_terragrunt_config(find_in_parent_folders())

  region = local.root_vars.locals.default_region
}

terraform {
  source = "../../../../../modules//deepmodule"
}

dependency "vpc" {
  config_path = "../vpc"
  //   mock_outputs_allowed_terraform_commands = ["validate"] <- not working
  mock_outputs = {
      internal_vpc = { id = "MOCK"}
      internal_public_subnets = { ids = ["subnet-MOCK"] }
      internal_private_subnets = { ids = ["subnet-MOCK"] }
  }
}

inputs = {
  vpc_id	        = dependency.vpc.outputs.internal_vpc.id
  network           = {
    public_subnets  = tolist(dependency.vpc.outputs.internal_public_subnets.ids)
    private_subnets = tolist(dependency.vpc.outputs.internal_private_subnets.ids)
  }
}


generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents  = <<-EOF
    provider "aws" {
      region = "${local.region}"
    }
  EOF
}

[czenker]

The issue occurred for us after updating from terraform 0.13 to 0.15 and terragrunt from 0.23 to 0.29.

👉 We could fix the issue by deleting the .terraform directory in the dependend project. Now it works even without disable_dependency_optimization = true.

This might also explain why @yorinasub17 could not reproduce the issue.

[alxy]

I had the same issue and I had the following configuration that caused this bug for me:

config = {
    key                  = "${path_relative_to_include()}/terraform.tfstate"
    resource_group_name  = get_env("REMOTE_STATE_RESOURCE_GROUP", "rg-terragrunt-backend-state")
    storage_account_name = get_env("REMOTE_STATE_STORAGE_ACCOUNT", "stterragruntstate")
    container_name       = get_env("REMOTE_STATE_STORAGE_CONTAINER", "terragrunt")
  }

This means, the backend configuration is dynamic and determined by environment variables. However, I needed to run terragrunt from an automation pipeline (in our case Azure pipelines), and then the same backend configuration is used for the modules and all the dependencies.
I think a better practice is to hardcode the configuration in terragrunt.hcl to seamlessly work with different state files for different modules:

config = {
    key                  = "${path_relative_to_include()}/terraform.tfstate"
    resource_group_name  = "rg-terragrunt-backend-state"
    storage_account_name = "stterragruntstate"
    container_name       = "deployment-stamp-eu1-dev"
  }

Hope it helps if someone finds himself in the same situation.

[naul1]

Same issue as the original post:

TG = v0.29.10
TF = v1.0.0

[yorinasub17]

@naul1 FYI terragrunt does not yet officially support tf 1.0. Please follow #1710 for when we provide support.

[yorinasub17]

I thought https://terragrunt.gruntwork.io/docs/features/caching/ meant we could safely get rid of these or are you talking about something else?

@khushil the reason the cache is no longer ephemeral if you are using workspaces is because the cache now contains environmental context (the enabled workspace) that is recorded on disk.

[yorinasub17]

@valdestron when you say

My modules was in plan stage.

Do you mean the modules were in a clean slate with no deployed infrastructure? In that case the dependency fetching is properly giving you an error because there are no outputs (because the dependent module hasn't been applied) when your configuration expects it. Using mock_outputs to support plan is the proper solution for that.

[aalmazanarbs]

@valdestron when you say

My modules was in plan stage.

Do you mean the modules were in a clean slate with no deployed infrastructure? In that case the dependency fetching is properly giving you an error because there are no outputs (because the dependent module hasn't been applied) when your configuration expects it. Using mock_outputs to support plan is the proper solution for that.

Problem continues in 2022 latest version...

mock_outputs solution is very poor since plan tells one thing and apply does other. In addition, having production code with "mocks" is very "strange".

[ludichrislyts]

In that case the dependency fetching is properly giving you an error because there are no outputs (because the dependent module hasn't been applied) when your configuration expects it.

This was the solution for me. I was trying to test changes to infrastructure using a custom prefix, and those resources did not exist in AWS. Running init and apply on the modules supplying the outputs for the dependent modules solved this issue.

[defyjoy]

Is there any movement to this ? This is a very peculiar case since this is a classic case of terraform plan and identifying dependencies. Mock is a way out though but it is a very misguiding in plan vs apply .

[camarrone]

Hello everyone.

I'm using the latest version (v0.38.7) and if I use mock_outputs terragrunt always uses the values from the mock_outputs and never detects the outputs from the dependecy, even if I set skip_outputs = false

Has anyone solved this?

Thanks.

[shbedev]

We experienced the same issue when importing resources into terragrunt.
Apparently, when running terragrunt state pull in the dependency path, we found that the "outputs" object in the state was empty. This can also be checked by running terragrunt output.
Running terragrunt refresh solves it!

[kenchan0130]

Even in my environment, running terragrunt refresh like @shbedev 's method solved this problem.

However, if the terraform project is large, for example, with many dependencies, terragrunt refresh is very slow.
Does anyone know how to optimize the execution speed by getting only the output?

[PassionateDeveloper86]

Have the same problem, tried everything but didnt get though it. For me it was a reason to not use terragrunt any more.

[ds-evo]

after terragrunt run-all refresh i could see the outputs (without mocking)

[SankethBK]

Getting same error, I am trying to run terragrunt run-all init in https://github.com/vallard/K8sClass/tree/master/terragrunt/stacks/stage using
terragrunt version v0.43.0
Terraform v1.3.7 on darwin_arm64

Full error

ERRO[0013] Module /Users/sanketh.bk/my_projects/k8s/eks/k8sclass/terragrunt/stacks/stage/eks has finished with an error: /Users/sanketh.bk/my_projects/k8s/eks/k8sclass/terragrunt/stacks/stage/vpc/terragrunt.hcl is a dependency of /Users/sanketh.bk/my_projects/k8s/eks/k8sclass/terragrunt/stacks/stage/eks/terragrunt.hcl but detected no outputs. Either the target module has not been applied yet, or the module has no outputs. If this is expected, set the skip_outputs flag to true on the dependency block. prefix=[/Users/sanketh.bk/my_projects/k8s/eks/k8sclass/terragrunt/stacks/stage/eks] ERRO[0013] 1 error occurred: * /Users/sanketh.bk/my_projects/k8s/eks/k8sclass/terragrunt/stacks/stage/vpc/terragrunt.hcl is a dependency of /Users/sanketh.bk/my_projects/k8s/eks/k8sclass/terragrunt/stacks/stage/eks/terragrunt.hcl but detected no outputs. Either the target module has not been applied yet, or the module has no outputs. If this is expected, set the skip_outputs flag to true on the dependency block.

[DanielViglione]

Yeah this error still persists even after 2 years of the original post. To get this to work, unfortunately I had to generate the outputs directly in the CI/CD environment:

  - name: Check Account Outputs
    run: |
      cd modules/account-settings/infrastructure
      terraform init
      terraform refresh
      terraform output

And then after that, the dependency block is able to retrieve the outputs. It's a terrible hack, but it is clear the underlying issue here will not be resolved for some time.

[DanielViglione]

mock_outputs is also a terrible solution for this. The output of a plan using mock outputs does not reflect the actual real plan that should be, e.g. renaming resource app-test -> app-fake. So if all I did was want to change the name of one resource, the mock_outputs would show several changes that need to take place since the real resources are suffixed with test rather than fake, for example.

[DanielViglione]

There is also an issue with using mock_outputs. If you use mock_outputs, such as a vpc_id and the source module uses a terraform data source, e.g:

data "aws_vpc" "selected" {
  id          = var.vpc_id
}

Then the plan will fail with "no matching vpc id" found, if you use a mock vpc id string, which by the name "mock_outputs" suggests you should be able to use mock strings.

[aceqbaceq]

mocking variable is terrible solution

[corjanZee]

Why does it still try to reslove the inputs whilst the skip flag is set to true and skip_outputs is set to true

[baileyivancic]

Has there been any progress on this? This is currently blocking a refactor of our infrastructure, as it will not allow an init of modules which are new and have dependencies. As others have stated, using mock_outputs seems to be a very poor solution that introduces risk between init/plan and apply.

[snorrea]

I'm facing the same issue. My project will ditch Terragrunt and go for plain Terraform. For context, to try this out, we set up a simple example repo with literally two Azure resources, a resource group and a storage account, as two different modules with a dependency on the resource group from the storage account. While I was originally excited by the prospect of having less repeated code, when fundamental functionality like this isn't in place, what's even the point? None of the workarounds work for me.
Keeping my fingers crossed this gets picked up soon, meanwhile going WET...

[alxy]

@snorrea can you share the code of that Sample repository? From my experience terragrunt works very Well, and often Times it is really just a configuration issue.

[jvanenckevort]

I have worked with Terragrunt and Terraform for a long time. Because this issue hasn't been fixed for years, I am going to move on from Terragrunt. The positive effects of Terragrunt do not outweigh the negative side-effects, for example the flawed dependency resolution. But for larger projects, I have also noted that breaking the state up doesn't really benefit the build time in terms of caching and speed. And dividing the state in different substates was exactly one of the most valuable things Terragrunt had to offer.

The problem here is that when running a terragrunt run-all plan, newly added resources or newly added attributes (i.e. outputs) are not being resolved before terragrunt.hcl files that actually use those new attributes/modules as a dependency. This is probably a fundamental issue, hence why it hasn't been resolved yet. On the other hand, a plan on a submodule outputs newly added attributes, so why can other modules not wait before their dependencies are fully planned and outputs are known?

And then comes the usual answer that you can mock outputs. I don't want to mock outputs everytime I add a new module or attribute to a module. I only want to mock when I test something, this is not something I want to put inside my production configuration files. A mistake is easily made and when a resource is renamed, it could be destroyed.

[hypervtechnics]

We experimented with examining the plan output and generating locals stubs in the terraform code using a special provider marking the corresponding variables of the outputs in the dependent module as not yet known during plan time.

This way you can still have a glance at the impact and have your changes plannable to some extent but of course with limitations. It has probably been mentioned a couple times before, but things are complex. What if e.g. you are using the kubernetes provider in the dependent module and configure it using a data block using the cluster name got from the first module? The cluster does not exist yet and the data will fail.

Terragrunt can not fix this. Instead the concept should be solved on a higher level in terraform itself. Tools like terragrunt help in splitting things up in an attempt to fix a flaw or unconsidered use case in the design of how terraform works...

[paololazzari]

I just ran into this, however there was an error on my part. This was my layout (gke module depends on vpc) that I had:

└── dev
│  ├── gke
│  │   └── terragrunt.hcl
│  │   └── cluster.tf
│  └── vpc
│      └── terragrunt.hcl
│      └── network.tf
└── modules
   └── gke
   │   └── main.tf
   │   └── variables.tf
   └── vpc
       └── main.tf
       └── outputs.tf
       └── variables.tf

$ terragrunt plan
ERRO[0002] /tmp/dev/vpc/terragrunt.hcl is a dependency of /tmp/dev/gke/terragrunt.hcl but detected no outputs. Either the target module has not been applied yet, or the module has no outputs. If this is expected, set the skip_outputs flag to true on the dependency block. 

to fix this, I added dev/vpc/outputs.tf

$ cd /tmp/dev/vpc
$ terraform apply # (to add the output)
$ cd /tmp/dev/gke
$ terragrunt plan
No changes. Your infrastructure matches the configuration.

so final layout:

└── dev
│  ├── gke
│  │   └── terragrunt.hcl
│  │   └── cluster.tf
│  └── vpc
│      └── terragrunt.hcl
│      └── network.tf
│      └── outputs.tf
└── modules
    └── gke
    │   └── main.tf
    │   └── variables.tf
    └── vpc
        └── main.tf
        └── outputs.tf
        └── variables.tf

with:

/dev/gke/terragrunt.hcl:

...

dependency "vpc" {
  config_path = "../vpc"
}

inputs = {
  vpc_id = dependency.vpc.outputs.vpc_id
}

/dev/gke/cluster.tf:

variable "vpc_id" {
  type = string
}

module "gke" {
  source            = "../../modules/gke"
  ....
  network           = var.vpc_id
  ...
}

/dev/vpc/network.tf:

module "vpc" {
  source               = "../../modules/vpc"
  ...
}

dev/vpc/outputs.tf:

output "vpc_id" {
  value = module.vpc.vpc_id
}

modules/vpc/main.tf:

resource "google_compute_network" "vpc_network" {
  ...
}

modules/vpc/outputs.tf:

output "vpc_id" {
  value = google_compute_network.vpc_network.id
}

[protonforest]

how is this a bug? imagine not being able to create multiple resources at the same time just cause they have dependencies on each other.