Data sources should allow empty results without failing

[clstokes]

This is an enhancement request. Data sources should allow empty results without failing the plan or apply. This would allow for fallback or fallthrough scenarios that are needed in certain situations. An optional allow_empty_result attribute on the data source would support many use cases where a datasource value is not explicitly required. The specific attribute name isn't important to me.

As an example, the configuration below currently fails because the aws_ami datasource will not find any results. However I still want the apply to continue using a fallback AMI.

data "aws_ami" "main" {
  most_recent = true

  filter {
    name   = "name"
    values = ["blah-blah"]
  }
}

output "ami" {
  value = "${coalesce(data.aws_ami.main.id,"ami-abc-123")}"
}

(This is just an example so please don't get hung up on this specific use case.)

[dohoangkhiem]

We desperately needed this for a long time, couldn't find an effective way to deal with such resources that could be created only once (for example, security group in AWS) and used later in other configurations (while creating a separate configurations for such shared resources are not reasonable this case).

[rwaffen]

i am sitting right at the same problem. would really be helpful if empty returns where possible!
(and yes, i am right at that "search ami and don't find one, but want to have backup value"-problem 😏 )

[sclausson]

Agree that this would be useful. I would like to use "aws_ebs_snapshot" data source to look for a snapshot that matches on some filters and create a volume from it. But if none is found, then I would like to create a volume from a default snapshot id.

[shazChaudhry]

Could this be a valid solution: #11782 ?

[ryanking]

@shazChaudhry that appears to only work on maps, yeah?

[practicalint]

I have hit this a few times. But my latest use case is in trying to get around the way that Plan cannot tell that a resource will be created in the apply when the filter values are computed, in particular when using modules. Like I am trying to build a security group list from a map using names and use a security group data source filtering on name to find the corresponding IDs. If a security group is one that will be created in the same Apply, it fails the Plan to "no matching SecurityGroup found".
In my case even though I am using some maps, the possible solution above does not work, unless I were to play games with some default security group it would find in place of looking for a to-be-created resource.
I have tried several alternative approaches to get around the various limitations I described, but like I run into all the time, each one lands on another limitation. For example, I came up with an elegant way to filter the to-be-created SGs out of the list so lookup would not be attempted in the Plan (and then I could merge the to-be-created IDs with the data resource found IDs), but that makes the resulting list computed which won't allow me to use it's length as the count variable.

I have a fair understanding of the terraform model and that the plan needs to create the graph to know what will really happen in Apply, but overall it continues to be difficult to create dynamic and DRY terraform "code" to deal with more complex/sophisticated provisioning scenarios.

[steverukuts]

I have the same problem:

1. My ECS [0] services and task definitions are provisioned using Terraform
2. An script runs as part of our CI process to update the task definition to another version

This sequence of events happens:

1. I create the services and task definitions using Terraform. The task definition revision is 1
2. The CI process executes and updates the task definition to 2. It deletes revision 1. The script copies the content of the task definition so it's identical. [1]
3. I execute Terraform apply. It attempts to set the task definition revision back to 1
4. Terraform bails out with an error

Now, I could use a data block to pull in the latest version of my task definition. Then I can do what the ecs_task_definition docs says and use ${max(... to get the latest revision.

However, if I am creating the environment for the first time this is not possible. This does not work if you are creating the environment for the first time, but it will on subsequent runs:

resource "aws_ecs_task_definition" "mongo" {
  family = "mongodb"
  # ...
}

data "aws_ecs_task_definition" "mongo" {
  task_definition = "${aws_ecs_task_definition.mongo.family}"
}

I worked around it by using a templating library (mustache) to generate my terraform resources and it only includes the data resource when we are not creating the environment for the first time. I've generated other parts of my configuration before to workaround the lack of foreach in the language (after all, Terraform is not a scripting language) but this is the first time I've had to write scripts that pay attention to state.

It would appear that data blocks are only useful if the referenced resource was created outside the environment manually or by another Terraform configuration. I understand the logic of throwing an error if the resource does not exist as a default but I feel we need an option to allow it to return a null resource. A warning in the output seems like an acceptable middle ground instead.

I'd submit a PR for that behaviour myself but I'd like to first see if there's a difference between how I'm using the software and how the maintainers intended it to be used. Perhaps my use-case is too weird for Terraform to cover it, or perhaps I'm meant to integrate Terraform more deeply in my CI pipeline. Aside from that, it's done everything else quite marvellously so far.

[0] Amazon Elastic Container Service
[1] For those unfamiliar with ECS, that's really how you have to do it. I wish it wasn't necessary to change the state of anything but there is no "gracefully reload my services please" API call.

[steverukuts]

I was actually able to work around this problem using Terraform and I no longer need to use a templating library to do what I described. I need to add a depends_on parameter to the data block:

resource "aws_ecs_task_definition" "mongo" {
  family = "mongodb"
  # ...
}

data "aws_ecs_task_definition" "mongo" {
  task_definition = "${aws_ecs_task_definition.mongo.family}"
  depends_on = ["aws_ecs_task_definition.mongo"]
}

This has the downside that every time I run terraform plan, it lists the dependent service as a planned change as it doesn't immediately know the state of the mongo data block. However, when I run the configuration, no objects are updated that don't need to be.

I'm a bit confused about why I explicitly have to write depends_on in the data block; I had assumed that referencing the task definition with ${... would be sufficient to take the dependency. It's not even mentioned that you can do this with a data resource in the documentation.

In light of this, perhaps no change is required.

[practicalint]

That is an interesting find on depends_on working in the data block. In my case and I believe others in the thread, the creation of the resource is not necessarily happening in the same root or module and the data block is using more variable data for the look-up. So I think supporting this kind of behavior (allowing not found) should still be pursued.

[BlaineBradbury]

I have the same issue. Even if NULL were not an option, it would be good if I could simply specify a default resultset or value for the data source to return if it finds no results. I could then use this value in combination with ternary logic elsewhere to make some decision.

[trilitheus]

Hitting similar issue - creating a new ASG to take over a previous one, I set the desired count to the number of instances currently in the 'to be deposed' ASG pulled from 'aws_instances' data source.
However, if I'm spinning up a new VPC for example, no previous instances exist so I'd like to set desired_count to a default value

[mightystoosh]

Bump. This is a truly needed feature to deal with already created infrastructure:

`
data "aws_vpn_gateway" "selected" {
filter {
name = "tag:Name"
values = ["doesnotexist"]
}
filter {
name = "state"
values = ["available"]
}
filter {
name = "attachment.vpc-id"
values = ["${var.vpc_id}"]
}
}

resource "aws_vpn_gateway" "vpn_gw" {
count = "${1 - signum(length(data.aws_vpn_gateway.selected.id))}"
vpc_id = "${var.vpc_id}"

tags {
Name = "makeme"
}
depends_on = ["data.aws_vpn_gateway.selected"]

}`

This should create the resource if it cannot find the data source.

Any better way of doing this?

[steverukuts]

@mightystoosh - for already created infrastructure you can import the entity to your Terraform state.

If you've only got a couple of environments to worry about that's a perfectly fine workflow and might work for you. Given you're talking about a NAT gateway I guess it will. Then, you just treat it as any old resource.

This issue is about situations where a resource may or may not exist and either possibility is totally fine. However, in the situation I described, AWS quietly added a feature to force a redeployment of a service in ECS so this issue no longer affects me personally. That said, I think it would still be useful, such as in @trilitheus' situation.

[mightystoosh]

What is best practices for adding infrastructure? Basically I want to setup a bunch of VPN connections based on a list and connect it to a single VGW.
So a single VGW would be created and then I would create a aws_customer_gateway, aws_vpn_connection, and aws_vpn_connection_route(s) for each customer.
If I could find existing infrastructure then multiple workspaces might work. If I could have variable names in resources this might work.

It'd be nice if I could iterate over a list in Terraform or even run terraform multiple times using
VGW doesn't exists
terraform apply -var 'name=TEST' ip=8.8.8.8' -var 'region=us-east-1'
VGW now exists
terraform apply -var 'name=TEST1' ip=8.8.8.4' -var 'region=us-east-1'
VGW doesn't exists
terraform apply -var 'name=TEST' ip=8.8.8.8' -var 'region=us-east-2'
VGW now exists
terraform apply -var 'name=TEST1' ip=8.8.8.4' -var 'region=us-east-2'

I'm new to Terraform and it's nice, but I don't really see a way of adding multiple endpoints easily without copying of code.