Feature proposal: Allow a module to directly contain content structured in folders

[stevehorsfield]

Current Terraform Version

Terraform v0.12.9

Use-cases

I want to be able to effectively structure content residing within a single Terraform module, to support improved developer productivity and quality. Both productivity and quality are directly tied to the cognitive load that a developer encounters when making changes. I commonly manage Terraform modules with over 100 files and 200 resources and having all of the .tf files in a single folder is burdensome, even when good naming conventions are employed and a good editor is used. When introducing a new colleague to the code, it is much harder than it needs to be to show the structure of the code.

At first glance, modules are suggested as a way of solving this but this actually creates substantial side effects and increases the size of the codebase and worsens productivity. The reasons for this are multiple. Modules are not free. Use of modules changes the way in which objects are integrated with changed interpolation structure, definition of variables and definition of outputs, as well as changing the structure of stored state.

Not all resources fit well into a module sub-structure. For example, if I have an AWS IAM role attached to an instance profile and that instance profile attached to an EC2 instance. These are resources that naturally fit together, however each time I want to extend the rights of the instance via the instance profile I need to add or adapt a clause in the profile. This might be to grant access to an S3 bucket, or to perform some action on another resource (ECR, Lambda, Route53) and so on. Very quickly the arbitrary boundary between the module and other resources in the state breaks down and forces very obscure hacky approaches. In my experience over several years with Terraform, this has led me to favour a single module approach in almost all cases.

Attempted Solutions

I have attempted to avoid the use of modules entirely and to only use folder structures for secondary content such as user-data scripts and other deployed artifacts. However this does not address the core problem.

I have used modules but I've often ended up removing them as they cause more issues than they solve.

I have looked at using build-time designs, and this is certainly possible but hard to make generic for all Terraform users as you end up building a secondary folder structure that is composed from the original. See also https://medium.com/datamindedbe/avoiding-copy-paste-in-terraform-two-approaches-for-multi-environment-infra-as-code-setups-b26b7251cb11,

Proposal

Add direct support for included folders into the logic of Terraform itself. This is a natural extension of the existing design and does not need to introduce any unexpected behaviour or side effects if implemented well.

#22971 is a start for this.

This is very similar in definition to #7823 but I feel that commentary deviated from the original request. Note that @mitchellh originally stated:

There is an "include" core enhancement request that I agreed with so I think we will eventually support something like this.

The design is very simple:

folder {
  source = "some-path" # includes all .tf files in that folder in the same way that Terraform usually works
}

The behaviour is identical to if the referenced .tf files were in the same folder as the referencing module.

Ideally, but perhaps optional, would be to also adapt interpolations based on paths so that they were relative to the included folder.

The design of this feature has zero impact on the behaviour of modules, providers or other interpolations with the possible exception of how modules are packaged (which must already include anything referenced relative to the source folder).

References

• Add include functionality #1478
• Feature Proposal: Subfolder support that is backward compatible #7823
• Initial PR: Adding 'folder' directive for structuring module content on disk #22971

[teamterraform]

Thanks for sharing this proposal, @stevehorsfield!

We want to be very clear up front that we have a strong preference for finding ways to meet your use-cases within the existing module concept as opposed to adding a new alternative that significantly overlaps with the capabilities of modules, since our primary concern with this proposal is how to give a clear message to new users about when they would use either approach.

Therefore we'd like to begin this discussion by understanding exactly why module composition isn't an appropriate solution in your case. In order to understand that, it would be great to see some motivating examples of situations where modules didn't work for you and why that was, so that we can then use that as a basis for evaluating a variety of different solutions to the underlying problems.

Thanks again for sharing this!

[stevehorsfield]

Certainly.

So let me start by saying that I: (a) do use modules in some cases; (b) use multiple Terraform state files (and corresponding repositories) where the overall coupling between resources is low. I'll start by expanding on this before going into the module issue.

Multiple Terraform state files

So, using multiple Terraform state files makes sense when you want to reuse existing state and you can accept that plans will not show secondary effects. For example, if I have one state file that contains administrative account definitions and another state file that describes resources in an environment then there is little overlap and the former are not subject to frequent create/destroy cycles. This means that the Terraform plans will usually show anything significant and act as an effective guard against human error. In contrast, if I have a lot of separate state files for individual resources (such as EC2 instances, RDS servers, load balancers, VPC security groups), the plans can actually do significant harm without it being apparent. This understanding drives me toward larger state files. I get a lot of benefit from the Terraform plan-apply checks, alongside usual technical review. I've seen teams not take this approach and they quickly lose sight of the damage they can be causing by making what seem to be innocuous changes.

When modules make sense

So now, I have a large state file and I would certainly like to use modules to simplify. One example is that I might use a module to pull in several Terraform remote state resources and to expose the useful outputs. This can be a reasonable effort as it changes infrequently and has relatively few inputs and few outputs. It's not less readable than using the remote state directly and the module syntax can make it clear that this is something imported. Of course, I have no indication before executing a plan if something in that repo has changed and what the impact is. And in any case, it's too late. I needed that input before changing the other repo (hence my point above).

I might also use modules if I have a repeating design that is really intended to be exactly the same each time, or has some defined modularity. An example might be exposing a website via a CloudFront distribution with its associated WAF objects, Route53 DNS entries and so on. I really have a choice here about using the same repo or putting into one or more other repos. This is based on whether I need to also adjust resources that are not in the module when making changes to the module.

That's great and it works where there are clear boundaries between functionality that are one-way, and ideally with few points of connection. Once I have to start describing tens or more input variables and exposing as many outputs, then that's a lot of wiring logic and actually is less readable than just having the resources in-line.

So on to some concrete examples

Let me take for example a set of EKS worker nodes. I don't want a module to represent a single resource (such as an EC2 instance) and I also don't want to tie all of my instances to a single configuration as this makes it hard to evolve them safely. So let's assume I have a blue-green arrangement and I do use modules:

module eks-workers-blue
- resource aws_autoscaling_group - eks-nodes-blue-zone0
- resource aws_autoscaling_group - eks-nodes-blue-zone1
- resource aws_autoscaling_group - eks-nodes-blue-zone2
- resource aws_launch_configuration - eks-nodes-blue
- data template_file eks-nodes-blue-userdata
module eks-workers-green
- resource aws_autoscaling_group - eks-nodes-green-zone0
- resource aws_autoscaling_group - eks-nodes-green-zone1
- resource aws_autoscaling_group - eks-nodes-green-zone2
- resource aws_launch_configuration - eks-nodes-green
- data template_file eks-nodes-green-userdata
- (file content) launch script
resource aws_security_group - eks-node
resource aws_iam_role eks-nodes
resource aws_iam_role_policy eks-nodes
resource aws_iam_instance_profile eks-nodes
data aws_iam_policy_document "eks-nodes-assume"
data aws_iam_policy_document "eks-nodes"
(file content) custom-ca-certificates
(file content) custom-s3-content-1
(file content) custom-s3-content-2

This is a case where modules just about work. I'd still argue that I get no added value from using modules. I'm not reusing any content, I need two distinct copies anyway. With modules I still have to add all the wiring of variables and outputs. Most of the variables are already existing either as resources in the root module or as variables that are already exposed at the root. But I have to add a lot of extra boilerplate content that adds no functional value, makes it harder to read and changes the interpolation syntax as well. If I want to add a new S3 location for the instance profile, I have to change it in the module but worse I also have to add an input and a variable for this, when I should just be referencing it directly. If I wanted the impedance, I would have chosen to use a different repository. I'm not using modules here to get productivity for repeated content, only for readability by structuring into folders. It's worse, because instead of referencing something as aws_security_group.eks-control-plane.id which has obvious meaning, it becomes var.eks-control-plane-security-group-id which doesn't tell you where it actually comes from.

Note that I've still got quite a bit of stuff in the root module because it isn't just used by the included modules. I can't just keep adding security groups because AWS has a limit on this, and I don't want to duplicate content that I share with other instances as part of a standardised approach to user-data.

Things get worse when you start looking at bidirectional content such as security group rules. Security groups don't work well inside modules because the rules are symmetric. Allow ingress from security group A into security group B. Allow egress from security group A to security group B. I need both security groups to exist but one is associated with a load balancer, for example, and the other with an instance. This leads to very weird module structure such as:

resource "aws_security_group" "a" { ... }
resource "aws_security_group" "b" { ... }
module "thing-a" {
...
  security_group_a_id = "${aws_security_group.a.id}"
  security_group_b_id = "${aws_security_group.a.id}"
}
module "thing-b" {
...
  security_group_a_id = "${aws_security_group.a.id}"
  security_group_b_id = "${aws_security_group.a.id}"
}
resource "aws_security_group_rule" "a-egress-b" {
  security_group_id = "${aws_security_group.a.id}"
  type = "egress"
  source_security_group_id = "${aws_security_group.b.id}"
  ...
}
resource "aws_security_group_rule" "b-ingress-a" {
  security_group_id = "${aws_security_group.b.id}"
  type = "ingress"
  source_security_group_id = "${aws_security_group.a.id}"
  ...
}

This would be much more readable with just a folder to contain the security group rules or at least no concern about ordering.

## Summary

I understand this isn't a perfect description of the issue. In my experience, my Terraform state is constantly evolving and grows in complexity over time. Using modules adds to the complexity and also makes it harder to restructure things (not least due to having to manually manipulate the state data). This is not on par with refactoring efforts in any other modern programming language. It's very risky and I think unnecessarily complex.

Modules are great where there is very low coupling and/or high re-use potential.

These are nuances but in my experience, the benefit of modules just doesn't pay off when compared to the price. If you made an option in a module to participate in the calling module directly, that would be great, but then it wouldn't really be a module would it?

[sandangel]

I agree that coupling modules and folders makes the code much harder to structure.
For example I have a module ECS cluster that I reuse in multiple places, like cluster for prometheus grafana, cluster for consul, cluster for launching production, staging...

This module contains policies for auto-scaling cluster, autoscaling group, user-data, launch template, AMIs...

I want to organize auto-scaling policies to its own folder (7~8 .tf files), but I don’t want to introduce another policies module because I want all the configuration to be only at the top level module (not in both top level module and nested module)
And I don’t want to reuse this auto-scaling policies as a standalone module either, it should always couple with this ECS cluster module and doesn’t make sense to use somewhere else (some metrics names are only used in ECS cluster).

I end up putting all the policy .tf files in the same folder with other resource files, which makes harder for me whenever I want to find and change some policies config.

[divad1196]

Hi,
I went with a quit similar proposal in another issue and thanks to @crw I landed up on this topic.
I was wondering why this topic went idle for more than 2 years now?

I think that this proposal is a bit hard to grasp, there are a lot said in it when we can address the matter in a simpler manner: We can not organize our files in a simple way.
E.g. If I want a compute instance in a subfolders, how can I attach it to a network declared in the parent folder? I will have to pass it explicitly. Same goes for providers.

I don't think that we should need another tool as terragrunt for this kind of purpose.

Also, I would say that we may want to reference a specific file and not only a whole folder. So the "folder" keyword wouldn't suit well here and I would use "include" instead.

Thank you for the response.

[balusarakesh]

Hi,

We are also interested in this feature, we shouldn't be using another tool like terragrunt for this purpose.

• Thank you

[Manah7]

Hi,

I think that this feature can be useful to a lot of users, especially those managing medium-sized infrastructures. I am also interested by this feature.

Thank you.

[b0ric]

Would be really great to have this feature.
It's not always convenient to create a module and the inability to use sub-directories feels rather counter-intuitive.

[Hecedu]

I began learning Terraform in 2023, and as a daily user in 2024 I think this feature would improve my workflow significantly.

[andrew-j-hagner]

Current implementation seems to force you to use modules for both code re-use/abstraction (intended) and conceptual/file organization (forced)... For simple, small services this may not cause too much hassle, but in practice it seems to always arise. As stated in the original issue, modules have a cost: You can use a module for organization, but ONLY if you also use it as an abstraction... by default it hides the resources created from the consumer. And getting around that by abusing outputs sure smells like an anti-pattern to me.

Many languages allow you to divide up conceptual units across files/folders for the sake of organization, e.g. C# namespaces or partial classes (vs nuget packages/libraries for abstraction/reuse). You could argue thats an artifact of tooling being designed around file systems, but w/e... it gives us some way for us humans to try to keep larger things organized.

I'd argue the current implementation causes people to often adjust their intended design to accommodate the tooling limitation.

I'd love to see some alternative organization option, be it supporting subfolders or some other way that helps keep "larger" tf projects organized without forcing abstraction. After all, if terraform is infra-as-code, shouldn't we be able to organize our code just like pretty much every other language allows?

Maybe current implementation encourages smaller projects because of this pain/cost, but maybe KISS should be the responsibility of the team rather than the tool. 🤷

[nicolajknudsen]

I'm developing a terraform provider using the terraform provider framework. In such a project the documentation can be autogenerated from the .tf files in the examples directory. However the provider .tf files has to be placed in examples/provider and resource .tf files has to be placed in examples/resources/my-resource for the documentation tool to discover them. This prevents me from actually running the example without either 1. duplicate the actual running code to .tf files outside the discovery paths for the doc autogen tool or 2. make these directories into proper modules, but then the overhead code will be included in the documentation which is of course a deal breaker.

[bflad]

Hi @nicolajknudsen 👋 Your comment appears relevant to the terraform-plugin-docs tooling and may warrant a feature request in that issue tracker or opening a topic in HashiCorp Discuss to walk through how to accomplish your use case. It should be possible to have additional Terraform configuration files in your examples directories that do not appear in the final provider documentation.