Exploratory Idea: Terragrunt as the Open Source version of Terraform Going Forward?

[josh-padnick]

Intro

Gruntwork co-founder and Chief Product Officer here. I'm the decision maker on where we take the Terragrunt project going forward.

In #2658, we've been thinking through what the recent HashiCorp license change means both for Terragrunt and the broader Terraform community at large. In this issue, I'd like to explore an unusual idea: What if Terragrunt were to be the open source path for Terraform going forward?

Disclaimer

I'm optimizing for speed here, and somewhat thinking out loud. These are not polished ideas; feedback is very welcome.

Impact on the Terraform ecosystem

Many of us in the Terraform ecosystem have built our companies and careers around the assumption that Terraform would be open source forever. The recent license change is like the ground beneath our feet changing, and we now need to chart a new path forward.

Each of the so-called "TACOS" providers (SpaceLift, Env0, Scalr, Terrateam, Digger, ControlMonkey, and parts of the Gruntwork offering) all share a common challenge: How can we offer a CI/CD solution for Terraform if Terraform itself is no longer open source? Here are the options I'm aware of:

1. Come up with technical workarounds that honor the spirit of the new BSL v1.1 license. If we can't host or embed Terraform, is there some way we can distribute our offering in a way that does not constitute hosting or embedding?
2. Find an alternative path forward with open source Terraform. Terraform v1.5.5 is MPLv2, but all future versions are going to be BSL v1.1. So if option 1 doesn't work, then perhaps there is some kind of open fork of Terraform that all the TACOS can use?

It's this option 2 I'd like to explore.

Terragrunt as an open path forward for Terraform

Terragrunt essentially reads a custom set of Terragrunt configuration files, prepares the right files, and figures out what vanilla Terraform commands to run. Historically, we've assumed that Terraform would continue to improve and we would gradually change the feature set in Terragrunt to match. But what if those feature improvements happened in Terragrunt instead of Terraform going forward?

As an example, let's take a look at the features in Terraform v1.6.0:

• "terraform test is now a first-class feature."
  • We could build this kind of support into Terragrunt or our other popular open source tool, Terratest.
• "config: Terraform can now track some additional detail about values that won't be known until the apply step, such as the range of possible lengths for a collection or whether an unknown value can possibly be null."
  • This implements Refined Unknown Values, allowing some operations with unknown values to produce known results hashicorp/terraform#33234). This is a core Terraform improvement, and it's not obvious to me how Terragrunt would support this unless Terragrunt becomes the execution engine itself.

Initial conclusions

Based on the above "thinking out loud", here are some initial conclusions:

• Terragrunt could elegantly support net new functionality. For example, adding support for a new type of testing is a natural fit for Terragrunt.
• Supporting core execution improvements would require a heavier lift. I guess that Terragrunt would now have to start taking over some of the core execution? Perhaps Terragrunt embeds the latest MPLv2 version of Terraform and then accepts contributions from the community to gradually improve this? Of course, we'd also need to invest a lot more resources in Terragrunt to handle this load.

Open source philosophy

We'd have to discuss this internally before I can make any commitments, but as the Gruntwork Chief Product Officer, I can say that based on what I know today, I'd be comfortable with us making a public commitment to keeping Terragrunt FOSS forever. There are already Gruntwork competitors benefitting from Terragrunt who effectively contribute no value in return and yeah, that kind of sucks. But I like the idea that Gruntwork products compete on their own merits and we enjoy a privileged position as the project maintainer who presumably has their product act together.

Terragrunt roadmap

We've been actively thinking through the updated vision for Terragrunt. Speaking candidly, we've made mostly incremental improvements over the past few years, but recently we've been considering what a more significant investment in Terragrunt might look like, and this new HashiCorp license change twist has emerged as a key consideration.

Update: For vanilla Terraform users

An important part of the approach here would be that non-Terragrunt users would be able to use "vanilla Terraform" configurations. Perhaps we'd update the subcommands so that you could run a "pure" terraform apply command or a terragrunt apply command, depending on your preference.

Feedback is welcome!

This is all very new territory for us as an industry, and again, I'm really thinking out loud here. Please do let me know your feedback.

[lorengordon]

Relevant, digger forked already: https://github.com/diggerhq/open-terraform

[josh-padnick]

Relevant, digger forked already: https://github.com/diggerhq/open-terraform

Thanks for sharing! That tells me there's some energy behind the idea of forking the last MPL version of Terraform, which is probably the direction this proposal goes. Maybe the question here is: If there's to be a Terraform fork, who should lead it, who should staff it, and what commitments would such a project need to make to build trust in the community?

Figuring out the right parameters for the community fork (if that's even the right solution here) would need some more thinking.

[jamengual]

why not to create a consortium between the competitors where the main porpuse is to keep the fork free and up to par with new terraform features without invalidating each other flavoured options?

all of you ( spacelift terragrunt , env0 etc) have the same need to have a terraform fork.

the problem I see with one company leading it is that there is room for Biased opinionated contributions that might collided with other competitor's bias/ideas and henceforth breaking the trust.

[chenrui333]

on the similar note, maybe could to have open-terraform github org or something like.

[reegnz]

Just a "wild" idea: maybe coordinate on a terraform fork under the CNCF umbrella? At this point a lot of users would be uncomfortable with a single vendor supervising the project, a foundation sounds like the proper governance format. That's the kind of public commitment that's actually convincing enough that you guys also won't pull a BUSL later on. Terraform is ripe for a foundation to govern it.

CNCF is the format to follow IMHO.

[ChristophEckerle]

I agree with the ideas - one managed open source fork within CNCF, would be a powerful move.

[o6uoq]

HashiCorp is now known as CashiCorp.

[aavileli]

Do you need to pin your Terragrunt support to v1.5.5 of Terraform before the BSL licence kicks in. It would be nice to put a blog on your website if Terragrunt is affected

[josh-padnick]

@aavileli We just published a blog post at https://blog.gruntwork.io/the-impact-of-the-hashicorp-license-change-on-gruntwork-customers-5fcd096ba86a where we endorse exactly the approach you suggest.

[josh-padnick]

Update: Gruntwork supports a vendor-neutral, community-driven, truly open source Terraform, which we are pledging support to as part of OpenTF.

Going forward, Terragrunt will support OpenTF. If HashiCorp reverts their license change, we'll re-evaluate the best approach for the Terragrunt community.

See also our recent blog post: The future of Terraform must be open.

Thank you for the suggestions here in this thread. They gave us energy and confidence as we pursued an initiative that, to us also, seemed like a "wild" idea, but which we now feel is clearly in the best interest of the entire community.