Any plans for Terragrunt support?

[vainkop]

Any plans for Terragrunt support?
Terragrunt is a must have for the most part of multi environment setups.
Thanks

[yitsushi]

The system uses tf-exec to call terraform commands. The runner does not call terraform binary as a system command directly. As far as I know, Terragrunt is a wrapper around the terraform and does not have a lib for that. Right now the system is heavily depends on the API of tf-exec, and in future plans we may abstract it away, but it will still use tf-exec or the same mechanism for OpenTofu.

If we would just use terraform with exec.Command (or similar logic), we would be logical to assume we can swap out a binary and just call that, but that's not the case.

[danielkza]

Lack of terragrunt support is an absolute no-go for many users and organisations, using tf-exec knowing that it makes using terragrunt impossible is bound to hamper this projects adoption.

[artem-nefedov]

I feel like there's no need for Terragrunt, since all the problems it solves are already solved via in cluster controllers (tf-controller itself + whatever you use to apply Terraform manifests).
We did use Terragrunt originally, but when moving to tf-controller its functionality was replaced by tf-controller and flux (including multi-env setup).

[vainkop]

I feel like there's no need for Terragrunt, since all the problems it solves are already solved via in cluster controllers (tf-controller itself + whatever you use to apply Terraform manifests). We did use Terragrunt originally, but when moving to tf-controller its functionality was replaced by tf-controller and flux (including multi-env setup).

Terragrunt usefulness is off topic for this issue but I'll comment that you simply cannot replace it with pure terraform unless you do a lot of refactoring & extra boiler code writing which doesn't make any sense for really complex multi env setups, especially existing ones.

[artem-nefedov]

you simply cannot replace it with pure terraform

True. That's why I didn't suggest that, but rather said the problems are solved in cluster controllers.
Some examples:

• Using Kustomize (supported by flux, argocd, and native kubectl), you can apply same Terraform manifests multiple times with env-specific overlays from a single entrypoint.
• Using Helm, you can achieve similar things with templating.
• Using Kyverno, you can replicate one resource across multiple namespaces, while changing some fields.
• ...etc.

Naturally it will still need some work to migrate.

[danielkza]

@artem-nefedov All these techniques would would lock you into this controller, make local deployments impossible or extremely difficult, and introduce non-standard tooling into a world with already extremely complex tooling. Claiming this controller can replace terragrunt is just not based on reality on any large organisation, and makes the chance that this can compete with Atlantis nil.

[kvendingoldo]

At this moment if you need to have a support of Terraform as well as OpenTofu (and Terragrunt :) ) in one tool you can use https://github.com/tofuutils/tenv which my team wrote some months ago. A lot of users switched to that tool to unify version management in the world of Terraform.

You're welcome to open any issues or contribute to tenv.

[vainkop]

At this moment if you need to have a support of Terraform as well as OpenTofu (and Terragrunt :) ) in one tool you can use https://github.com/tofuutils/tenv which my team wrote some months ago. A lot of users switched to that tool to unify version management in the world of Terraform.

You're welcome to open any issues or contribute to tenv.

So how does it help to introduce Terragrunt support into this controller?

[ilithanos]

At this moment if you need to have a support of Terraform as well as OpenTofu (and Terragrunt :) ) in one tool you can use https://github.com/tofuutils/tenv which my team wrote some months ago. A lot of users switched to that tool to unify version management in the world of Terraform.
You're welcome to open any issues or contribute to tenv.

So how does it help to introduce Terragrunt support into this controller?

It doesn't, it offers a completely different solution rather than help introduce terragrunt to this controller.

I get that migrating existing large terragrunt infrastructure deployments to the tofu-controller isn't something likely to happen, but the tofu-controller does solve the same issues, just in a different way more native to the gitops/flux/k8s stack, and is a good tool for solving that issue as well.

Terragrunt in the tofu-controller isn't likely to happen as it's build around tf-exec/tofu-exec.

I could see a terragrunt controller happening as a seperate project under the flux-iac organization, but I don't see it as a good fit for the tofu-controller specifically.

[vainkop]

It doesn't, it offers a completely different solution rather than help introduce terragrunt to this controller.

But to be completely honest here, terragrunt in the tofu-controller isn't likely to happen as it's build around tf-exec/tofu-exec and solves the issues terragrunt also solved in a different way.

I could see a terragrunt controller happening as a seperate project under the flux-iac organization, but I don't see it as a good fit for the tofu-controller specifically.

Terragrunt is Terraform with added benefits. You can run regular Terraform files without a terragrunt.hcl, but you're not tapping into Terragrunt's full potential that way. You can write a lot of repetitive Terraform code replicating what Terragrunt does & breaking DRY principle but why do that if there's Terragrunt?
Adding Terragrunt support into this controller would be a win-win, enhancing functionality without drawbacks, so I can't agree with your point.

Regarding tenv: it's all about managing versions of existing tools, which doesn't "offer a completely different solution" & it doesn't quite fit into this discussion & issue.

[ilithanos]

We won't ever agree on the terragrunt in tofu-controller discussion since you are basically repeating the same arguments over and over again without understanding the counter argument.

tenv is an alternative for the problem of having tofu, tf and terragrunt versions coexisting. It's not that different a problem to what you are suggesting we do in the tofu-controller by implementing support for a 3rd way of parsing Terraform. But let's leave that one alone for now.

You really like terragrunt, I get it, but it doesn't change that it's not a fit into the tofu-controller. It's been discussed so many times now, and the result still ends up being the same. It's not a good fit, and technically would require a rewrite of most of the codebase. It's not just simply replacing a binary.

I'm all for a similar solution to the tofu-controller for deploying and running terragrunt based iac specifically. But that should be a seperate project under the flux-iac community.

There is already a topic in our GH discussions where chanwit also suggested that as a solution to the community wanting a controller for terragrunt. But such a project would require community members with skill, time and patience to drive it.

If I had the time I would gladly help get that started, but I don't. My time is better spend on getting the tofu controller to where it needs to be, as I only have time for one large open source project currently. The most I can offer is to guide and help any maintainers/community members that can spend the time maintaining such a controller. And as far as I can tell, most of our maintainers are in the same boat. Remember we are maintaining this on our own free time now, it's no longer a company owned project.

[vainkop]

You really like terragrunt, I get it, but it doesn't change that it's not a fit into the tofu-controller.

It's not about me liking or disliking Terragrunt, but about only Terragrunt doing the job at scale & following DRY principle while pure Terraform just don't.
It was summarized by another commenter here

All these techniques would would lock you into this controller, make local deployments impossible or extremely difficult, and introduce non-standard tooling into a world with already extremely complex tooling. Claiming this controller can replace terragrunt is just not based on reality on any large organisation, and makes the chance that this can compete with Atlantis nil.

Now you're saying:

technically would require a rewrite of most of the codebase. It's not just simply replacing a binary
such a project would require community members with skill, time and patience to drive it
If I had the time I would gladly help get that started, but I don't
Remember we are maintaining this on our own free time now, it's no longer a company owned project.

Which boils down to some people not having time for it.
But at the same time you acknowledge that

There is already a topic in our GH discussions where chanwit also suggested that as a solution to the community wanting a controller for terragrunt

And I see no arguments for your claim that

it's not a fit into the tofu-controller

because from what you said it seems that if all of a sudden some other community members decided to implement that & found time for it rewriting most of the codebase then community would welcome the result because the controller will be able to run pure Terraform + Terragrunt same way as Terragrunt does.

Regarding

tenv is an alternative for the problem of having tofu, tf and terragrunt versions coexisting. It's not that different a problem to what you are suggesting we do in the tofu-controller by implementing support for a 3rd way of parsing Terraform.

There is no problem tofu, terraform & terragrunt binaries coexisting in the system. If you want to have a different version you can simply swap the binary or use your project but it's a completely different issue which doesn't seem related here at all.

There is a lot of value in Terragrunt + GitOps & I wish that people involved in development of now called tofu-controller could implement support for Terragrunt simply because they seem to be much closer than others to achieving that ✌🏻

[ilithanos]

@vainkop it is about you liking the way terragrunt does things and locking you into that tool rather than being locked in to how the tofu-controller does things. It's about a request for a different way of doing this at scale, than the way the tofu-controller is solving these issues.

There are currently no plans for terragrunt being part of the tofu-controller, but there are the option of writing a seperate controller to handle that.

Terragrunt and tofu-controller solves the same issue in two different ways, and neither of them are standard terraform when it comes to handling states and dryness of terraform at scale.

It's two different approaches of solving the problem of terraform at scale. And currently they are mutually exclusive. If you like the way tofu-controller works, test it out and use that to solve the problem. If you like the way terragrunt works, use that.

This is actually about you ( and others in the same boat as you ) preferring how terragrunt handles terraform at scale and that is fine to, but that doesn't change the fact that it's a bad fit for the tofu-controller.

Terragrunt does templating, statemangement and more in a way that would conflict with the main design of the tofu-controller project. There are more issues to this than just a simple it's a matter of time. And as such it should be a separate controller under the flux-iac project if such a controller was to be created. So my suggestion will be to take that discussion to the GH Discussions page instead of this issue, and try looking into if you could help making that happen rather that trying to force us as maintainers to agree with your idea that it needs to be within the tofu-controller.

The tofu-controller could be used as a starting point for creating such a terragrunt-controller, but it should not be part of the tofu-controller directly.

I'll be closing this issue for now, as there are currently no plans of supporting this within the tofu controller, but we are open to continue the discussion of creating a seperate controller that can handle this usecase. Please move that discussion to the GH discussion found here:

GH discussion of terragrunt support options

[vainkop]

try looking into if you could help making that happen

Unfortunately my level of Golang isn't yet good enough for such an adventure.

rather that trying to force us as maintainers to agree with your idea that it needs to be within the tofu-controller

That's called "proving one's point of view," I ain't forcing anyone. Your point of view has been taken from the first message & I disagree.

If you like the way tofu-controller works, test it out and use that to solve the problem. If you like the way terragrunt works, use that.

I've tried the now called tofu-controller & unfortunately it doesn't fit the tasks me & my team are working with in many projects. But we are using terragrunt in many many projects including the ones with pretty complex multi account multi component infra some of which are under relatively high load & basically the missing reconcilation loop is a pain which we partly solve by doing continious apply which works but doesn't look too good.

[kvendingoldo]

@vainkop

There is no problem tofu, terraform & terragrunt binaries coexisting in the system. If you want to have a different version you can simply swap the binary or use your project but it's a completely different issue which doesn't seem related here at all.

It's a big problem, if you have 10+ projects with different versions of OpenTofu/Terraform. It's not a simple task to keep 20 different binaries and switch between them.

[vainkop]

There is no problem tofu, terraform & terragrunt binaries coexisting in the system. If you want to have a different version you can simply swap the binary or use your project but it's a completely different issue which doesn't seem related here at all.

It's a big problem, if you have 10+ projects with different versions of OpenTofu/Terraform. It's not a simple task to keep 20 different binaries and switch between them.

That's only a problem if you're deploying from a local machine which I rarely do. I'm not saying that tenv & similar projects aren't useful, just not relevant to this issue.