Standalone provider cache server

[erpel]

Summary

Starting from the interest in #3231 this is making the case for providing a terragrunt subcommand to start a stand alone provider cache server. In the discussion, creating a RFC is suggested.

Having a single cache server instance serving many parallel terragrunt invocations would improve efficiency on Terraform Automation & Collaboration Software (TACOS) platforms like Atlantis.

Motivation

Organisations running self hosted TACOS platforms or similar services like Atlantis for merge request workflow integration often find themselves with systems that will run terragrunt plan/apply and other commands frequently and in parallel.
Existing provider caching has proven lacking for such situations, which lead to terragrunt adding provider caching functionality.

For these systems having a single cache storage location is the most efficient way to use the cache. Past issues have shown that spinning up many cache servers in parallel pointing to the same directories can lead to locking issues among others.

Running a single permanent cache server on the automation system host allows for the most efficient use of cache. All terragrunt processes launched could connect to the same cache server serving providers from a unified cache location without locking issues.

Proposal

Introduce a new subcommand like terragrunt cache-server. This command does nothing but start a cache server and not return unless a fatal error is encountered or signal to stop is received.
Server parameters can be set using established terragrunt configuration methods like command line arguments or environment variables. These parameters include cache dir location, registry configuration, listening host/port and the authentication token.

Users of a system like atlantis could add the cache server process to the host/pod running atlantis and extend the atlantis configuration to invoke terragrunt in pipelines with the settings needed to connect to the standalone cache server. This would include enabling caching, providing a server URI and authentication token. Adding these options would differ depending on the actual TACOS used.

Technical Details

• terragrunt
  • New command added. Expected to mostly hook in to existing functionality.
  • Option to instruct terragrunt to connect to an existing server when needed instead of starting a new one

Press Release

Standalone provider cache server for efficient TACOS hosting

Terragrunt introduces the ability to run a standalone cache server giving TACOS operators more control to ensure efficient reuse of downloaded providers.

A single cache server process enables limitless parallelism enabling operators to scale workflow automation efficiently with minimal overhead.

Standalone cache server is available as of [RELEASE]. To learn more about how to integrate with your selfhosted TACAS, check the documentation.

Drawbacks

Operations for the additional component of the cache server increases the overhead for teams providing TACOS. This includes but is not limited to keeping it up to date and monitoring availability.

Terragrunt might need to improve handling situations where a cache server is supposed to be used but can't be reached, added complexity will complicate troubleshooting in some scenarios.

Sharing a cache server with untrusted entities could be enabled through this and might bring security issues like cache poisoning into the setup.

A long running cache server shared across many terragrunt invocations may increase the requirements for the cache server implementation itself compared over running a server for shorter times with limited scope.

Alternatives

• Using the cache server as is per terragrunt invocation
• Falling back on OpenTofu/terraform built in caching options that may improve in the future
• Implementing some form of provider caching outside of the realm of terragrunt

Migration Strategy

None required

Unresolved Questions

Are there other use cases for this outside of hosting systems that run terragrunt as a service integrated into team workflows (TACOS)?

Do other systems bring additional requirements to be able to integrate a standalone server?

Would running a central server on a network location be a useful scenario? This might introduce many additional security considerations compared to running via localhost only.

References

• Too many requests (429) on local cache server with high parallelism #3234
• Parallel terragrunt runs with provider cache conflict and fail #3292
• Terragrunt Provider Cache not working properly when using it with Atlantis #3076
• Investigate: Avoid Starting Provider Cache Server When it Doesn't Help #3325

Proof of Concept Pull Request

No response

Support Level

• I have Terragrunt Enterprise Support
• I am a paying Gruntwork customer

Customer Name

No response

[yhakbar]

Deleted a comment that was likely an attempt to get folks to download malware. Reported the user to GitHub.

[yhakbar]

How do you imagine this external cache server is hosted, @erpel ? As a separate container/server with a file system mount to allow for the cached providers to be accessed?

Can you explain why multiple Terragrunt processes are running instead of a run-all invocation? If I'm understanding right, that would result in one Terragrunt process spinning up one go routine for the server, and multiple routines for the underlying OpenTofu/Terraform executions, right?

[erpel]

Thanks for your interest.

In our situation I'd like to add the cache server as a separate container in the Atlantis pod. Configure it to listen on localhost and using an EFS file system mounted in both the main and the cache container at the same path.
The question about the file system made me realize that a cache server on a different host makes no sense, as file system access for both sides, cache server and client is required.

Our setup with Atlantis has one instance covering several repositories and some are "monorepos" with many teams working on them. We're not using any of the run-all commands at the moment, our structure is not laid out in a way that makes that immediately useful. Even with that remedied, unrelated MRs would still be running as separate invocations, so terragrunt is likely to always be active multiple times in parallel.

[gnuletik]

We are also looking for this feature but for a different use-case.

When we are running multiple terragrunt commands like this :

terragrunt apply -target 'aws_s3_bucket_policy.bucket_policy[0]'
terragrunt apply -target 'aws_s3_bucket_policy.bucket_policy[1]'

We are waiting for the provider cache to start / stop for each command.

On projects with many providers, starting the provider cache can be slow (~30 seconds).
This is quite cumbersome having to wait for the provider cache to start between each commands when it could be re-used between commands.

This can become impossible to use the provider cache if you have a lot of these commands to run.

[alonalmog82]

I +1 all of the above use cases - they all apply to our implementation.

I'll add that even without a TF collaboration framework, a single developer working on a dozen terragrunt modules, planning and testing multiple times, I would have enabled a local cache server on my laptop. Avoid the 2 sec spin-up time X 100 plans I would do that day, it's annoying.

[p5]

Throwing thoughts out there. Could this just utilise an existing object storage, rather than being an entire new service to host? Like configuration which sets the equivalent of something like:

plugin_cache_dir = "s3://my-cache-bucket/plugin-cache"

The administrator can then choose whether this is an S3 or MinIO bucket, or some K8s volume management service (Longhorn? Not too familiar with K8s)

[alonalmog82]

P5, using an external object store defeats the purpose of using a local cache.
Additionally, this will require setting an additional object store, and force us to handle authentication with it.

[mwos-sl]

So one of the listed alternatives is:

Using the cache server as is per terragrunt invocation

Question: are mutliple https cachce servers running in parallel are thread safe?
I understand it is not super efficient to have running several same https proxy servers, but for now I'm more interested if this at least solves the locking problem?