Terraform with large set of resources take very long time to run

[yzargari]

Hello,

I'm provisioning resources on a single tfstate. It's been growing steadily and I'm now at ~2900 resources. I've seen an increase in the time it takes to run the plan as I added more resources - with most time it just seem to hang before refreshing the state (I suppose it is walking the graph to understand resource dependencies) and some time after the refresh ends (before printing out the plan).

I reached a place where it takes 20-25 minutes just to run the plan and I'm starting to wonder if this is a normal behavior or a bug.

I've also tried running the plan with TF_LOG set to trace and I'm noticing many times it just sits there and not print anything. Most noticeably between these lines of log:

2018/10/03 08:38:54 [TRACE] Graph after step *terraform.graphTransformerMulti:
2018/10/03 08:42:41 [TRACE] Graph after step *terraform.ModuleVariableTransformer:

2018/10/03 08:42:53 [TRACE] Graph after step *terraform.graphTransformerMulti:
2018/10/03 08:46:22 [TRACE] Graph after step *terraform.ModuleVariableTransformer:

2018/10/03 08:46:34 [TRACE] Graph after step *terraform.OutputTransformer:
2018/10/03 08:50:03 [TRACE] Graph after step *terraform.ModuleVariableTransformer:

2018/10/03 08:50:51 [TRACE] Graph after step *terraform.graphTransformerMulti:
2018/10/03 08:54:18 [TRACE] Graph after step *terraform.ModuleVariableTransformer:

Has anyone else notice this? If anyone from the HashiCorp team wants to have a look - I can provide my TF files for review...

Terraform version v0.11.7

[bennycornelissen]

I’ve experienced this in previous versions of terraform. Since then I’ve reworked my setup to consist of multiple separate plans by grouping resources by lifecycle. This made my terraform runs quicker, and also made updating things a lot safer since every plan has a limited scope.

Whether or not this is a bug: I don’t think so. Refreshing the state in your case means a ton of API calls. That takes time. If you’re unlucky: a lot of time.

[yzargari]

Thanks @bennycornelissen, I obviously get it that making a ton of API calls take time - but the majority of the time I have to wait isn't even when the API calls are made, but rather before that (when TF is waling the graph)...

I'll reiterate saying that in trace logging, I get several occasions where it just hangs - whereas in trace logging, I'd expect it to always print out what it's doing. I can't say that TF isn't doing anything during these times as CPU is at 100% during that time...

I'll edit my initial report with some trace lines...

[bennycornelissen]

@yzargari I apologize if my comment seemed disrespectful. I meant no disrespect.

Reading your initial post again, I understand the waiting occurs before actually refreshing anything? So like this:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

==> Long wait occurs here <==

aws_s3_bucket.blog: Refreshing state... (ID: some-blog-bucket)
...

Am I correct? I would think you're right Terraform is busy walking the graph.. With 2900 resources (and possibly a ton of dependencies between those?) I can imagine this will take some time.. whether or not 25mins is usual behavior, I don't know. In my case it was stuck at that specific point for anywhere between 5 and 10mins (much less resources though), but it needed to consume a fair bit of remote state.

[yzargari]

@bennycornelissen your comment was more than OK and did not seem disrespectful. I appreciate any feedback I can get...

[jbardin]

Hi @yzargari,

Sorry you're having trouble with this. Terraform isn't currently optimized for large sets of resources, primarily because managing large sets of resources in a single config has other operational challenges, so we tend to recommend smaller, more modular configs.

Large graphs, especially ones that are highly connected (a few thousand nodes can easily create many millions of edges), take considerable resources to traverse and manipulate. There will be some more performance improvements in this area eventually, but it's generally advised to avoid such large configs for now.

[yzargari]

Hello @jbardin, thanks for the feedback.

The idea of having such large config is to manage entire environments and verifying they're set properly in a single TF execution... We have a single terraform set of configuration which we apply on different environment, so we get the same config across our dev, staging and production environment (with certain changes to scale according to the needs). If I split my 50+ microservices to 50+ terraform sets of configs - I'd have to run terraform 50 times to check if changes are needed.

I just wanted to make sure this use case is something the Hashicorp team is aware of...

[beatcracker]

I'd have to run terraform 50 times to check if changes are needed.

@yzargari But then you could run them in parallel!

[mblahay]

I want to let the developers know that I to am having this problem, and I have less than 1000 resources. My configuration is just creating hundreds of buckets using the Google provider and it is not desirable to split the management of said resources into multiple state files. I too am noticing high processor usage while the application is in a state where nothing is printed out by the trace logging.

[yzargari]

sounds like the same issue I was having. I ended up creating my own solution for the problem - which essentially involves writing a wrapper script for terraform that splits the large state file to a much smaller state with just the resources I want to apply, then creating a temp dir with just the necessary files (containing vars, locals, providers, etc and the resource I want to apply) and applying on that temp dir. When done, the script merges the smaller state back into the bigger state. This doesn't solve the problem when you need to apply on the entire stack, but for us we usually need to run on a single module.
Bottom line is reduction of most runs from ~30 minutes to ~1 minute...

[mblahay]

An interesting approach, especially if most of the work is additive in nature. I will look into it, thanks.