v1.8 is ~3x slower than v1.7.x #34984
Comments
|
Hi @y3ti, Thanks for filing the issue! Can you provide some more detail around how you are measuring the change? |
jbardin
added
the
waiting for reproduction
|
I was able to reproduce it, although the timings seem ~15% worse. The binaries were built from git using 1.7.5 1.8.0 Flamegraphs: |
|
Setup: clean up: When I reran it again: If I clean up my workspace: I see in the trace log that terraform spent 8 seconds here: |
|
Thanks for the profile data @mschfh, we will look into that! @y3ti, that's interesting, because that delay seems to be waiting for the provider plugin process to start, terraform isn't doing anything at that point. Can you check the output of |
apparentlymart
added
the
v1.8
|
I can confirm that our usual 20m plan/apply was hanging several times for more than 2 hours until we finally noticed it was due to the terraform version. |
|
@jbardin Yes, I used the exact same versions of the plugins in each case. I attempted to reproduce the issue on GitHub Actions, but everything functioned as expected (no issues): https://github.com/y3ti/terraform-1.8-test/actions/runs/8663897264 In my situation, I manage several large Terraform configurations (4000-5000 resources). Here are a few screenshots from the execution of Real Memory Usage: 6,73 GB vs 8,92 GB Terraform 1.7.5
Terraform 1.8.0
|
|
|
|
I tried to replicate the issue by using a publicly available GitHub Repository with a Terraform configuration that mimics the same test case. However, I couldn't reproduce the problem on a small-sized Terraform configuration codebase. I can reproduce the issue with larger configurations (1000+ Terraform data resources), Unfortunately, the code is not publicly available, so I can't share it. The screenshot I sent relates to this large Terraform configuration. I enabled debug logs and noticed that Terraform is stuck at this point: |
|
Thanks @y3ti. The I've tried a handful of pathological configurations I have handy, and nothing is turning up any repeatable differences yet. Maybe we can find some common feature from the worst cases here, like all running under |
|
@jbardin I have removed the Terragrunt dependency. I attempted to gather more data from the trace logs, but there was no difference (debug vs trace). Terraform 1.7.5Terraform 1.8.0As observed, there was a pause of 5 minutes in logging between PS. I tested v1.8.1 but the problem still exists. |
|
Thanks @y3ti, the pause at The first example you had paused in a different location, that however was a much smaller configuration to begin with, but given the last config is the location of the stall always the same? I'm trying to rule out some obscure runtime+OS interaction vs something that Terraform itself is doing. Another random idea is if you add the env variable |
|
Let's disregard the example from my initial comment. I attempted to use an open-source repository to provide a sample configuration that you could reproduce on your laptop. As previously mentioned, I attempted to reproduce the issue using GitHub Actions. However, the example Terraform configuration was not extensive enough, and I was unable to replicate the problem. Let's focus on the example from my previous three comments. As you suggested I ran Here is the output: |
|
Thank you @y3ti! That at least makes it clear that something is going on in there, the heap is growing throughout that pause. If anyone has a public reproduction case it would help, but knowing that v1.8 is allocating significantly more memory around that time is very useful! |
|
@jbardin I have just tested
Memory usage is also lower: 
Thank you very much! |
For me terraform plan still takes a very long time with 1.8.2 version. |
|
@fpeixotoj, if that is the case there must be a different issue you are experiencing. If you have data showing the difference, and preferably a way to reproduce the problem, please file a new issue. Thanks! |
Terraform Version
Terraform Configuration Files
Debug Output
n/a
Expected Behavior
Actual Behavior
Steps to Reproduce
git clone https://github.com/philips-labs/terraform-aws-github-runner.gitcd terraform-aws-github-runner/examples/multi-runner/time terraform validateAdditional Context
Terraform v1.8 is ~3x slower than v1.7 and consumes ~2,5x more memory.
References
No response
The text was updated successfully, but these errors were encountered: