New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
terraform destroy command is not working and seems to be hanging on Mac OS Big Sur #27350
Comments
Hi! Thanks for reporting this. I believe that you're having a real issue here and I'd like to reproduce it locally. To do that, I have to be able to run this and run it on my workstation without inventing any details in order to be confident we're seeing the same behavior. As-is, I have tried some simple create/destroys on my M1 Macbook running Big Sur and it worked, and so I'm stuck - I need to understand your specific use case. Can you please restate your reproduction case such that I can copy-paste it and run it locally? Ideally, this would use the null resource provider rather than a real provider in order to minimize external dependencies. Also, can you explain what hardware you're using? Is this on an M1 ARM platform or an X86 mac? |
I've had a similar issue on my M1 on Big Sur creating a few basic AWS resources. It isn't 100% reproducible yet, sometimes the apply/destroy works, sometimes it does not. I've not seen it on just the null provider yet, but I have seen it with just an aws vpc resource. It may be more than one issue, or a single root cause, I'm not sure. I've seen: destroy fail with an error: destroy.log TF Version: 0.14.2 TF:
Let me know if there is anything more I can provide / do to help. |
I am experiencing the same on my M1 MBP. Some runs are lightning fast. And some runs hang. I believe this has to do with M1's rosetta2 translating the x86-64 plugin. When a run hangs, the process takes 100-140% CPU. Since investigation requires reproducibility, I made this crude tester:
With this I get reliable hangs. Happening at a random iteration, but they do happen. Following output had a quick hang:
I have sampled the hanging process. The output is at https://gist.github.com/rtoma/0da8efd5c9204345fb82d72b68100cdc for someone able to interpret it. |
Searching for
This stops the issue for me. |
I am not getting the issue consistently every time. Few times it runs fine but in 20% of the cases it fails executing the same script. My hardware configuration is MacBook Air (M1, 2020) , Chip - Apple M1, RAM - 16 GB, Storage - 1 TB SSD |
Thanks.. seems like with this setting it has been working for me. I did not see the hanging issue manifesting after setting the value suggested by you |
Can confirm this solved terraform apply stage hanging issue for me on M1 chip also. |
@danieldreier So we have diagnosed this 15 months ago. Why does this remain a problem with Terraform? Why haven't the arm64 binaries been fixed? We see it constantly with Terraform 1.x versions up through 1.1.8 (latest as of this comment) on Big Sur or Monterey. |
Per #27350 (comment), this seems to be an issue with Rosetta2 and providers for Terraform that have not been compiled for @jorhett can you confirm you are seeing this issue with arm64 terraform and arm64 providers? If so, that would be a new issue as I am understanding this thread. |
That environment variable to override some behavior in the Go runtime does seem to be a suggested workaround for a number of different Go runtime bugs, some of which have already been fixed and some are still under investigation. Here are some I'm aware of:
These and others seem to trace back to golang/go#24543, which is a runtime goroutine preemption technique which on Unix systems like macOS is implemented by asking the OS to interrupt the program using signals. Signals can potentially interrupt active system calls, which must then be handled by any code that was waiting for those system calls to complete, and it seems that at least some of these problems have a root cause that some other part of the Go runtime is not properly handling being interrupted in that way. (I focused above on listing ones that seemed to be related to macOS here, but I note that there are similar problems on other platforms which include this signal-based preemption scheme.) Emulation layers such as Rosetta add extra tricky interactions to a signal-based scheme like this, because presumably Rosetta is the one initially receiving the signal, translating it to make sense in the emulated amd64 context, and delivering it to the Go runtime. Although about QEMU rather than Rosetta, golang/go#36981 is an example of such a problem which arises only in the case of user-mode emulation where there's an extra level of indirection between the OS and the program. This With all of this said, it seems like the best we can do here is keep updating to newer versions of the Go runtime as they become available. Given how many different ways these problems seem to be appearing in Go, I don't know that we'll ever be able to confidently say that all problems with this root cause have been addressed, but if we keep this issue focused primarily on the situation of running amd64 Terraform under Rosetta emulation on M1 Macs as @crw proposed then at least we have a more constrained problem space to focus on. The |
Sorry, I found that tfenv's confused handling caused us to have intermingled amd/arm64 versions in different repos. Upgrading to 2.2.3 and removing all versions solved it. Ignore my comments about it failing on arm64 |
Still the case on MacOS Monterey 12.4
Using darwin_amd64 instead of arm64 with rosetta :( Sporadically spewing errors:
|
intermittently it's happening to me too: MacOS Monterey 12.4 , M1 Pro
|
These two recent messages seem to suggest that it was the provider itself rather than Terraform CLI which crashed, although providers are subject to the same constraints Terraform CLI is: we build the The best answer is to use the executables that were built for the actual platform you are using. If you cannot do that for some reason, discussion above suggests that you can tell the Go runtime to use its old scheduling technique by setting the environment variable However, this is only a workaround and we do not intend to invest in ensuring that the |
Terraform Version
Terraform v0.14.2
Debug Output
Crash Output
Expected Behavior
terraform destroy should have destroyed the created infrastructure.
Actual Behavior
On executing the terraform destroy command on terminal, the command was in hanging state for good amount of time. Then I tried to exit out of shell by doing ctrl+c but it didn't exit
Steps to Reproduce
terraform destroy
Additional Context
I was running this command on macOS Big Sur
The text was updated successfully, but these errors were encountered: