CPU high with terroform processes

[ozbillwang]

On last Friday, I run several terraform plan commands on my Macbook. I didn't use my laptop at weekend.

Today morning, I found CPU usage is high:

$ ps -ef|grep terraform
 502 47902     1   0 Fri09am ??         9:50.81 /Volumes/sd/Downloads/terraform/terraform-provisioner-file
 502 47903     1   0 Fri09am ??         9:49.94 /Volumes/sd/Downloads/terraform/terraform-provider-terraform
 502 47904     1   0 Fri09am ??         9:50.52 /Volumes/sd/Downloads/terraform/terraform-provisioner-remote-exec
 502 47905     1   0 Fri09am ??         9:49.48 /Volumes/sd/Downloads/terraform/terraform-provider-template
 502 47907     1   0 Fri09am ??         9:50.48 /Volumes/sd/Downloads/terraform/terraform-provisioner-local-exec
 502 47909     1   0 Fri09am ??         9:47.66 /Volumes/sd/Downloads/terraform/terraform-provisioner-chef
 502 62617 56876   0  9:38am ttys004    0:00.00 grep terraform

PID    COMMAND      %CPU TIME     #TH   #WQ  #PORT MEM    PURG   CMPRS  PGRP  PPID STATE    BOOSTS             %CPU_ME %CPU_OTHRS UID  FAULTS    COW      MSGSENT    MSGRECV    SYSBSD     SYSMACH
0      kernel_task  96.8 02:19:47 160/5 0    2-    927M+  0B     0B     0     0     running   0[0]              0.00000 0.00000    0    219573    0        777411627+ 705252304+ 0          0
47902  terraform-pr 34.5 10:01.71 8/1   0    26    96K    0B     1852K  47900 1     running  *0[1]         0.00000 0.00000    502  89017056+ 15       89015463+  11486+     1311       2043
47905  terraform-pr 31.0 10:00.30 8/1   0    26    92K    0B     1992K  47900 1     running  *0[1]         0.00000 0.00000    502  88938340+ 14       88936772+  11740+     1567       2194
47904  terraform-pr 30.9 10:01.43 8/1   0    26    92K    0B     1828K  47900 1     running  *0[1]         0.00000 0.00000    502  88624025+ 15       88622459+  11503+     1358       2017
47907  terraform-pr 30.5 10:01.35 9/1   0    29    96K    0B     1664K  47900 1     running  *0[1]         0.00000 0.00000    502  88974850+ 14       88973401+  11345+     1323       2031
47909  terraform-pr 30.1 09:58.38 8     0    26    96K    0B     1808K  47900 1     sleeping *0[1]          0.00000 0.00000    502  89851393+ 15       89849807+  11856+     1365       2043

So my question is, after i run terraform plan , why some terraform processes keep running in background? Why it takes so high CPU?

[apparentlymart]

It's definitely a bug if plugin processes like these keep running after the main Terraform process has exited. I've added the "bug" tag to represent this.

Do you recall if your plan command completed successfully, or if there were errors during processing? Did you interrupt (e.g. Ctrl+C) Terraform while it was running? None of these situations should cause this situation you're seeing, but understanding the context should make it easier to narrow down the cause here.

[jen20]

Hi @SydOps - are you running a build from master or a released version of Terraform?

[ozbillwang]

@jen20
The version of terraform is v0.6.9 and downloaded from terraform.io website.

$ terraform --version
Terraform v0.6.9

[ozbillwang]

@apparentlymart

Yes I did interrupted once, when run terraform plan and I only run plan command several times, not run other terraform commands.

[ozbillwang]

One more finding.

the process kernel_task is highest CPU usage, but after I killed all terraform processes, kernel_task is gone (or not on top anymore).

[phinze]

@SydOps thanks for reporting this!

Can you provide the config you were using when this scenario occurred?

[ozbillwang]

@phinze

I am not sure if I can share official *.tf files here. But I can give the github repository that we used for reference.

https://github.com/mozilla/socorro-infra/tree/master/terraform

[phinze]

Unfortunately this one is going to be difficult to run down without some way of reproducing it for analysis. As far as I can tell - what we know so far is "Interrupting Terraform at certain points can result in stray plugins in a hot loop"

[phinze]

Moving this report over to go-plugin since the problem is in that library. We're going to integrate that library directly in the next major release, and we can always backport a fix if necessary.

hashicorp/go-plugin#10

[ozbillwang]

Thanks for the updating. Thanks for your time.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.