- 
                Notifications
    
You must be signed in to change notification settings  - Fork 10.1k
 
Description
While testing Terraform 0.15-alpha for the Nomad team's E2E nightly tests, I encountered a panic when interrupting a remote-exec provisioner that was hung (for reasons that are related to my terrible script and not Terraform). This resulted in a panic that I've narrowed down to a file provisioner that was waiting for the remote-exec provisioner with a depends_on.
Terraform Version
$ terraform version
Terraform v0.15.0-alpha20210107
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.22.0
+ provider registry.terraform.io/hashicorp/local v2.0.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/tls v3.0.0
Terraform Configuration Files
My testing configuration is at hashicorp/nomad#9759. But a more minimal reproduction is as follows:
resource "null_resource" "example" {
  connection {
    type            = "ssh"
    user            = "vagrant"
    host            = "127.0.0.1"
    port            = 2222
    private_key     = file(".vagrant/machines/linux-ui/virtualbox/private_key")
  }
  provisioner "remote-exec" {
    inline = ["sleep 10"]
  }
}
resource "null_resource" "example_depends_on" {
  depends_on = [
    null_resource.example
  ]
  provisioner "file" {
    source      = "main.go"
    destination = "/tmp/main.go"
  }
  connection {
    type            = "ssh"
    user            = "vagrant"
    host            = "127.0.0.1"
    port            = 2222
    private_key     = file(".vagrant/machines/linux-ui/virtualbox/private_key")
  }
}
Debug Output
Trace log: https://gist.github.com/tgross/73d8d660c98dd080d02d3d38c88e74b0
Crash Output
Crash log: https://gist.github.com/tgross/605e8a3121b0733216701bd0cc67a9ac
$ TF_LOG=trace TF_LOG_PATH=out.log terraform apply -auto-approve
null_resource.example: Creating...
null_resource.example: Provisioning with 'remote-exec'...
null_resource.example (remote-exec): Connecting to remote host via SSH...
null_resource.example (remote-exec):   Host: 127.0.0.1
null_resource.example (remote-exec):   User: vagrant
null_resource.example (remote-exec):   Password: false
null_resource.example (remote-exec):   Private key: true
null_resource.example (remote-exec):   Certificate: false
null_resource.example (remote-exec):   SSH Agent: true
null_resource.example (remote-exec):   Checking Host Key: false
null_resource.example (remote-exec):   Target Platform: unix
null_resource.example (remote-exec): Connected!
^CInterrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...
Stopping operation...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2d044e7]
goroutine 48 [running]:
github.com/hashicorp/terraform/builtin/provisioners/file.(*provisioner).Stop(0xc0006a34e0, 0x0, 0x0)
        /home/circleci/project/project/builtin/provisioners/file/resource_provisioner.go:183 +0x67
github.com/hashicorp/terraform/terraform.(*Context).watchStop.func1(0xc0005d02a0, 0xc0005d0300, 0xc0005d0240, 0xc0005ac780)
        /home/circleci/project/project/terraform/context.go:848 +0x59f
created by github.com/hashicorp/terraform/terraform.(*Context).watchStop
        /home/circleci/project/project/terraform/context.go:802 +0xc5
!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.310624677.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.
When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.
SECURITY WARNING: the "crash.310624677.log" file that was created may contain
sensitive information that must be redacted before it is safe to share
on the issue tracker.
[1]: https://github.com/hashicorp/terraform/issues
!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
Expected Behavior
When interrupting, stop file provisioner gracefully. The behavior on Terraform 0.14.3 looks like this:
$ terraform apply -auto-approve
null_resource.example: Creating...
null_resource.example: Provisioning with 'remote-exec'...
null_resource.example (remote-exec): Connecting to remote host via SSH...
null_resource.example (remote-exec):   Host: 127.0.0.1
null_resource.example (remote-exec):   User: vagrant
null_resource.example (remote-exec):   Password: false
null_resource.example (remote-exec):   Private key: true
null_resource.example (remote-exec):   Certificate: false
null_resource.example (remote-exec):   SSH Agent: true
null_resource.example (remote-exec):   Checking Host Key: false
null_resource.example (remote-exec): Connected!
^CInterrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...
Stopping operation...
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Actual Behavior
When interrupting the file provisioner, there's a panic.
Steps to Reproduce
Deploy a Vagrant machine as a test target. Update the private key strings in the main.tf file to point to the correct file.
terraform init
terraform apply
Wait for the remote-exec to start, and then hit Ctrl-C