Skip to content

file provisioner panics on SIGINT #27449

@tgross

Description

@tgross

While testing Terraform 0.15-alpha for the Nomad team's E2E nightly tests, I encountered a panic when interrupting a remote-exec provisioner that was hung (for reasons that are related to my terrible script and not Terraform). This resulted in a panic that I've narrowed down to a file provisioner that was waiting for the remote-exec provisioner with a depends_on.

Terraform Version

$ terraform version
Terraform v0.15.0-alpha20210107
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.22.0
+ provider registry.terraform.io/hashicorp/local v2.0.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/tls v3.0.0

Terraform Configuration Files

My testing configuration is at hashicorp/nomad#9759. But a more minimal reproduction is as follows:

resource "null_resource" "example" {
  connection {
    type            = "ssh"
    user            = "vagrant"
    host            = "127.0.0.1"
    port            = 2222
    private_key     = file(".vagrant/machines/linux-ui/virtualbox/private_key")
  }

  provisioner "remote-exec" {
    inline = ["sleep 10"]
  }
}

resource "null_resource" "example_depends_on" {
  depends_on = [
    null_resource.example
  ]

  provisioner "file" {
    source      = "main.go"
    destination = "/tmp/main.go"
  }

  connection {
    type            = "ssh"
    user            = "vagrant"
    host            = "127.0.0.1"
    port            = 2222
    private_key     = file(".vagrant/machines/linux-ui/virtualbox/private_key")
  }
}

Debug Output

Trace log: https://gist.github.com/tgross/73d8d660c98dd080d02d3d38c88e74b0

Crash Output

Crash log: https://gist.github.com/tgross/605e8a3121b0733216701bd0cc67a9ac

$ TF_LOG=trace TF_LOG_PATH=out.log terraform apply -auto-approve
null_resource.example: Creating...
null_resource.example: Provisioning with 'remote-exec'...
null_resource.example (remote-exec): Connecting to remote host via SSH...
null_resource.example (remote-exec):   Host: 127.0.0.1
null_resource.example (remote-exec):   User: vagrant
null_resource.example (remote-exec):   Password: false
null_resource.example (remote-exec):   Private key: true
null_resource.example (remote-exec):   Certificate: false
null_resource.example (remote-exec):   SSH Agent: true
null_resource.example (remote-exec):   Checking Host Key: false
null_resource.example (remote-exec):   Target Platform: unix
null_resource.example (remote-exec): Connected!
^CInterrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...
Stopping operation...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2d044e7]

goroutine 48 [running]:
github.com/hashicorp/terraform/builtin/provisioners/file.(*provisioner).Stop(0xc0006a34e0, 0x0, 0x0)
        /home/circleci/project/project/builtin/provisioners/file/resource_provisioner.go:183 +0x67
github.com/hashicorp/terraform/terraform.(*Context).watchStop.func1(0xc0005d02a0, 0xc0005d0300, 0xc0005d0240, 0xc0005ac780)
        /home/circleci/project/project/terraform/context.go:848 +0x59f
created by github.com/hashicorp/terraform/terraform.(*Context).watchStop
        /home/circleci/project/project/terraform/context.go:802 +0xc5




!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.310624677.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.310624677.log" file that was created may contain
sensitive information that must be redacted before it is safe to share
on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Expected Behavior

When interrupting, stop file provisioner gracefully. The behavior on Terraform 0.14.3 looks like this:

$ terraform apply -auto-approve
null_resource.example: Creating...
null_resource.example: Provisioning with 'remote-exec'...
null_resource.example (remote-exec): Connecting to remote host via SSH...
null_resource.example (remote-exec):   Host: 127.0.0.1
null_resource.example (remote-exec):   User: vagrant
null_resource.example (remote-exec):   Password: false
null_resource.example (remote-exec):   Private key: true
null_resource.example (remote-exec):   Certificate: false
null_resource.example (remote-exec):   SSH Agent: true
null_resource.example (remote-exec):   Checking Host Key: false
null_resource.example (remote-exec): Connected!
^CInterrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...
Stopping operation...


Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Actual Behavior

When interrupting the file provisioner, there's a panic.

Steps to Reproduce

Deploy a Vagrant machine as a test target. Update the private key strings in the main.tf file to point to the correct file.

terraform init
terraform apply

Wait for the remote-exec to start, and then hit Ctrl-C

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions