Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform plugin 1.14.0 crashing #4109

Closed
Puneeth-n opened this issue Apr 6, 2018 · 6 comments · Fixed by #4148
Closed

Terraform plugin 1.14.0 crashing #4109

Puneeth-n opened this issue Apr 6, 2018 · 6 comments · Fixed by #4148
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/firehose Issues and PRs that pertain to the firehose service.
Milestone

Comments

@Puneeth-n
Copy link
Contributor

Terraform 0.10.8
Terraform-provider-aws: 1.14.0

my crashlog is 54 MB so attaching gzip here
crash.log.gz

@bflad
Copy link
Contributor

bflad commented Apr 6, 2018

Hi @Puneeth-n sorry you ran into trouble here. Taking a look right now, but looks related to the aws_kinesis_firehose_delivery_stream resource.

Relevant panic log:

2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: panic: runtime error: invalid memory address or nil pointer dereference
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1896cb8]
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4:
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: goroutine 6968 [running]:
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*MapFieldWriter).setSet(0xc43b1f42e0, 0xc4428c8d20, 0x3, 0x3, 0x369c1a0, 0x0, 0xc43a9d4e10, 0xffffffffffffffff, 0x0)
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/field_writer_map.go:343 +0x1d8
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*MapFieldWriter).set(0xc43b1f42e0, 0xc4428c8d20, 0x3, 0x3, 0x369c1a0, 0x0, 0x0, 0x0)
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/field_writer_map.go:107 +0x2ac
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*MapFieldWriter).setObject(0xc43b1f42e0, 0xc43ff202a0, 0x2, 0x2, 0x317f1c0, 0xc4428c8c90, 0xc43a0cd3b0, 0x0, 0x1893fa4)
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/field_writer_map.go:225 +0x241
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*MapFieldWriter).set(0xc43b1f42e0, 0xc43ff202a0, 0x2, 0x2, 0x317f1c0, 0xc4428c8c90, 0x1, 0x1)
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/field_writer_map.go:109 +0x22b
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*MapFieldWriter).setList.func1(0x37e329a, 0x1, 0x317f1c0, 0xc4428c8c90, 0x0, 0x0)
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/field_writer_map.go:123 +0x139
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*MapFieldWriter).setList(0xc43b1f42e0, 0xc43d7dd530, 0x1, 0x1, 0x2e822e0, 0xc43ff20240, 0xc43a9d5590, 0x10, 0xc43a9c8d20)
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/field_writer_map.go:143 +0x242
2018-04-06T23:51:16.160+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*MapFieldWriter).set(0xc43b1f42e0, 0xc43d7dd530, 0x1, 0x1, 0x2e822e0, 0xc43ff20240, 0x1, 0x1)
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/field_writer_map.go:103 +0x10b
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*MapFieldWriter).WriteField(0xc43b1f42e0, 0xc43d7dd530, 0x1, 0x1, 0x2e822e0, 0xc43ff20240, 0x0, 0x0)
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/field_writer_map.go:89 +0x49e
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*ResourceData).Set(0xc440f02620, 0x3764e31, 0x10, 0x2e822e0, 0xc43ff20240, 0x0, 0x0)
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource_data.go:189 +0x12b
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.flattenKinesisFirehoseDeliveryStream(0xc440f02620, 0xc4315d6cc0, 0xc437d69e68, 0x0)
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_kinesis_firehose_delivery_stream.go:406 +0x6ca
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsKinesisFirehoseDeliveryStreamRead(0xc440f02620, 0x33141a0, 0xc42a91d900, 0xc440f02620, 0x0)
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_kinesis_firehose_delivery_stream.go:1568 +0x34a
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc43a9c96c0, 0xc4300b26e0, 0x33141a0, 0xc42a91d900, 0xc43e4e0b90, 0x1000f0101032101, 0x80000000000)
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:354 +0x17e
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc442164e70, 0xc4300b2690, 0xc4300b26e0, 0x63b7458, 0x0, 0x1000)
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x9a
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc43a8cd4c0, 0xc42ce87c10, 0xc42ce87e20, 0x0, 0x0)
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:549 +0x4e
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: reflect.Value.call(0xc42037ac00, 0xc4314a58d0, 0x13, 0x374cde0, 0x4, 0xc445711f20, 0x3, 0x3, 0x10036d3, 0xc4204d2000, ...)
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/usr/local/go/src/reflect/value.go:434 +0x905
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: reflect.Value.Call(0xc42037ac00, 0xc4314a58d0, 0x13, 0xc43c5ea720, 0x3, 0x3, 0xc43c5ea720, 0xc43c5ea728, 0xc438bf6d40)
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/usr/local/go/src/reflect/value.go:302 +0xa4
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: net/rpc.(*service).call(0xc44261c840, 0xc42dab2870, 0xc44198c708, 0xc435ac7880, 0xc43befd6c0, 0x2e3c8e0, 0xc42ce87c10, 0x16, 0x2e3c920, 0xc42ce87e20, ...)
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/usr/local/go/src/net/rpc/server.go:381 +0x142
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: created by net/rpc.(*Server).ServeCodec
2018-04-06T23:51:16.161+0200 [DEBUG] plugin.terraform-provider-aws_v1.14.0_x4: 	/usr/local/go/src/net/rpc/server.go:475 +0x36b

@bflad bflad added bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/firehose Issues and PRs that pertain to the firehose service. labels Apr 6, 2018
@bflad bflad added this to the v1.14.1 milestone Apr 6, 2018
@bflad
Copy link
Contributor

bflad commented Apr 6, 2018

It appears this is occurring because in the API response from Firehose the CloudWatchLoggingOptions is empty in the S3 description. This has been a source of a large portion of crashes with this resource, but we haven't been able to reproduce it in an acceptance test configuration.

Would you be able to provide a sanitized version of your configuration along with what region you're hitting?

I can patch what I believe the issue is in the code, but it would be nice to finally know how this happening and ensure it never comes back.

@Puneeth-n
Copy link
Contributor Author

Puneeth-n commented Apr 7, 2018

@bflad Here you go! If you have the patch, let me know. Ill compile the plugin and test it and can give you feedback :)

Firehose module

###############################################################################
#                                    VARIABLES                                #
###############################################################################

variable name {}

variable account_id {}

variable region {
  default = "eu-west-1"
}

variable destination {
  default = "s3"
}

variable s3_configuration {
  type    = "map"
  default = {}
}

variable role_name {}

variable enable {}

###############################################################################
#                                    MAIN                                     #
###############################################################################

resource "aws_kinesis_firehose_delivery_stream" "stream" {
  count       = "${var.enable}"
  name        = "${var.name}"
  destination = "${var.destination}"

  s3_configuration {
    role_arn        = "${lookup(var.s3_configuration, "role_arn")}"
    bucket_arn      = "${lookup(var.s3_configuration, "bucket_arn")}"
    buffer_interval = "${lookup(var.s3_configuration, "buffer_interval", 60)}"
    prefix          = "${lookup(var.s3_configuration, "prefix")}"
  }
}

data "template_file" "firehose_iam_policy" {
  template = "${file("${path.module}/iam_policy.tpl")}"

  vars {
    account    = "${var.account_id}"
    region     = "${var.region}"
    bucket_arn = "${lookup(var.s3_configuration, "bucket_arn")}"
    name       = "${var.name}"
  }
}

resource "aws_iam_role_policy" "firehose_iam_policy" {
  count  = "${var.enable}"
  name   = "firehose-policy-${var.name}"
  role   = "${var.role_name}"
  policy = "${data.template_file.firehose_iam_policy.rendered}"
}

Configuration:

module "deutschebahn_stream" {
  source = "../../terraform_modules/ct-firehose/"

  enable = "${var.enable_firehose_data_logs}"

  name        = "fulfillment-deutschebahn-${var.ct_environment_name}"
  destination = "s3"

  s3_configuration {
    role_arn        = "${aws_iam_role.firehose_role_purchase_logs.arn}"
    bucket_arn      = "${var.production_env ? lookup(var.ct_master_vpc, "purchase_logs_arn") : "arn:aws:s3:::purchase-logs-${var.ct_environment_name}"}"
    buffer_interval = 60
    prefix          = "fulfillment-deutschebahn-${var.ct_environment_name}/"
  }

  account_id = "${var.account_id}"
  role_name  = "${aws_iam_role.firehose_role_purchase_logs.name}"
}

@bflad
Copy link
Contributor

bflad commented Apr 10, 2018

The fix for this has been merged into master and will release with v1.14.1 of the AWS provider, likely tomorrow.

@bflad
Copy link
Contributor

bflad commented Apr 11, 2018

This has been released in version 1.14.1 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

@ghost
Copy link

ghost commented Apr 6, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Apr 6, 2020
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/firehose Issues and PRs that pertain to the firehose service.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants