Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get error when updating SG embedded ingress rules with description #2069

Closed
takeda-joao opened this issue Oct 26, 2017 · 39 comments · Fixed by #4416
Closed

Get error when updating SG embedded ingress rules with description #2069

takeda-joao opened this issue Oct 26, 2017 · 39 comments · Fixed by #4416
Labels
bug Addresses a defect in current functionality. service/ec2 Issues and PRs that pertain to the ec2 service.
Milestone

Comments

@takeda-joao
Copy link

Hi,

I am using terraform v0.10.8 and aws provider 1.1.0 and I am getting the following error when updating a security group ingress rules adding description field.
I am able to update the SG, but needed to remove all rules manually. Then terraform works and updates the SG. When I try to update the SG again, the error happens.
This error does not happen if the SG does not have any rules with description field.

terraform command line
$ terraform apply -target

=== output of terraform
Error: Error applying plan:

1 error(s) occurred:

  • aws_security_group.client-demo-sg-devops: 1 error(s) occurred:

  • aws_security_group.****: Error revoking security group ingress rules: InvalidPermission.NotFound: The specified rule does not exist in this security group.
    status code: 400, request id: 2bfe24f7-a980-4bbc-a58d-54e5935f57a6

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

@takeda-joao
Copy link
Author

Weird.
Made more changes, and there is a SG that is updated fine, with description included.
This second SG has same number of ingress rules of the one getting error and does not error when updated.
Making more tests.

@takeda-joao
Copy link
Author

Looks like it is related to the lenght of the description field content.
If the content lenght of the description field of all ingress rules in the SG is the same, the error does not happen.

@bflad
Copy link
Contributor

bflad commented Oct 26, 2017

Potentially related? #1959

@takeda-joao
Copy link
Author

No, it is not the lenght.

@takeda-joao
Copy link
Author

Yes, #1959 may be related.
From the second run of terraform apply, without changes, and getting error, noticed it is comparing rules with the last rule of the security group.
Thanks,

@jaymecd
Copy link
Contributor

jaymecd commented Nov 1, 2017

@takeda-joao good catch

@colin-lyman
Copy link

Same issue experienced here.

@colin-lyman
Copy link

Just tested AWS Provider 1.2 and issue still persists.

@paddycarver paddycarver added the bug Addresses a defect in current functionality. label Nov 21, 2017
@SmaranNA
Copy link

SmaranNA commented Dec 7, 2017

Any update on this issue? Im also facing this issue in my implementation.

@jaymecd
Copy link
Contributor

jaymecd commented Dec 7, 2017

it was fixed in #1959 and released as part of v1.3.1

@hugohenley
Copy link

Same issue here. Waiting for v1.3.1 to test.

@reznet
Copy link

reznet commented Dec 7, 2017

I believe this issue still exists. I created new security groups with terraform and when i re-run apply, terraform complains with

* module.webui.aws_security_group.web: 1 error(s) occurred:

* aws_security_group.web: Error revoking security group ingress rules: InvalidPermission.NotFound: The specified rule does not exist in this security group.
	status code: 400, request id: 20c88505-d009-42ba-bc30-c0444cef2e94

setting the trace flag and looking at the aws call, terraform is passing the wrong description to aws:

Action=RevokeSecurityGroupIngress&GroupId=sg-b25012c7&IpPermissions.1.FromPort=3389&IpPermissions.1.IpProtocol=tcp&IpPermissions.1.IpRanges.1.CidrIp=172.16.0.0%2F16&IpPermissions.1.IpRanges.1.Description=RDP&IpPermissions.1.ToPort=3389&Version=2016-11-15

terraform is trying to remove the rule for CIDR 172.16.0.0/16 and description RDP, but the description for that CIDR block is actually RDP from Workspaces. There is a rule for a separate CIDR block whose description is RDP

Here's a snippet from my security group:

  ingress {
      description = "RDP"
      from_port = 3389
      to_port = 3389
      protocol = "tcp"
      cidr_blocks = ["10.0.0.0/8"]
  }

  ingress {
      description = "RDP from Workspaces"
      from_port = 3389
      to_port = 3389
      protocol = "tcp"
      cidr_blocks = ["${var.workspaces_cidr_blocks}"]
  }

From the terraform plan output, I see that terraform thinks it needs to delete the rule for RDP and create a rule for RDP For Workspaces, but it doesn't.

      ingress.1615219479.cidr_blocks.#:       "1" => "0"
      ingress.1615219479.cidr_blocks.0:       "172.16.0.0/16" => ""
      ingress.1615219479.description:         "RDP" => ""
      ingress.1615219479.from_port:           "3389" => "0"
      ingress.1615219479.ipv6_cidr_blocks.#:  "0" => "0"
      ingress.1615219479.protocol:            "tcp" => ""
      ingress.1615219479.security_groups.#:   "0" => "0"
      ingress.1615219479.self:                "false" => "false"
      ingress.1615219479.to_port:             "3389" => "0"
      ingress.2147368551.cidr_blocks.#:       "0" => "1"
      ingress.2147368551.cidr_blocks.0:       "" => "172.16.0.0/16"
      ingress.2147368551.description:         "" => "RDP from Workspaces"
      ingress.2147368551.from_port:           "" => "3389"
      ingress.2147368551.ipv6_cidr_blocks.#:  "0" => "0"
      ingress.2147368551.protocol:            "" => "tcp"
      ingress.2147368551.security_groups.#:   "0" => "0"
      ingress.2147368551.self:                "" => "false"
      ingress.2147368551.to_port:             "" => "3389"

Terraform version:

Terraform v0.11.1
+ provider.aws v1.5.0
+ provider.template v1.0.0
+ provider.terraform v1.0.2

@takeda-joao
Copy link
Author

Problem persists.
I am also using provider.aws v1.5.0 and it makes impossible to run apply unless you remove de description field manually (and from terraform states).

@tsiang
Copy link

tsiang commented Dec 13, 2017

I'm seeing this problem as well. If I go ahead and delete the description in the console then delete the description field in tf state then terraform apply will run. But if I try to run apply again without changing anything, it will fail.

@jaymecd
Copy link
Contributor

jaymecd commented Dec 14, 2017

@reznet tnx for feedback, confirmed.
This issue exists only when using embedded ingress {} or egress {} blocks of aws_security_group resource.

Separate declaration of the group and rules works as expected and is more preferred way to do it.

resource "aws_security_group" "test" {
  name        = "test-rules"
  description = "test"
}

resource "aws_security_group_rule" "test_rule1" {
  type        = "ingress"
  description = "RDP"
  from_port   = "3389"
  to_port     = "3389"
  protocol    = "tcp"
  cidr_blocks = ["10.0.0.0/8"]

  security_group_id = "${aws_security_group.test.id}"
}

resource "aws_security_group_rule" "test_rule2" {
  type        = "ingress"
  description = "RDP from Workspaces"
  from_port   = "3389"
  to_port     = "3389"
  protocol    = "tcp"
  cidr_blocks = ["170.0.0.0/8"]

  security_group_id = "${aws_security_group.test.id}"
}

Could we mention embedded blocks in issue title? to make it more concrete?

@radeksimko wdyt?

@takeda-joao takeda-joao changed the title Get error when updating SG ingress rules with description Get error when updating SG embedded ingress rules with description Dec 18, 2017
@karimodm
Copy link

Same issue here.

@MooreDerek
Copy link

MooreDerek commented Jan 11, 2018

Seeing this too.

tf v.0.11.2
aws.provider 1.2.0 but it's fixed with aws-provider 1.6.0

@jaymecd
Copy link
Contributor

jaymecd commented Jan 11, 2018

Source of issue has been found - it's data struct returned by resourceAwsSecurityGroupIPPermGather()

Consider following TF resource:

resource "aws_security_group" "test" {
  name  = "test"

  ingress {
      description = "RDP 11111"
      from_port = 3389
      to_port = 3389
      protocol = "tcp"
      cidr_blocks = ["1.1.1.1/32"]
  }

  ingress {
      description = "RDP 22222"
      from_port = 3389
      to_port = 3389
      protocol = "tcp"
      cidr_blocks = ["2.2.2.2/32"]
  }
}
Local / Remote structs comparison (collapsed)

LOCAL:

[]interface {}{
    map[string]interface {}{
        "protocol":"tcp", 
        "ipv6_cidr_blocks":[]interface {}{}, 
        "self":false, 
        "description":"RDP 11111", 
        "from_port":3389, 
        "cidr_blocks":[]interface {}{"1.1.1.1/32"}, 
        "security_groups":*Set(map[string]interface {}(nil)), 
        "to_port":3389
    }, 
    map[string]interface {}{
        "self":false, 
        "description":"RDP 22222", 
        "to_port":3389, 
        "protocol":"tcp", 
        "security_groups":*Set(map[string]interface {}(nil)), 
        "from_port":3389, 
        "cidr_blocks":[]interface {}{"2.2.2.2/32"}, 
        "ipv6_cidr_blocks":[]interface {}{}
    }
}

REMOTE:

[]map[string]interface {}{
    map[string]interface {}{
        "description":"RDP 22222", 
        "from_port":3389, 
        "to_port":3389, 
        "protocol":"tcp", 
        "cidr_blocks":[]string{"1.1.1.1/32", "2.2.2.2/32"}
    }
}

So remote struct has only 1 item with detached description from cidr_blocks, that's why it could not be properly mapped to the local struct.

I'm going to work on the fix in the next days.

@kenske
Copy link

kenske commented Jan 12, 2018

What's the best workaround for this in the meantime?

@jaymecd
Copy link
Contributor

jaymecd commented Jan 13, 2018

@kenske, use dedicated aws_security_group_rule resource

@jaymecd
Copy link
Contributor

jaymecd commented Jan 16, 2018

It's a little bit more complex that I thought before. @radeksimko @Ninir need ur advice.

Data structure of cidr_blocks, ipv6_cidr_blocks, prefix_list_ids & security_groups blocks from resource_aws_security_group.go should be upgraded from []string to []map[string]string to keep description together with value of the block or to slice of new internal type (I prefer this one):

type sgItem struct {
    Value string
    Description string
}

wdyt? Tnx

@radeksimko radeksimko added the service/ec2 Issues and PRs that pertain to the ec2 service. label Jan 17, 2018
@red97gst
Copy link

any updates?

@ScottSorrentino
Copy link

@kenske I hit this same problem recently and worked around it by simply removing the descriptions from the similar port/protocol inline rules. After working around it, I found this open issue. I didn't want to go with the suggestion from @jaymecd since we use Terraform to ensure the SG rules exactly match what's in the definition. I don't think that's possible with aws_security_group_rule or, if it is, I haven't been able to find a way to have it clear out rules someone manually added via the AWS console/API.

I did need to enable debugging/tracing to figure out exactly what was being called (and failing) via RevokeSecurityGroupIngress, then I manually set the description on those rules in AWS so the next time terraform apply tried to remove them it would succeed. After that, and with no competing description fields, every terraform plan from that point forward was clean.

@kenske
Copy link

kenske commented Feb 2, 2018

@ScottSorrentino It was kinda painful, but I did follow @jaymecd 's suggestion and it worked. It even worked for the scenario you are suggesting, clearing rules that were manually added.

@sysadmin1139
Copy link

Since the business-case for inline rules seems to be missing, here it is in detail.

Inline-style declarations are a way of saying only these rules, where separate resource style says at least these rules. From a compliance point of view, only these rules is a policy enforcement mechanism, where at least these rules merely ensures approved functionality is present. This has very big impacts when it comes time for an audit.

When the auditors ask us:

Please describe how your firewall rules are enforced through change-control.

The use of inline style allows us to avoid having to build a complicated system of CloudTrail event ingestion and correlation with our change-request system to show that only approved changes are implemented and that change-drift is both detected and alarmed, if not outright prevented. Instead, we can give them the commit-log for our terraform repo. The auditors understand git-repos and merge-request approval flows. They have said, in essence:

Oh! Like Puppet, but for security-group definitions!

And that makes the whole process go so much easier. This is one of the biggest value-adds Terraform gives us in our infrastructure.

@piersf
Copy link

piersf commented Mar 27, 2018

I just hit this issue today as well.
However, in my case where I have a rule like the below

resource "aws_security_group_rule" "DEFAULT_DO_NOT_USE-1" {
  type            = "ingress"
  from_port       = 0
  to_port         = 0
  protocol        = "-1"
  self            = "true"
  description = "Allow all incoming traffic from withing this security group"
  security_group_id = "${aws_security_group.DEFAULT_DO_NOT_USE.id}"
}

I can't use aws_security_group_rule as suggested above by @jaymecd as I will end up hitting a different bug explained in #1920

Any other solution?

@piersf
Copy link

piersf commented Mar 28, 2018

Hi again,
After a lot of painful playing with this, I've concluded that the issue is when using the description field for either an ingress or egress rule and applying again after the first time. Below I write the steps that got me to this conclusion.

I have the below resource declaration (note description is commented out except in first ingress):

resource "aws_security_group" "bastion_priv_SG-master" {
  name        = "bastion_priv_SG-master"
  description = "Security group for bastion host"
  vpc_id = "${aws_vpc.us_n_virg_VPC-master-10_100_0_0-16.id}"
  
  # incoming
  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["xx.xx.xx.xx/32"]
    description = "Allow all incoming traffic from xx.xx.xx.xx/32"
  }
  
  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    self        = "true"
    #description = "Allow all incoming traffic from within this SG"
  }
  
  ingress {
    from_port   = 9100
    to_port     = 9100
    protocol    = "tcp"
    security_groups = ["${data.aws_security_group.app_priv_SG_QAMonitor.id}"]
    #description = "Allow all incoming traffic from app_priv_SG_QAMonitor SG"
  }

  # outgoing
  egress {
    from_port       = 0
    to_port         = 0
    protocol        = "-1"
    cidr_blocks     = ["0.0.0.0/0"]
    #description = "Allow all outgoing traffic from this SG to the Internet"
  }
  
  tags {
    Name = "bastion_priv_SG-master"
  }
}

Applying that the first time works fine and I see the expected changes in the security group. Strangely terraform deletes and recreates the first ingress rule. But it applies successfully. Below is the result of the apply action:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ aws_security_group.bastion_priv_SG-master
      ingress.1242592041.cidr_blocks.#:              "1" => "0"
      ingress.1242592041.cidr_blocks.0:              "xx.xx.xx.xx/32" => ""
      ingress.1242592041.description:                "" => ""
      ingress.1242592041.from_port:                  "0" => "0"
      ingress.1242592041.ipv6_cidr_blocks.#:         "0" => "0"
      ingress.1242592041.protocol:                   "-1" => ""
      ingress.1242592041.security_groups.#:          "0" => "0"
      ingress.1242592041.self:                       "false" => "false"
      ingress.1242592041.to_port:                    "0" => "0"
      ingress.1698735776.cidr_blocks.#:              "0" => "0"
      ingress.1698735776.description:                "" => ""
      ingress.1698735776.from_port:                  "9100" => "9100"
      ingress.1698735776.ipv6_cidr_blocks.#:         "0" => "0"
      ingress.1698735776.protocol:                   "tcp" => "tcp"
      ingress.1698735776.security_groups.#:          "1" => "1"
      ingress.1698735776.security_groups.3308286315: "sg-5477a429" => "sg-5477a429"
      ingress.1698735776.self:                       "false" => "false"
      ingress.1698735776.to_port:                    "9100" => "9100"
      ingress.502957524.cidr_blocks.#:               "0" => "1"
      ingress.502957524.cidr_blocks.0:               "" => "xx.xx.xx.xx/32"
      ingress.502957524.description:                 "" => "Allow all incoming traffic from xx.xx.xx.xx/32"
      ingress.502957524.from_port:                   "" => "0"
      ingress.502957524.ipv6_cidr_blocks.#:          "0" => "0"
      ingress.502957524.protocol:                    "" => "-1"
      ingress.502957524.security_groups.#:           "0" => "0"
      ingress.502957524.self:                        "" => "false"
      ingress.502957524.to_port:                     "" => "0"
      ingress.753360330.cidr_blocks.#:               "0" => "0"
      ingress.753360330.description:                 "" => ""
      ingress.753360330.from_port:                   "0" => "0"
      ingress.753360330.ipv6_cidr_blocks.#:          "0" => "0"
      ingress.753360330.protocol:                    "-1" => "-1"
      ingress.753360330.security_groups.#:           "0" => "0"
      ingress.753360330.self:                        "true" => "true"
      ingress.753360330.to_port:                     "0" => "0"


Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_security_group.bastion_priv_SG-master: Modifying... (ID: sg-4183373a)
  ingress.1242592041.cidr_blocks.#:              "1" => "0"
  ingress.1242592041.cidr_blocks.0:              "xx.xx.xx.xx/32" => ""
  ingress.1242592041.description:                "" => ""
  ingress.1242592041.from_port:                  "0" => "0"
  ingress.1242592041.ipv6_cidr_blocks.#:         "0" => "0"
  ingress.1242592041.protocol:                   "-1" => ""
  ingress.1242592041.security_groups.#:          "0" => "0"
  ingress.1242592041.self:                       "false" => "false"
  ingress.1242592041.to_port:                    "0" => "0"
  ingress.1698735776.cidr_blocks.#:              "0" => "0"
  ingress.1698735776.description:                "" => ""
  ingress.1698735776.from_port:                  "9100" => "9100"
  ingress.1698735776.ipv6_cidr_blocks.#:         "0" => "0"
  ingress.1698735776.protocol:                   "tcp" => "tcp"
  ingress.1698735776.security_groups.#:          "1" => "1"
  ingress.1698735776.security_groups.3308286315: "sg-5477a429" => "sg-5477a429"
  ingress.1698735776.self:                       "false" => "false"
  ingress.1698735776.to_port:                    "9100" => "9100"
  ingress.502957524.cidr_blocks.#:               "0" => "1"
  ingress.502957524.cidr_blocks.0:               "" => "xx.xx.xx.xx/32"
  ingress.502957524.description:                 "" => "Allow all incoming traffic from xx.xx.xx.xx/32"
  ingress.502957524.from_port:                   "" => "0"
  ingress.502957524.ipv6_cidr_blocks.#:          "0" => "0"
  ingress.502957524.protocol:                    "" => "-1"
  ingress.502957524.security_groups.#:           "0" => "0"
  ingress.502957524.self:                        "" => "false"
  ingress.502957524.to_port:                     "" => "0"
  ingress.753360330.cidr_blocks.#:               "0" => "0"
  ingress.753360330.description:                 "" => ""
  ingress.753360330.from_port:                   "0" => "0"
  ingress.753360330.ipv6_cidr_blocks.#:          "0" => "0"
  ingress.753360330.protocol:                    "-1" => "-1"
  ingress.753360330.security_groups.#:           "0" => "0"
  ingress.753360330.self:                        "true" => "true"
  ingress.753360330.to_port:                     "0" => "0"
aws_security_group.bastion_priv_SG-master: Modifications complete after 2s (ID: sg-4183373a)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Now, if I apply again without changing anything in the resources, it will fail:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ aws_security_group.bastion_priv_SG-master
      ingress.1698735776.cidr_blocks.#:              "0" => "0"
      ingress.1698735776.description:                "" => ""
      ingress.1698735776.from_port:                  "9100" => "9100"
      ingress.1698735776.ipv6_cidr_blocks.#:         "0" => "0"
      ingress.1698735776.protocol:                   "tcp" => "tcp"
      ingress.1698735776.security_groups.#:          "1" => "1"
      ingress.1698735776.security_groups.3308286315: "sg-5477a429" => "sg-5477a429"
      ingress.1698735776.self:                       "false" => "false"
      ingress.1698735776.to_port:                    "9100" => "9100"
      ingress.2396394866.cidr_blocks.#:              "0" => "0"
      ingress.2396394866.description:                "Allow all incoming traffic from xx.xx.xx.xx/32" => ""
      ingress.2396394866.from_port:                  "0" => "0"
      ingress.2396394866.ipv6_cidr_blocks.#:         "0" => "0"
      ingress.2396394866.protocol:                   "-1" => ""
      ingress.2396394866.security_groups.#:          "0" => "0"
      ingress.2396394866.self:                       "true" => "false"
      ingress.2396394866.to_port:                    "0" => "0"
      ingress.502957524.cidr_blocks.#:               "1" => "1"
      ingress.502957524.cidr_blocks.0:               "xx.xx.xx.xx/32" => "xx.xx.xx.xx/32"
      ingress.502957524.description:                 "Allow all incoming traffic from xx.xx.xx.xx/32" => "Allow all incoming traffic from xx.xx.xx.xx/32"
      ingress.502957524.from_port:                   "0" => "0"
      ingress.502957524.ipv6_cidr_blocks.#:          "0" => "0"
      ingress.502957524.protocol:                    "-1" => "-1"
      ingress.502957524.security_groups.#:           "0" => "0"
      ingress.502957524.self:                        "false" => "false"
      ingress.502957524.to_port:                     "0" => "0"
      ingress.753360330.cidr_blocks.#:               "0" => "0"
      ingress.753360330.description:                 "" => ""
      ingress.753360330.from_port:                   "" => "0"
      ingress.753360330.ipv6_cidr_blocks.#:          "0" => "0"
      ingress.753360330.protocol:                    "" => "-1"
      ingress.753360330.security_groups.#:           "0" => "0"
      ingress.753360330.self:                        "" => "true"
      ingress.753360330.to_port:                     "" => "0"

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_security_group.bastion_priv_SG-master: Modifying... (ID: sg-4183373a)
  ingress.1698735776.cidr_blocks.#:              "0" => "0"
  ingress.1698735776.description:                "" => ""
  ingress.1698735776.from_port:                  "9100" => "9100"
  ingress.1698735776.ipv6_cidr_blocks.#:         "0" => "0"
  ingress.1698735776.protocol:                   "tcp" => "tcp"
  ingress.1698735776.security_groups.#:          "1" => "1"
  ingress.1698735776.security_groups.3308286315: "sg-5477a429" => "sg-5477a429"
  ingress.1698735776.self:                       "false" => "false"
  ingress.1698735776.to_port:                    "9100" => "9100"
  ingress.2396394866.cidr_blocks.#:              "0" => "0"
  ingress.2396394866.description:                "Allow all incoming traffic from xx.xx.xx.xx/32" => ""
  ingress.2396394866.from_port:                  "0" => "0"
  ingress.2396394866.ipv6_cidr_blocks.#:         "0" => "0"
  ingress.2396394866.protocol:                   "-1" => ""
  ingress.2396394866.security_groups.#:          "0" => "0"
  ingress.2396394866.self:                       "true" => "false"
  ingress.2396394866.to_port:                    "0" => "0"
  ingress.502957524.cidr_blocks.#:               "1" => "1"
  ingress.502957524.cidr_blocks.0:               "xx.xx.xx.xx/32" => "xx.xx.xx.xx/32"
  ingress.502957524.description:                 "Allow all incoming traffic from xx.xx.xx.xx/32" => "Allow all incoming traffic from xx.xx.xx.xx/32"
  ingress.502957524.from_port:                   "0" => "0"
  ingress.502957524.ipv6_cidr_blocks.#:          "0" => "0"
  ingress.502957524.protocol:                    "-1" => "-1"
  ingress.502957524.security_groups.#:           "0" => "0"
  ingress.502957524.self:                        "false" => "false"
  ingress.502957524.to_port:                     "0" => "0"
  ingress.753360330.cidr_blocks.#:               "0" => "0"
  ingress.753360330.description:                 "" => ""
  ingress.753360330.from_port:                   "" => "0"
  ingress.753360330.ipv6_cidr_blocks.#:          "0" => "0"
  ingress.753360330.protocol:                    "" => "-1"
  ingress.753360330.security_groups.#:           "0" => "0"
  ingress.753360330.self:                        "" => "true"
  ingress.753360330.to_port:                     "" => "0"

Error: Error applying plan:

1 error(s) occurred:

* aws_security_group.bastion_priv_SG-master: 1 error(s) occurred:

* aws_security_group.bastion_priv_SG-master: Error revoking security group ingress rules: InvalidPermission.NotFound: The specified rule does not exist in th
        status code: 400, request id: 98d8fa95-0aa2-40b8-851d-0a10bb25572b

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Where is the issue..?
In the result of the second apply action above, note the ingress rule with number 2396394866
Terraform, somehow and from somewhere sees an ingress rule that tries to delete and which doesn't exist anywhere in my state file or my security group in AWS. And that's why it complains that the security group ingress rule does not exist.
Why and how is Terraform seeing something that doesn't exist in the .tf or the state file is the question.

And like that isn't enough, I can't get out of this situation even if I comment out the description from the ingress rule. It will still fail with the same error. And I suspect that's because the state file now is in an inconsistent state since it doesn't rollback.

I have to manually delete the security group resource from the state file, do a refresh, import it back and then create one by one the ingress and egress rules without the description.

Can't really say where the bug is here, but I'd suggest to anyone else not providing a description for nested ingress and egress rules if you don't want to get in this situation.

I am running Terraform v0.11.5

Thanks,

@shanemeyers
Copy link

shanemeyers commented Mar 30, 2018

Got bit by this tonight. Even if I commented out the description it still triggered the bug. Only after completely deleting the line did things start to work as expected.

Terraform v0.11.5
+ provider.aws v1.13.0

@rajbettaswamy
Copy link

Bug reintroduced again in

Terraform v0.11.5
Provider "aws" (1.13.0)

@ibrahima
Copy link

ibrahima commented Apr 11, 2018

Was the bug actually ever fixed? I'm still seeing it too, although I can't figure out how to update my AWS provider (I'm on 1.6 and running terraform init does nothing to change that).

@piersf
Copy link

piersf commented Apr 12, 2018

@ibrahima still haven't seen any updates about the bug.
As for the provide version, try deleting the file similar to .terraform/plugins/linux_amd64/terraform-provider-aws_v1.14.1_x4 which is what I have and then run init again. This will update the version of the provider.

@ibrahima
Copy link

Ah okay, I actually figured that out after posting the comment (kinda counterintuitive, because the terraform init message says that it installed the latest version, but oh well). And now I actually am experiencing more bugs rather than fewer :(

@junkangli
Copy link

To update the provider, you run terraform init -upgrade.

@russellsherman
Copy link
Contributor

Experiencing this as well.

Terraform v0.11.5
+ provider.archive v1.0.3
+ provider.aws v1.11.0

@pmacdougall
Copy link

Just wanted to note that this also happens for the aws_default_security_group resource, where using individual aws_security_group_rule resources is not an option

@reza
Copy link

reza commented Apr 26, 2018

I face a similar issue when updating security group rules, it has nothing to do with description, it seems that the tf state file becomes out of sync with with the security group rules at apply time, it first removes/modifies the security rule then tries to add new properties.

$ terraform -version
Terraform v0.11.7

  • provider.aws v1.16.0
  • provider.null v1.0.0
  • provider.random v1.2.0
  • provider.template v1.0.0

@Caspain
Copy link

Caspain commented May 8, 2018

This is still present, i had manually delete all descriptions in aws and rerun the terraform script.

@carlosescura
Copy link

@Caspain I also had the same issue today. Deleting descriptions and applying again worked fine

@bflad bflad added this to the v1.19.0 milestone May 10, 2018
@bflad
Copy link
Contributor

bflad commented May 10, 2018

Hi folks 👋 Sorry this has been a longstanding issue with the AWS provider. The fix for this should be contained in #4416 which will be released with v1.19.0 of the AWS provider, likely middle of next week.

Shout outs to @loivis (and @svanharmelen who submitted an earlier, likely correct PR, which I admittedly should have reviewed and merged sooner: #3628)

Given there were so many various issues surrounding this bug, I will be locking this issue (amongst all the others) to encourage any lingering issues/discussion to be fully described in new issue(s) for consolidation. Thanks for your understanding.

@hashicorp hashicorp locked as resolved and limited conversation to collaborators May 10, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. service/ec2 Issues and PRs that pertain to the ec2 service.
Projects
None yet
Development

Successfully merging a pull request may close this issue.