Recreate "Security Group Rule" each time I execute “terrafrom apply/plan” #11011

AdimUser · 2017-01-03T19:57:11Z

When I execute terraform apply or plan without doing any changes into any terraform scripts, terraform is going to add same security group rules again and again.

let me describe about my terraform scripts.I have designed my terraform script as modules, security group is a module and security group rule is also a module.

++++++++++++++++Main.tf++++++++++++++++++++++++++++++++++++++++++++++++++
module "application_sg" {
source = "modules/securitygroups"
security_group_name = "Application Security Group"
vpc_id = "${module.vpc.vpc_id}"
}

module "rule2"{
source = "modules/securitygroups/rules"
type = "ingress"
from_port = 8080
to_port = 8080
protocol = "tcp"
cidr_blocks = ["20.0.1.0/24"]
security_group_id = "${module.application_sg.security_group}"
}

module "rule3"{
source = "modules/securitygroups/rules"
type = "ingress"
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["20.0.1.0/24"]
security_group_id = "${module.application_sg.security_group}"
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Lets say I am executing this scripts again and again using terraform plan.
This is the output I am getting.See it tries to add rule which is already existing.

~ module.application_sg.aws_security_group.security_group
ingress.#: "3" => "1"
ingress.2358522502.cidr_blocks.#: "1" => "0"
ingress.2358522502.cidr_blocks.0: "20.0.1.0/24" => ""
ingress.2358522502.from_port: "443" => "0"
ingress.2358522502.protocol: "tcp" => ""
ingress.2358522502.security_groups.#: "0" => "0"
ingress.2358522502.self: "false" => "false"
ingress.2358522502.to_port: "443" => "0"
ingress.3250959853.cidr_blocks.#: "1" => "0"
ingress.3250959853.cidr_blocks.0: "20.0.1.0/24" => ""
ingress.3250959853.from_port: "8080" => "0"
ingress.3250959853.protocol: "tcp" => ""
ingress.3250959853.security_groups.#: "0" => "0"
ingress.3250959853.self: "false" => "false"
ingress.3250959853.to_port: "8080" => "0"
ingress.753360330.cidr_blocks.#: "0" => "0"
ingress.753360330.from_port: "0" => "0"
ingress.753360330.protocol: "-1" => "-1"
ingress.753360330.security_groups.#: "0" => "0"
ingress.753360330.self: "true" => "true"
ingress.753360330.to_port: "0" => "0"

module.rule1.aws_security_group_rule.rule
cidr_blocks.#: "1"
cidr_blocks.0: "20.0.1.0/24"
from_port: "80"
protocol: "tcp"
security_group_id: "sg-17c13770"
self: "false"
source_security_group_id: ""
to_port: "80"
type: "ingress"
==============================================================================

shaunofneuron · 2017-02-16T00:53:29Z

Seeing same behavior Terraform 0.8.5 & 0.8.7 where security group resource is defined in a module and references to security group from security group rule cause cyclical create (security group rule)/ change (security group) state. Might have to wait until this approach can persist being modularized.

My code is almost exact to what @AdimUser described.

grubernaut · 2017-02-16T02:24:29Z

Hi @AdimUser thanks for the issue!

Would you be able to clarify for me if you are specifying ingress/egress rules inside of an aws_security_group resource, as well as adding rules with an aws_security_group_rule resource that references the initial security group?

For example, the following psuedo-config will reproduce the error you're seeing:

resource "aws_security_group" "foo" {
  ingress {
    ....
  }
  egress {
    ...
  }
}

resource "aws_security_group_rule" "bar" {
  security_group_id = "${aws_security_group.foo.id}"
  type = "ingress"
  ...
}

This is a known issue, as the aws_security_group resource will attempt to manage all of the security group rules for that resource, while the aws_security_group_rule resource does not establish that same contract between security group rules.

Does this match what you have in your configuration, or does the source security group not include any inline ingress/egress rules? Thanks!

aglover-zendesk · 2017-02-28T15:34:20Z

Also seeing this behavior on Terraform v0.8.6

I have a module for creating an RDS cluster and two security groups that should be used by the RDS cluster. Each of my security groups are defined with an aws_security_group resource and a aws_security_group_rule rule, like this:

resource "aws_security_group" "main" {
  name        = "${var.name}--admin"
  description = "Allows traffic to rds"
  vpc_id      = "${var.vpc_id}"

  ingress {
    from_port       = "${var.port}"
    to_port         = "${var.port}"
    protocol        = "TCP"
    security_groups = ["${var.security_groups}"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = -1
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags {
    Name        = "RDS cluster (${var.name})"
    Environment = "${var.environment}"
  }
}

resource "aws_security_group_rule" "allow_app" {
  type              = "ingress"
  from_port         = 3306
  to_port           = 3306
  protocol          = "tcp"
  cidr_blocks       = ["${var.app_subnets}"]
  security_group_id = "${aws_security_group.main.id}"
}

When I run apply, it strips all of the ingress rules. The next time I run apply, it adds them all back. Unfortunately, since I have two security groups, they are staggered so TF is always stripping ingress rules from one SG and re-adding them to the other SG. In other words, I can't get to the desired state of having both SGs populated with their ingress rules. Example output (subnets replaced with 'dummy' values):

...
module.stack.rds.aws_security_group.main: Modifying...
  ingress.#:                                    "2" => "1"
  ingress.2000300587.cidr_blocks.#:             "0" => "0"
  ingress.2000300587.from_port:                 "3306" => "3306"
  ingress.2000300587.protocol:                  "TCP" => "tcp"
  ingress.2000300587.security_groups.#:         "1" => "1"
  ingress.2000300587.security_groups.571983794: "sg-abc12345 => "sg-abc12345
  ingress.2000300587.self:                      "false" => "false"
  ingress.2000300587.to_port:                   "3306" => "3306"
  ingress.554452638.cidr_blocks.#:              "21" => "0"
  ingress.554452638.cidr_blocks.0:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.1:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.10:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.11:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.12:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.13:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.14:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.15:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.16:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.17:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.18:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.19:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.2:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.20:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.3:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.4:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.5:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.6:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.7:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.8:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.9:              "10.0.0.0/8" => ""
  ingress.554452638.from_port:                  "3306" => "0"
  ingress.554452638.protocol:                   "tcp" => ""
  ingress.554452638.security_groups.#:          "0" => "0"
  ingress.554452638.self:                       "false" => "false"
  ingress.554452638.to_port:                    "3306" => "0"
module.stack.rds.aws_security_group_rule.allow_zendesk_app_and_admin: Creating...
  cidr_blocks.#:            "" => "38"
  cidr_blocks.0:            "" => "10.0.0.0/8"
  cidr_blocks.1:            "" => "10.0.0.0/8"
  cidr_blocks.10:           "" => "10.0.0.0/8"
  cidr_blocks.11:           "" => "10.0.0.0/8"
  cidr_blocks.12:           "" => "10.0.0.0/8"
  cidr_blocks.13:           "" => "10.0.0.0/8"
  cidr_blocks.14:           "" => "10.0.0.0/8"
  cidr_blocks.15:           "" => "10.0.0.0/8"
  cidr_blocks.16:           "" => "10.0.0.0/8"
  cidr_blocks.17:           "" => "10.0.0.0/8"
  cidr_blocks.18:           "" => "10.0.0.0/8"
  cidr_blocks.19:           "" => "10.0.0.0/8"
  cidr_blocks.2:            "" => "10.0.0.0/8"
  cidr_blocks.20:           "" => "10.0.0.0/8"
  cidr_blocks.21:           "" => "10.0.0.0/8"
  cidr_blocks.22:           "" => "10.0.0.0/8"
  cidr_blocks.23:           "" => "10.0.0.0/8"
  cidr_blocks.24:           "" => "10.0.0.0/8"
  cidr_blocks.25:           "" => "10.0.0.0/8"
  cidr_blocks.26:           "" => "10.0.0.0/8"
  cidr_blocks.27:           "" => "10.0.0.0/8"
  cidr_blocks.28:           "" => "10.0.0.0/8"
  cidr_blocks.29:           "" => "10.0.0.0/8"
  cidr_blocks.3:            "" => "10.0.0.0/8"
  cidr_blocks.30:           "" => "10.0.0.0/8"
  cidr_blocks.31:           "" => "10.0.0.0/8"
  cidr_blocks.32:           "" => "10.0.0.0/8"
  cidr_blocks.33:           "" => "10.0.0.0/8"
  cidr_blocks.34:           "" => "10.0.0.0/8"
  cidr_blocks.35:           "" => "10.0.0.0/8"
  cidr_blocks.36:           "" => "10.0.0.0/8"
  cidr_blocks.37:           "" => "10.0.0.0/8"
  cidr_blocks.4:            "" => "10.0.0.0/8"
  cidr_blocks.5:            "" => "10.0.0.0/8"
  cidr_blocks.6:            "" => "10.0.0.0/8"
  cidr_blocks.7:            "" => "10.0.0.0/8"
  cidr_blocks.8:            "" => "10.0.0.0/8"
  cidr_blocks.9:            "" => "10.0.0.0/8"
  from_port:                "" => "3306"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-abc12345
  self:                     "" => "false"
  source_security_group_id: "" => "<computed>"
  to_port:                  "" => "3306"
  type:                     "" => "ingress"
module.stack.rds.aws_security_group.main: Modifications complete
...

aglover-zendesk · 2017-02-28T15:37:08Z

Just re-read your comment @grubernaut, looks like I fall squarely into the known issue space. What's the suggested workaround/solution? Don't use aws_security_group_rule resources for the time being?

grubernaut · 2017-02-28T15:48:24Z

@aglover-zendesk This happens "by design". At some point in the future we may look into allowing all of the aws_security_group_rule resources to fully quantify all of the security group rules for a security group, but this will be an opt-in feature, and isn't on the roadmap yet. 😄

The main purpose of designing the aws_security_group_rule to not describe the "full state" of a security group's rules, is that in a distributed Terraform setup, where multiple repositories and multiple state files makeup a user's infrastructure, the loose contract of an aws_security_group_rule works really well. Defining the security group rules inline allows for the strict contract where: "these are a definitive list of all of the security group rules for this security group resource".

The best solution is to either define all of a security group's rules inline, or none of the security group's rules inline. It's when a user defines both that the mismatch occurs, as the inline rules are parsed as "definitive" and attempt to overwrite the individually defined security group rules.

Hopefully this answers your question, happy to discuss further though!

aglover-zendesk · 2017-02-28T15:58:45Z

Hey @grubernaut thanks, that does make sense. I see you already have a warning/notice at the top of the aws_security_group page in the docs too, so that's on me. Thanks for the quick answer.

grubernaut · 2017-04-13T14:33:32Z

Hello all,

Going to close this for now, more than happy to discuss further if the issue persists. Thanks!

…itions this change takes guidance from terraforms behavior explained here: hashicorp/terraform#11011 [#147690737]

cavaliercoder · 2017-07-15T12:01:43Z

Thanks for your notes on this. The design does make sense.

I feel the user experience could be improved with a meaningful error though. If an aws_security_group has inline rules, could an error be thrown if it is ever referenced by an aws_security_group_rule?

…itions this change takes guidance from terraforms behavior explained here: hashicorp/terraform#11011 [#147690737]

ghost · 2020-04-08T02:09:22Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

paddycarver added bug provider/aws labels Jan 3, 2017

mperriere mentioned this issue Mar 30, 2017

aws_security_group_rule creates then destroy additionnal SG port every 2 apply #12522

Closed

jangrewe mentioned this issue Apr 5, 2017

aws_security_group_rule from data source gets added to/removed from aws_security_group on alternating runs #13370

Closed

grubernaut closed this as completed Apr 13, 2017

grubernaut mentioned this issue May 1, 2017

Terraform apply not idempotent for security group rule : v0.9.4 #14124

Closed

nateww mentioned this issue May 4, 2017

Associating pre-existing Security Group Rules to resource.aws_security_group_rule #14207

Closed

hashibot mentioned this issue Jun 13, 2017

Associating pre-existing Security Group Rules to resource.aws_security_group_rule hashicorp/terraform-provider-aws#742

Closed

88981 pushed a commit to 88981/pcf-pipelines that referenced this issue Aug 2, 2017

fix aws security group re-apply issue to use only external rule defin…

abcd1f9

…itions this change takes guidance from terraforms behavior explained here: hashicorp/terraform#11011 [#147690737]

marchenko-devpro mentioned this issue Dec 28, 2017

Several connection strings in one App Service hashicorp/terraform-provider-azurerm#651

Closed

rabidscorpio mentioned this issue Aug 25, 2018

aws_security_group_rule loses track of certain out-of-band changes hashicorp/terraform-provider-aws#3234

Closed

patrykorwat mentioned this issue Dec 1, 2019

Removed AWS SG cycles GoogleCloudPlatform/terraformer#306

Merged

ghost locked and limited conversation to collaborators Apr 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recreate "Security Group Rule" each time I execute “terrafrom apply/plan” #11011

Recreate "Security Group Rule" each time I execute “terrafrom apply/plan” #11011

AdimUser commented Jan 3, 2017 •

edited

Loading

shaunofneuron commented Feb 16, 2017

grubernaut commented Feb 16, 2017

aglover-zendesk commented Feb 28, 2017

aglover-zendesk commented Feb 28, 2017

grubernaut commented Feb 28, 2017

aglover-zendesk commented Feb 28, 2017

grubernaut commented Apr 13, 2017

cavaliercoder commented Jul 15, 2017

ghost commented Apr 8, 2020

Recreate "Security Group Rule" each time I execute “terrafrom apply/plan” #11011

Recreate "Security Group Rule" each time I execute “terrafrom apply/plan” #11011

Comments

AdimUser commented Jan 3, 2017 • edited Loading

Lets say I am executing this scripts again and again using terraform plan. This is the output I am getting.See it tries to add rule which is already existing.

shaunofneuron commented Feb 16, 2017

grubernaut commented Feb 16, 2017

aglover-zendesk commented Feb 28, 2017

aglover-zendesk commented Feb 28, 2017

grubernaut commented Feb 28, 2017

aglover-zendesk commented Feb 28, 2017

grubernaut commented Apr 13, 2017

cavaliercoder commented Jul 15, 2017

ghost commented Apr 8, 2020

AdimUser commented Jan 3, 2017 •

edited

Loading

Lets say I am executing this scripts again and again using terraform plan.
This is the output I am getting.See it tries to add rule which is already existing.