New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recreate "Security Group Rule" each time I execute “terrafrom apply/plan” #11011

Closed
AdimUser opened this Issue Jan 3, 2017 · 8 comments

Comments

Projects
None yet
6 participants
@AdimUser

AdimUser commented Jan 3, 2017

When I execute terraform apply or plan without doing any changes into any terraform scripts, terraform is going to add same security group rules again and again.

let me describe about my terraform scripts.I have designed my terraform script as modules, security group is a module and security group rule is also a module.

++++++++++++++++Main.tf++++++++++++++++++++++++++++++++++++++++++++++++++
module "application_sg" {
source = "modules/securitygroups"
security_group_name = "Application Security Group"
vpc_id = "${module.vpc.vpc_id}"
}

module "rule2"{
source = "modules/securitygroups/rules"
type = "ingress"
from_port = 8080
to_port = 8080
protocol = "tcp"
cidr_blocks = ["20.0.1.0/24"]
security_group_id = "${module.application_sg.security_group}"
}

module "rule3"{
source = "modules/securitygroups/rules"
type = "ingress"
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["20.0.1.0/24"]
security_group_id = "${module.application_sg.security_group}"
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Lets say I am executing this scripts again and again using terraform plan.
This is the output I am getting.See it tries to add rule which is already existing.

~ module.application_sg.aws_security_group.security_group
ingress.#: "3" => "1"
ingress.2358522502.cidr_blocks.#: "1" => "0"
ingress.2358522502.cidr_blocks.0: "20.0.1.0/24" => ""
ingress.2358522502.from_port: "443" => "0"
ingress.2358522502.protocol: "tcp" => ""
ingress.2358522502.security_groups.#: "0" => "0"
ingress.2358522502.self: "false" => "false"
ingress.2358522502.to_port: "443" => "0"
ingress.3250959853.cidr_blocks.#: "1" => "0"
ingress.3250959853.cidr_blocks.0: "20.0.1.0/24" => ""
ingress.3250959853.from_port: "8080" => "0"
ingress.3250959853.protocol: "tcp" => ""
ingress.3250959853.security_groups.#: "0" => "0"
ingress.3250959853.self: "false" => "false"
ingress.3250959853.to_port: "8080" => "0"
ingress.753360330.cidr_blocks.#: "0" => "0"
ingress.753360330.from_port: "0" => "0"
ingress.753360330.protocol: "-1" => "-1"
ingress.753360330.security_groups.#: "0" => "0"
ingress.753360330.self: "true" => "true"
ingress.753360330.to_port: "0" => "0"

  • module.rule1.aws_security_group_rule.rule
    cidr_blocks.#: "1"
    cidr_blocks.0: "20.0.1.0/24"
    from_port: "80"
    protocol: "tcp"
    security_group_id: "sg-17c13770"
    self: "false"
    source_security_group_id: ""
    to_port: "80"
    type: "ingress"
    ==============================================================================
@shaunofneuron

This comment has been minimized.

Show comment
Hide comment
@shaunofneuron

shaunofneuron Feb 16, 2017

Seeing same behavior Terraform 0.8.5 & 0.8.7 where security group resource is defined in a module and references to security group from security group rule cause cyclical create (security group rule)/ change (security group) state. Might have to wait until this approach can persist being modularized.

My code is almost exact to what @AdimUser described.

shaunofneuron commented Feb 16, 2017

Seeing same behavior Terraform 0.8.5 & 0.8.7 where security group resource is defined in a module and references to security group from security group rule cause cyclical create (security group rule)/ change (security group) state. Might have to wait until this approach can persist being modularized.

My code is almost exact to what @AdimUser described.

@grubernaut

This comment has been minimized.

Show comment
Hide comment
@grubernaut

grubernaut Feb 16, 2017

Contributor

Hi @AdimUser thanks for the issue!

Would you be able to clarify for me if you are specifying ingress/egress rules inside of an aws_security_group resource, as well as adding rules with an aws_security_group_rule resource that references the initial security group?

For example, the following psuedo-config will reproduce the error you're seeing:

resource "aws_security_group" "foo" {
  ingress {
    ....
  }
  egress {
    ...
  }
}

resource "aws_security_group_rule" "bar" {
  security_group_id = "${aws_security_group.foo.id}"
  type = "ingress"
  ...
}

This is a known issue, as the aws_security_group resource will attempt to manage all of the security group rules for that resource, while the aws_security_group_rule resource does not establish that same contract between security group rules.

Does this match what you have in your configuration, or does the source security group not include any inline ingress/egress rules? Thanks!

Contributor

grubernaut commented Feb 16, 2017

Hi @AdimUser thanks for the issue!

Would you be able to clarify for me if you are specifying ingress/egress rules inside of an aws_security_group resource, as well as adding rules with an aws_security_group_rule resource that references the initial security group?

For example, the following psuedo-config will reproduce the error you're seeing:

resource "aws_security_group" "foo" {
  ingress {
    ....
  }
  egress {
    ...
  }
}

resource "aws_security_group_rule" "bar" {
  security_group_id = "${aws_security_group.foo.id}"
  type = "ingress"
  ...
}

This is a known issue, as the aws_security_group resource will attempt to manage all of the security group rules for that resource, while the aws_security_group_rule resource does not establish that same contract between security group rules.

Does this match what you have in your configuration, or does the source security group not include any inline ingress/egress rules? Thanks!

@aglover-zendesk

This comment has been minimized.

Show comment
Hide comment
@aglover-zendesk

aglover-zendesk Feb 28, 2017

Also seeing this behavior on Terraform v0.8.6

I have a module for creating an RDS cluster and two security groups that should be used by the RDS cluster. Each of my security groups are defined with an aws_security_group resource and a aws_security_group_rule rule, like this:

resource "aws_security_group" "main" {
  name        = "${var.name}--admin"
  description = "Allows traffic to rds"
  vpc_id      = "${var.vpc_id}"

  ingress {
    from_port       = "${var.port}"
    to_port         = "${var.port}"
    protocol        = "TCP"
    security_groups = ["${var.security_groups}"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = -1
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags {
    Name        = "RDS cluster (${var.name})"
    Environment = "${var.environment}"
  }
}

resource "aws_security_group_rule" "allow_app" {
  type              = "ingress"
  from_port         = 3306
  to_port           = 3306
  protocol          = "tcp"
  cidr_blocks       = ["${var.app_subnets}"]
  security_group_id = "${aws_security_group.main.id}"
}

When I run apply, it strips all of the ingress rules. The next time I run apply, it adds them all back. Unfortunately, since I have two security groups, they are staggered so TF is always stripping ingress rules from one SG and re-adding them to the other SG. In other words, I can't get to the desired state of having both SGs populated with their ingress rules. Example output (subnets replaced with 'dummy' values):

...
module.stack.rds.aws_security_group.main: Modifying...
  ingress.#:                                    "2" => "1"
  ingress.2000300587.cidr_blocks.#:             "0" => "0"
  ingress.2000300587.from_port:                 "3306" => "3306"
  ingress.2000300587.protocol:                  "TCP" => "tcp"
  ingress.2000300587.security_groups.#:         "1" => "1"
  ingress.2000300587.security_groups.571983794: "sg-abc12345 => "sg-abc12345
  ingress.2000300587.self:                      "false" => "false"
  ingress.2000300587.to_port:                   "3306" => "3306"
  ingress.554452638.cidr_blocks.#:              "21" => "0"
  ingress.554452638.cidr_blocks.0:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.1:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.10:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.11:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.12:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.13:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.14:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.15:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.16:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.17:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.18:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.19:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.2:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.20:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.3:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.4:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.5:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.6:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.7:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.8:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.9:              "10.0.0.0/8" => ""
  ingress.554452638.from_port:                  "3306" => "0"
  ingress.554452638.protocol:                   "tcp" => ""
  ingress.554452638.security_groups.#:          "0" => "0"
  ingress.554452638.self:                       "false" => "false"
  ingress.554452638.to_port:                    "3306" => "0"
module.stack.rds.aws_security_group_rule.allow_zendesk_app_and_admin: Creating...
  cidr_blocks.#:            "" => "38"
  cidr_blocks.0:            "" => "10.0.0.0/8"
  cidr_blocks.1:            "" => "10.0.0.0/8"
  cidr_blocks.10:           "" => "10.0.0.0/8"
  cidr_blocks.11:           "" => "10.0.0.0/8"
  cidr_blocks.12:           "" => "10.0.0.0/8"
  cidr_blocks.13:           "" => "10.0.0.0/8"
  cidr_blocks.14:           "" => "10.0.0.0/8"
  cidr_blocks.15:           "" => "10.0.0.0/8"
  cidr_blocks.16:           "" => "10.0.0.0/8"
  cidr_blocks.17:           "" => "10.0.0.0/8"
  cidr_blocks.18:           "" => "10.0.0.0/8"
  cidr_blocks.19:           "" => "10.0.0.0/8"
  cidr_blocks.2:            "" => "10.0.0.0/8"
  cidr_blocks.20:           "" => "10.0.0.0/8"
  cidr_blocks.21:           "" => "10.0.0.0/8"
  cidr_blocks.22:           "" => "10.0.0.0/8"
  cidr_blocks.23:           "" => "10.0.0.0/8"
  cidr_blocks.24:           "" => "10.0.0.0/8"
  cidr_blocks.25:           "" => "10.0.0.0/8"
  cidr_blocks.26:           "" => "10.0.0.0/8"
  cidr_blocks.27:           "" => "10.0.0.0/8"
  cidr_blocks.28:           "" => "10.0.0.0/8"
  cidr_blocks.29:           "" => "10.0.0.0/8"
  cidr_blocks.3:            "" => "10.0.0.0/8"
  cidr_blocks.30:           "" => "10.0.0.0/8"
  cidr_blocks.31:           "" => "10.0.0.0/8"
  cidr_blocks.32:           "" => "10.0.0.0/8"
  cidr_blocks.33:           "" => "10.0.0.0/8"
  cidr_blocks.34:           "" => "10.0.0.0/8"
  cidr_blocks.35:           "" => "10.0.0.0/8"
  cidr_blocks.36:           "" => "10.0.0.0/8"
  cidr_blocks.37:           "" => "10.0.0.0/8"
  cidr_blocks.4:            "" => "10.0.0.0/8"
  cidr_blocks.5:            "" => "10.0.0.0/8"
  cidr_blocks.6:            "" => "10.0.0.0/8"
  cidr_blocks.7:            "" => "10.0.0.0/8"
  cidr_blocks.8:            "" => "10.0.0.0/8"
  cidr_blocks.9:            "" => "10.0.0.0/8"
  from_port:                "" => "3306"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-abc12345
  self:                     "" => "false"
  source_security_group_id: "" => "<computed>"
  to_port:                  "" => "3306"
  type:                     "" => "ingress"
module.stack.rds.aws_security_group.main: Modifications complete
...

aglover-zendesk commented Feb 28, 2017

Also seeing this behavior on Terraform v0.8.6

I have a module for creating an RDS cluster and two security groups that should be used by the RDS cluster. Each of my security groups are defined with an aws_security_group resource and a aws_security_group_rule rule, like this:

resource "aws_security_group" "main" {
  name        = "${var.name}--admin"
  description = "Allows traffic to rds"
  vpc_id      = "${var.vpc_id}"

  ingress {
    from_port       = "${var.port}"
    to_port         = "${var.port}"
    protocol        = "TCP"
    security_groups = ["${var.security_groups}"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = -1
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags {
    Name        = "RDS cluster (${var.name})"
    Environment = "${var.environment}"
  }
}

resource "aws_security_group_rule" "allow_app" {
  type              = "ingress"
  from_port         = 3306
  to_port           = 3306
  protocol          = "tcp"
  cidr_blocks       = ["${var.app_subnets}"]
  security_group_id = "${aws_security_group.main.id}"
}

When I run apply, it strips all of the ingress rules. The next time I run apply, it adds them all back. Unfortunately, since I have two security groups, they are staggered so TF is always stripping ingress rules from one SG and re-adding them to the other SG. In other words, I can't get to the desired state of having both SGs populated with their ingress rules. Example output (subnets replaced with 'dummy' values):

...
module.stack.rds.aws_security_group.main: Modifying...
  ingress.#:                                    "2" => "1"
  ingress.2000300587.cidr_blocks.#:             "0" => "0"
  ingress.2000300587.from_port:                 "3306" => "3306"
  ingress.2000300587.protocol:                  "TCP" => "tcp"
  ingress.2000300587.security_groups.#:         "1" => "1"
  ingress.2000300587.security_groups.571983794: "sg-abc12345 => "sg-abc12345
  ingress.2000300587.self:                      "false" => "false"
  ingress.2000300587.to_port:                   "3306" => "3306"
  ingress.554452638.cidr_blocks.#:              "21" => "0"
  ingress.554452638.cidr_blocks.0:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.1:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.10:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.11:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.12:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.13:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.14:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.15:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.16:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.17:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.18:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.19:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.2:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.20:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.3:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.4:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.5:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.6:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.7:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.8:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.9:              "10.0.0.0/8" => ""
  ingress.554452638.from_port:                  "3306" => "0"
  ingress.554452638.protocol:                   "tcp" => ""
  ingress.554452638.security_groups.#:          "0" => "0"
  ingress.554452638.self:                       "false" => "false"
  ingress.554452638.to_port:                    "3306" => "0"
module.stack.rds.aws_security_group_rule.allow_zendesk_app_and_admin: Creating...
  cidr_blocks.#:            "" => "38"
  cidr_blocks.0:            "" => "10.0.0.0/8"
  cidr_blocks.1:            "" => "10.0.0.0/8"
  cidr_blocks.10:           "" => "10.0.0.0/8"
  cidr_blocks.11:           "" => "10.0.0.0/8"
  cidr_blocks.12:           "" => "10.0.0.0/8"
  cidr_blocks.13:           "" => "10.0.0.0/8"
  cidr_blocks.14:           "" => "10.0.0.0/8"
  cidr_blocks.15:           "" => "10.0.0.0/8"
  cidr_blocks.16:           "" => "10.0.0.0/8"
  cidr_blocks.17:           "" => "10.0.0.0/8"
  cidr_blocks.18:           "" => "10.0.0.0/8"
  cidr_blocks.19:           "" => "10.0.0.0/8"
  cidr_blocks.2:            "" => "10.0.0.0/8"
  cidr_blocks.20:           "" => "10.0.0.0/8"
  cidr_blocks.21:           "" => "10.0.0.0/8"
  cidr_blocks.22:           "" => "10.0.0.0/8"
  cidr_blocks.23:           "" => "10.0.0.0/8"
  cidr_blocks.24:           "" => "10.0.0.0/8"
  cidr_blocks.25:           "" => "10.0.0.0/8"
  cidr_blocks.26:           "" => "10.0.0.0/8"
  cidr_blocks.27:           "" => "10.0.0.0/8"
  cidr_blocks.28:           "" => "10.0.0.0/8"
  cidr_blocks.29:           "" => "10.0.0.0/8"
  cidr_blocks.3:            "" => "10.0.0.0/8"
  cidr_blocks.30:           "" => "10.0.0.0/8"
  cidr_blocks.31:           "" => "10.0.0.0/8"
  cidr_blocks.32:           "" => "10.0.0.0/8"
  cidr_blocks.33:           "" => "10.0.0.0/8"
  cidr_blocks.34:           "" => "10.0.0.0/8"
  cidr_blocks.35:           "" => "10.0.0.0/8"
  cidr_blocks.36:           "" => "10.0.0.0/8"
  cidr_blocks.37:           "" => "10.0.0.0/8"
  cidr_blocks.4:            "" => "10.0.0.0/8"
  cidr_blocks.5:            "" => "10.0.0.0/8"
  cidr_blocks.6:            "" => "10.0.0.0/8"
  cidr_blocks.7:            "" => "10.0.0.0/8"
  cidr_blocks.8:            "" => "10.0.0.0/8"
  cidr_blocks.9:            "" => "10.0.0.0/8"
  from_port:                "" => "3306"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-abc12345
  self:                     "" => "false"
  source_security_group_id: "" => "<computed>"
  to_port:                  "" => "3306"
  type:                     "" => "ingress"
module.stack.rds.aws_security_group.main: Modifications complete
...
@aglover-zendesk

This comment has been minimized.

Show comment
Hide comment
@aglover-zendesk

aglover-zendesk Feb 28, 2017

Just re-read your comment @grubernaut, looks like I fall squarely into the known issue space. What's the suggested workaround/solution? Don't use aws_security_group_rule resources for the time being?

aglover-zendesk commented Feb 28, 2017

Just re-read your comment @grubernaut, looks like I fall squarely into the known issue space. What's the suggested workaround/solution? Don't use aws_security_group_rule resources for the time being?

@grubernaut

This comment has been minimized.

Show comment
Hide comment
@grubernaut

grubernaut Feb 28, 2017

Contributor

@aglover-zendesk This happens "by design". At some point in the future we may look into allowing all of the aws_security_group_rule resources to fully quantify all of the security group rules for a security group, but this will be an opt-in feature, and isn't on the roadmap yet. 😄

The main purpose of designing the aws_security_group_rule to not describe the "full state" of a security group's rules, is that in a distributed Terraform setup, where multiple repositories and multiple state files makeup a user's infrastructure, the loose contract of an aws_security_group_rule works really well. Defining the security group rules inline allows for the strict contract where: "these are a definitive list of all of the security group rules for this security group resource".

The best solution is to either define all of a security group's rules inline, or none of the security group's rules inline. It's when a user defines both that the mismatch occurs, as the inline rules are parsed as "definitive" and attempt to overwrite the individually defined security group rules.

Hopefully this answers your question, happy to discuss further though!

Contributor

grubernaut commented Feb 28, 2017

@aglover-zendesk This happens "by design". At some point in the future we may look into allowing all of the aws_security_group_rule resources to fully quantify all of the security group rules for a security group, but this will be an opt-in feature, and isn't on the roadmap yet. 😄

The main purpose of designing the aws_security_group_rule to not describe the "full state" of a security group's rules, is that in a distributed Terraform setup, where multiple repositories and multiple state files makeup a user's infrastructure, the loose contract of an aws_security_group_rule works really well. Defining the security group rules inline allows for the strict contract where: "these are a definitive list of all of the security group rules for this security group resource".

The best solution is to either define all of a security group's rules inline, or none of the security group's rules inline. It's when a user defines both that the mismatch occurs, as the inline rules are parsed as "definitive" and attempt to overwrite the individually defined security group rules.

Hopefully this answers your question, happy to discuss further though!

@aglover-zendesk

This comment has been minimized.

Show comment
Hide comment
@aglover-zendesk

aglover-zendesk Feb 28, 2017

Hey @grubernaut thanks, that does make sense. I see you already have a warning/notice at the top of the aws_security_group page in the docs too, so that's on me. Thanks for the quick answer.

aglover-zendesk commented Feb 28, 2017

Hey @grubernaut thanks, that does make sense. I see you already have a warning/notice at the top of the aws_security_group page in the docs too, so that's on me. Thanks for the quick answer.

@grubernaut

This comment has been minimized.

Show comment
Hide comment
@grubernaut

grubernaut Apr 13, 2017

Contributor

Hello all,

Going to close this for now, more than happy to discuss further if the issue persists. Thanks!

Contributor

grubernaut commented Apr 13, 2017

Hello all,

Going to close this for now, more than happy to discuss further if the issue persists. Thanks!

@grubernaut grubernaut closed this Apr 13, 2017

xchapter7x added a commit to pivotal-cf/pcf-pipelines that referenced this issue Jun 22, 2017

fix aws security group re-apply issue to use only external rule defin…
…itions

this change takes guidance from terraforms behavior explained here: hashicorp/terraform#11011

[#147690737]

xchapter7x added a commit to pivotal-cf/pcf-pipelines that referenced this issue Jul 7, 2017

fix aws security group re-apply issue to use only external rule defin…
…itions

this change takes guidance from terraforms behavior explained here: hashicorp/terraform#11011

[#147690737]
@cavaliercoder

This comment has been minimized.

Show comment
Hide comment
@cavaliercoder

cavaliercoder Jul 15, 2017

Thanks for your notes on this. The design does make sense.

I feel the user experience could be improved with a meaningful error though. If an aws_security_group has inline rules, could an error be thrown if it is ever referenced by an aws_security_group_rule?

cavaliercoder commented Jul 15, 2017

Thanks for your notes on this. The design does make sense.

I feel the user experience could be improved with a meaningful error though. If an aws_security_group has inline rules, could an error be thrown if it is ever referenced by an aws_security_group_rule?

88981 pushed a commit to 88981/pcf-pipelines that referenced this issue Aug 2, 2017

fix aws security group re-apply issue to use only external rule defin…
…itions

this change takes guidance from terraforms behavior explained here: hashicorp/terraform#11011

[#147690737]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment