Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AWS network ACL's and security groups keep loosing their rules on each alternating plan/apply. #222

Closed
hashibot opened this issue Jun 13, 2017 · 2 comments
Closed

AWS network ACL's and security groups keep loosing their rules on each alternating plan/apply. #222

hashibot opened this issue Jun 13, 2017 · 2 comments
Labels
bug Addresses a defect in current functionality. service/ec2 Issues and PRs that pertain to the ec2 service. stale Old or inactive issues managed by automation, if no further action taken these will get closed.

Comments

@hashibot
Copy link

This issue was originally opened by @bernielomax as hashicorp/terraform#8129. It was migrated here as part of the provider split. The original body of the issue is below.

Hi,

I keep experiencing an issue where terraform keeps modifying my AWS ACL and SG's on every run. It keeps alternating between applying the ACL/SG rule or removing them. Heres some of the plan out put below. On this particular example it will then add my rules for application_acl and will remove the rules for the database_acl... It just keeps alternating between applying rules and removing them on each run! This was also happening on a previous version for me. Looks like this could possible be caused by the route tables refreshing every run?

~ aws_network_acl.application_acl
    subnet_ids.#:          "0" => "3"
    subnet_ids.2063153051: "" => "subnet-b3cba9c5"
    subnet_ids.3899139489: "" => "subnet-30ae0068"
    subnet_ids.4142336080: "" => "subnet-4e24712a"

~ aws_network_acl.database_acl
    ingress.#:                     "7" => "1"
    ingress.1047571647.action:     "allow" => ""
    ingress.1047571647.cidr_block: "172.25.16.0/24" => ""
    ingress.1047571647.from_port:  "0" => "0"
    ingress.1047571647.icmp_code:  "0" => "0"
    ingress.1047571647.icmp_type:  "0" => "0"
    ingress.1047571647.protocol:   "-1" => ""
    ingress.1047571647.rule_no:    "100" => "0"
    ingress.1047571647.to_port:    "0" => "0"
    ingress.1736622700.action:     "allow" => "allow"
    ingress.1736622700.cidr_block: "0.0.0.0/0" => "0.0.0.0/0"
    ingress.1736622700.from_port:  "32768" => "32768"
    ingress.1736622700.icmp_code:  "0" => "0"
    ingress.1736622700.icmp_type:  "0" => "0"
    ingress.1736622700.protocol:   "6" => "tcp"
    ingress.1736622700.rule_no:    "500" => "500"
    ingress.1736622700.to_port:    "60999" => "60999"
    ingress.1786450.action:        "allow" => ""
    ingress.1786450.cidr_block:    "172.25.17.0/24" => ""
    ingress.1786450.from_port:     "22" => "0"
    ingress.1786450.icmp_code:     "0" => "0"
    ingress.1786450.icmp_type:     "0" => "0"
    ingress.1786450.protocol:      "6" => ""
    ingress.1786450.rule_no:       "21" => "0"
    ingress.1786450.to_port:       "22" => "0"
    ingress.2727494838.action:     "allow" => ""
    ingress.2727494838.cidr_block: "172.25.16.0/24" => ""
    ingress.2727494838.from_port:  "22" => "0"
    ingress.2727494838.icmp_code:  "0" => "0"
    ingress.2727494838.icmp_type:  "0" => "0"
    ingress.2727494838.protocol:   "6" => ""
    ingress.2727494838.rule_no:    "20" => "0"
    ingress.2727494838.to_port:    "22" => "0"
    ingress.3281948791.action:     "allow" => ""
    ingress.3281948791.cidr_block: "172.25.17.0/24" => ""
    ingress.3281948791.from_port:  "0" => "0"
    ingress.3281948791.icmp_code:  "0" => "0"
    ingress.3281948791.icmp_type:  "0" => "0"
    ingress.3281948791.protocol:   "-1" => ""
    ingress.3281948791.rule_no:    "101" => "0"
    ingress.3281948791.to_port:    "0" => "0"
    ingress.348602793.action:      "allow" => ""
    ingress.348602793.cidr_block:  "172.25.18.0/24" => ""
    ingress.348602793.from_port:   "0" => "0"
    ingress.348602793.icmp_code:   "0" => "0"
    ingress.348602793.icmp_type:   "0" => "0"
    ingress.348602793.protocol:    "-1" => ""
    ingress.348602793.rule_no:     "102" => "0"
    ingress.348602793.to_port:     "0" => "0"
    ingress.921427448.action:      "allow" => ""
    ingress.921427448.cidr_block:  "172.25.18.0/24" => ""
    ingress.921427448.from_port:   "22" => "0"
    ingress.921427448.icmp_code:   "0" => "0"
    ingress.921427448.icmp_type:   "0" => "0"
    ingress.921427448.protocol:    "6" => ""
    ingress.921427448.rule_no:     "22" => "0"
    ingress.921427448.to_port:     "22" => "0"

+ aws_network_acl_rule.allow_all_application_subnets.0
    cidr_block:     "172.25.8.0/23"
    egress:         "false"
    from_port:      "0"
    network_acl_id: "acl-9d3f06f9"
    protocol:       "-1"
    rule_action:    "allow"
    rule_number:    "100"
    to_port:        "0"

+ aws_network_acl_rule.allow_all_application_subnets.1
    cidr_block:     "172.25.10.0/23"
    egress:         "false"
    from_port:      "0"
    network_acl_id: "acl-9d3f06f9"
    protocol:       "-1"
    rule_action:    "allow"
    rule_number:    "101"
    to_port:        "0"

+ aws_network_acl_rule.allow_all_application_subnets.2
    cidr_block:     "172.25.12.0/23"
    egress:         "false"
    from_port:      "0"
    network_acl_id: "acl-9d3f06f9"
    protocol:       "-1"
    rule_action:    "allow"
    rule_number:    "102"
    to_port:        "0"

-/+ aws_network_acl_rule.allow_all_database_subnets.0
    cidr_block:     "172.25.16.0/24" => "172.25.16.0/24"
    egress:         "false" => "false"
    from_port:      "0" => "0"
    network_acl_id: "acl-983f06fc" => "acl-983f06fc"
    protocol:       "all" => "-1" (forces new resource)
    rule_action:    "allow" => "allow"
    rule_number:    "100" => "100"
    to_port:        "0" => "0"

-/+ aws_network_acl_rule.allow_all_database_subnets.1
    cidr_block:     "172.25.17.0/24" => "172.25.17.0/24"
    egress:         "false" => "false"
    from_port:      "0" => "0"
    network_acl_id: "acl-983f06fc" => "acl-983f06fc"
    protocol:       "all" => "-1" (forces new resource)
    rule_action:    "allow" => "allow"
    rule_number:    "101" => "101"
    to_port:        "0" => "0"

-/+ aws_network_acl_rule.allow_all_database_subnets.2
    cidr_block:     "172.25.18.0/24" => "172.25.18.0/24"
    egress:         "false" => "false"
    from_port:      "0" => "0"
    network_acl_id: "acl-983f06fc" => "acl-983f06fc"
    protocol:       "all" => "-1" (forces new resource)
    rule_action:    "allow" => "allow"
    rule_number:    "102" => "102"
    to_port:        "0" => "0"

+ aws_network_acl_rule.allow_dmz_ssh_to_application_subnets.0
    cidr_block:     "172.25.0.0/23"
    egress:         "false"
    from_port:      "22"
    network_acl_id: "acl-9d3f06f9"
    protocol:       "6"
    rule_action:    "allow"
    rule_number:    "20"
    to_port:        "22"

+ aws_network_acl_rule.allow_dmz_ssh_to_application_subnets.1
    cidr_block:     "172.25.2.0/23"
    egress:         "false"
    from_port:      "22"
    network_acl_id: "acl-9d3f06f9"
    protocol:       "6"
    rule_action:    "allow"
    rule_number:    "21"
    to_port:        "22"

+ aws_network_acl_rule.allow_dmz_ssh_to_application_subnets.2
    cidr_block:     "172.25.4.0/23"
    egress:         "false"
    from_port:      "22"
    network_acl_id: "acl-9d3f06f9"
    protocol:       "6"
    rule_action:    "allow"
    rule_number:    "22"
    to_port:        "22"

-/+ aws_network_acl_rule.allow_dmz_ssh_to_database_subnets.0
    cidr_block:     "172.25.16.0/24" => "172.25.16.0/24"
    egress:         "false" => "false"
    from_port:      "22" => "22"
    network_acl_id: "acl-983f06fc" => "acl-983f06fc"
    protocol:       "tcp" => "6" (forces new resource)
    rule_action:    "allow" => "allow"
    rule_number:    "20" => "20"
    to_port:        "22" => "22"

-/+ aws_network_acl_rule.allow_dmz_ssh_to_database_subnets.1
    cidr_block:     "172.25.17.0/24" => "172.25.17.0/24"
    egress:         "false" => "false"
    from_port:      "22" => "22"
    network_acl_id: "acl-983f06fc" => "acl-983f06fc"
    protocol:       "tcp" => "6" (forces new resource)
    rule_action:    "allow" => "allow"
    rule_number:    "21" => "21"
    to_port:        "22" => "22"

-/+ aws_network_acl_rule.allow_dmz_ssh_to_database_subnets.2
    cidr_block:     "172.25.18.0/24" => "172.25.18.0/24"
    egress:         "false" => "false"
    from_port:      "22" => "22"
    network_acl_id: "acl-983f06fc" => "acl-983f06fc"
    protocol:       "tcp" => "6" (forces new resource)
    rule_action:    "allow" => "allow"
    rule_number:    "22" => "22"
    to_port:        "22" => "22"

~ aws_route_table.private_route_table.0
    route.3165135897.cidr_block:                "" => "0.0.0.0/0"
    route.3165135897.gateway_id:                "" => "nat-0f398e86a54620af5"
    route.3165135897.instance_id:               "" => ""
    route.3165135897.nat_gateway_id:            "" => ""
    route.3165135897.network_interface_id:      "" => ""
    route.3165135897.vpc_peering_connection_id: "" => ""
    route.667727132.cidr_block:                 "0.0.0.0/0" => ""
    route.667727132.gateway_id:                 "" => ""
    route.667727132.instance_id:                "" => ""
    route.667727132.nat_gateway_id:             "nat-0f398e86a54620af5" => ""
    route.667727132.network_interface_id:       "" => ""
    route.667727132.vpc_peering_connection_id:  "" => ""

~ aws_route_table.private_route_table.1
    route.1925574287.cidr_block:                "0.0.0.0/0" => ""
    route.1925574287.gateway_id:                "" => ""
    route.1925574287.instance_id:               "" => ""
    route.1925574287.nat_gateway_id:            "nat-0db98bf955d2eceb8" => ""
    route.1925574287.network_interface_id:      "" => ""
    route.1925574287.vpc_peering_connection_id: "" => ""
    route.444407989.cidr_block:                 "" => "0.0.0.0/0"
    route.444407989.gateway_id:                 "" => "nat-0db98bf955d2eceb8"
    route.444407989.instance_id:                "" => ""
    route.444407989.nat_gateway_id:             "" => ""
    route.444407989.network_interface_id:       "" => ""
    route.444407989.vpc_peering_connection_id:  "" => ""

~ aws_route_table.private_route_table.2
    route.2589993565.cidr_block:                "0.0.0.0/0" => ""
    route.2589993565.gateway_id:                "" => ""
    route.2589993565.instance_id:               "" => ""
    route.2589993565.nat_gateway_id:            "nat-0a8044e6b5e90fe5d" => ""
    route.2589993565.network_interface_id:      "" => ""
    route.2589993565.vpc_peering_connection_id: "" => ""
    route.897214756.cidr_block:                 "" => "0.0.0.0/0"
    route.897214756.gateway_id:                 "" => "nat-0a8044e6b5e90fe5d"
    route.897214756.instance_id:                "" => ""
    route.897214756.nat_gateway_id:             "" => ""
    route.897214756.network_interface_id:       "" => ""
    route.897214756.vpc_peering_connection_id:  "" => ""

~ aws_security_group.application_sg
    ingress.#:                                    "3" => "1"
    ingress.2963200659.cidr_blocks.#:             "0" => "0"
    ingress.2963200659.from_port:                 "22" => "0"
    ingress.2963200659.protocol:                  "tcp" => ""
    ingress.2963200659.security_groups.#:         "1" => "0"
    ingress.2963200659.security_groups.446249355: "sg-26dfee40" => ""
    ingress.2963200659.self:                      "false" => "false"
    ingress.2963200659.to_port:                   "22" => "0"
    ingress.3575264873.cidr_blocks.#:             "1" => "1"
    ingress.3575264873.cidr_blocks.0:             "0.0.0.0/0" => "0.0.0.0/0"
    ingress.3575264873.from_port:                 "32768" => "32768"
    ingress.3575264873.protocol:                  "tcp" => "tcp"
    ingress.3575264873.security_groups.#:         "0" => "0"
    ingress.3575264873.self:                      "false" => "false"
    ingress.3575264873.to_port:                   "60999" => "60999"
    ingress.753360330.cidr_blocks.#:              "0" => "0"
    ingress.753360330.from_port:                  "0" => "0"
    ingress.753360330.protocol:                   "-1" => ""
    ingress.753360330.security_groups.#:          "0" => "0"
    ingress.753360330.self:                       "true" => "false"
    ingress.753360330.to_port:                    "0" => "0"

~ aws_security_group.database_sg
    ingress.#:                                    "3" => "1"
    ingress.2963200659.cidr_blocks.#:             "0" => "0"
    ingress.2963200659.from_port:                 "22" => "0"
    ingress.2963200659.protocol:                  "tcp" => ""
    ingress.2963200659.security_groups.#:         "1" => "0"
    ingress.2963200659.security_groups.446249355: "sg-26dfee40" => ""
    ingress.2963200659.self:                      "false" => "false"
    ingress.2963200659.to_port:                   "22" => "0"
    ingress.3575264873.cidr_blocks.#:             "1" => "1"
    ingress.3575264873.cidr_blocks.0:             "0.0.0.0/0" => "0.0.0.0/0"
    ingress.3575264873.from_port:                 "32768" => "32768"
    ingress.3575264873.protocol:                  "tcp" => "tcp"
    ingress.3575264873.security_groups.#:         "0" => "0"
    ingress.3575264873.self:                      "false" => "false"
    ingress.3575264873.to_port:                   "60999" => "60999"
    ingress.753360330.cidr_blocks.#:              "0" => "0"
    ingress.753360330.from_port:                  "0" => "0"
    ingress.753360330.protocol:                   "-1" => ""
    ingress.753360330.security_groups.#:          "0" => "0"
    ingress.753360330.self:                       "true" => "false"
    ingress.753360330.to_port:                    "0" => "0"

My template looks like the following for these items:

resource "aws_network_acl_rule" "allow_dmz_ssh_to_application_subnets" {
  count = "${length(split(",",lookup(var.availability_zone_map, var.region)))}"
  network_acl_id = "${aws_network_acl.application_acl.id}"
  cidr_block = "${element(split(",",var.dmz_subnets), count.index)}"
  rule_action = "allow"
  protocol = "6"
  from_port = "22"
  to_port = "22"
  rule_number = "${count.index + 20}"
}

resource "aws_network_acl_rule" "allow_all_application_subnets" {
  count = "${length(split(",",lookup(var.availability_zone_map, var.region)))}"
  network_acl_id = "${aws_network_acl.application_acl.id}"
  cidr_block = "${element(split(",",var.application_subnets), count.index)}"
  rule_action = "allow"
  protocol = "-1"
  from_port = "0"
  to_port = "0"
  rule_number = "${count.index + 100}"
}

resource "aws_network_acl" "database_acl" {
  vpc_id = "${aws_vpc.vpc.id}"
  subnet_ids = ["${aws_subnet.database_subnet.*.id}"]
  ingress {
    protocol   = "tcp"
    rule_no    = "500"
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = "32768"
    to_port    = "60999"
  }
  egress {
    protocol   = "-1"
    rule_no    = "100"
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = "0"
    to_port    = "0"
  }
  tags {
    Name = "${var.environment}-${var.application}-database-acl"
  }
}

resource "aws_network_acl_rule" "allow_dmz_ssh_to_database_subnets" {
  count = "${length(split(",",lookup(var.availability_zone_map, var.region)))}"
  network_acl_id = "${aws_network_acl.database_acl.id}"
  cidr_block = "${element(split(",",var.database_subnets), count.index)}"
  rule_action = "allow"
  protocol = "6"
  from_port = "22"
  to_port = "22"
  rule_number = "${count.index + 20}"
}

resource "aws_network_acl_rule" "allow_all_database_subnets" {
  count = "${length(split(",",lookup(var.availability_zone_map, var.region)))}"
  network_acl_id = "${aws_network_acl.database_acl.id}"
  cidr_block = "${element(split(",",var.database_subnets), count.index)}"
  rule_action = "allow"
  protocol = "-1"
  from_port = "0"
  to_port = "0"
  rule_number = "${count.index + 100}"
}

Terraform Version

Terraform v0.7.0

Affected Resource(s)

Please list the resources as a list, for example:

  • aws_network_acl
  • aws_network_acl_rule
  • aws_security_group
  • aws_security_group_rule
@hashibot hashibot added the bug Addresses a defect in current functionality. label Jun 13, 2017
@stefano-m stefano-m mentioned this issue Jul 18, 2017
Closed
@enm10k enm10k mentioned this issue Nov 28, 2017
Merged
@radeksimko radeksimko added the service/ec2 Issues and PRs that pertain to the ec2 service. label Jan 25, 2018
@github-actions
Copy link

github-actions bot commented Apr 5, 2020

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

@github-actions github-actions bot added the stale Old or inactive issues managed by automation, if no further action taken these will get closed. label Apr 5, 2020
@github-actions github-actions bot closed this as completed May 5, 2020
@ghost
Copy link

ghost commented Jun 5, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Jun 5, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Addresses a defect in current functionality. service/ec2 Issues and PRs that pertain to the ec2 service. stale Old or inactive issues managed by automation, if no further action taken these will get closed.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@radeksimko @hashibot