Multiple aws_sqs_queue_policy adds only one policy to queue

[cu12]

Terraform Version

v0.8.7

Affected Resource(s)

Please list the resources as a list, for example:

• aws_sqs_queue_policy
• aws_iam_policy_document

Terraform Configuration Files

variable "environments" {
  default = "prod,stag"
}

variable "vendorbackend_sns_topics" {
  type = "map"

  default = {
    region1_stag = "arn:aws:sns:eu-west-1:xxxxxxxxxxxx:snstopic_stag"
    region1_prod = "arn:aws:sns:eu-west-1:xxxxxxxxxxxx:snstopic_prod"
    region2_stag   = "arn:aws:sns:ap-southeast-1:xxxxxxxxxxxx:snstopic_stag"
    region2_prod   = "arn:aws:sns:ap-southeast-1::xxxxxxxxxxxx:snstopic_prod"
  }
}

resource "aws_sqs_queue" "q" {
  count = "${length(compact(split(",", var.environments)))}"

  name = "q-${element(split(",", var.environments), count.index)}"
}

# region 1
data "aws_iam_policy_document" "allow_sendmessage-region1" {
  count = "${length(compact(split(",", var.environments)))}"

  statement {
    actions = ["sqs:SendMessage"]

    principals {
      type        = "*"
      identifiers = ["*"]
    }

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [
        "${var.sns_topics["region1_${element(split(",", var.environments), count.index)}"]}",
      ]
    }
  }
}

resource "aws_sqs_queue_policy" "allow_sendmessage-region1" {
  count = "${length(compact(split(",", var.environments)))}"

  queue_url = "${element(aws_sqs_queue.q.*.id, count.index)}"
  policy    = "${element(data.aws_iam_policy_document.allow_sendmessage-region1.*.json, count.index)}"
}

# region 2
data "aws_iam_policy_document" "allow_sendmessage-region2" {
  count = "${length(compact(split(",", var.environments)))}"

  statement {
    actions = ["sqs:SendMessage"]

    principals {
      type        = "*"
      identifiers = ["*"]
    }

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [
        "${var.sns_topics["region2_${element(split(",", var.environments), count.index)}"]}",
      ]
    }
  }
}

resource "aws_sqs_queue_policy" "allow_sendmessage-region2" {
  count = "${length(compact(split(",", var.environments)))}"

  queue_url = "${element(aws_sqs_queue.q.*.id, count.index)}"
  policy    = "${element(data.aws_iam_policy_document.allow_sendmessage-region2.*.json, count.index)}"
}

Expected Behavior

Should have created two separate policies.

Actual Behavior

Created only one policy attached to the queue.
In each plan TF would like to modify the policies and during apply it applies them successfully.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform plan
2. terraform apply
3. terraform plan

Important Factoids

N/A

References

N/A

[ericwestfall]

This same behavior exists within the aws_sns_topic and aws_sns_topic_policy resources and I believe it is actually expected. The confusion comes from the distinction between policy and permission.

Policy vs. Permission

The SQS developer guide states the following in regards to SQS policies:

policy: The document that acts as a container for one or more statements.

That is to say, if you want to apply an SQS queue policy with multiple statements, they should reside within a single policy container. For example:

  "Version": "2012-10-17",
  "Id": "sqspolicy",
  "Statement": [
    {
      "Sid": "First",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:111045819866:test-queue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "${aws_sqs_queue.q.arn}"
        }
      }
    },
    {
      "Sid": "Second",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:ReceiveMessage",
      "Resource": "arn:aws:sqs:us-east-1:111045819866:test-queue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sqs:us-east-1:111045819866:test-queue"
        }
      }
    }
  ]
}

The distinction becomes even clearer when you examine the documentation for the AddPermission action:

Note
AddPermission writes an Amazon-SQS-generated policy. If you want to write your own policy, use SetQueueAttributes to upload your policy. For more information about writing your own policy, see Using The Access Policy Language in the Amazon SQS Developer Guide.

Terraform is calling the SetQueueAttributes action and uploading a policy not adding a permission.

Actual Terraform Behavior

A test like the one you provided above shows that Terraform actually overwrites the first policy with the second policy.

Here is a simplified SQS test that produces the behavior in question.

For each of the aws_sqs_queue_policy resources, Terraform is calling the SetQueueAttributes action against the SQS queue. In our test case, whichever policy was applied to the queue first gets overwritten with the second policy once Terraform processes the second aws_sqs_queue_policy resource.

The simplest way to prove this is by first creating an SQS queue with a single aws_sqs_queue_policy resource and running terraform apply.

resource "aws_sqs_queue" "q" {
  name = "test-queue-3"
}

resource "aws_sqs_queue_policy" "test" {
  queue_url = "${aws_sqs_queue.q.id}"
  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Id": "sqspolicy",
  "Statement": [
    {
      "Sid": "First",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "${aws_sqs_queue.q.arn}",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "${aws_sqs_queue.q.arn}"
        }
      }
    }
  ]
}
POLICY
}

The queue policy looks like we expect; note the policy Sid is First:

  "Version": "2012-10-17",
  "Id": "sqspolicy",
  "Statement": [
    {
      "Sid": "First",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:111045819866:test-queue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sqs:us-east-1:111045819866:test-queue"
        }
      }
    }
  ]
}

Now if we add a second aws_sqs_queue_policy resource:

  name = "test-queue"
}

resource "aws_sqs_queue_policy" "test" {
  queue_url = "${aws_sqs_queue.q.id}"
  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Id": "sqspolicy",
  "Statement": [
    {
      "Sid": "First",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "${aws_sqs_queue.q.arn}",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "${aws_sqs_queue.q.arn}"
        }
      }
    }
  ]
}
POLICY
}

resource "aws_sqs_queue_policy" "test2" {
  queue_url = "${aws_sqs_queue.q.id}"
  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Id": "sqspolicy",
  "Statement": [
    {
      "Sid": "Second",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:ReceiveMessage",
      "Resource": "${aws_sqs_queue.q.arn}",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "${aws_sqs_queue.q.arn}"
        }
      }
    }
  ]
}
POLICY
}

When we run an apply, Terraform overwrites the SQS queue policy with the second one. Here is the new SQS queue policy, note the policy Sid is now Second:

  "Version": "2012-10-17",
  "Id": "sqspolicy",
  "Statement": [
    {
      "Sid": "Second",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:ReceiveMessage",
      "Resource": "arn:aws:sqs:us-east-1:111045819866:test-queue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sqs:us-east-1:111045819866:test-queue"
        }
      }
    }
  ]
}

Possible Fixes

• A simple update to the documentation for the aws_sqs_queue_policy and aws_sns_topic_policy resources explaining that only one resource declaration should be present. Reference links can be provided that point to the appropriate AWS documentation on policy syntax.

• It might also be worth considering a new resource like aws_sqs_permission that wraps the AddPermission action for managing basic SQS queue permissions. Not sure if the maintainers think that would muddy the waters or not.

@grubernaut I'm happy to update the documentation if you agree with that approach?

[bosgood]

For anyone looking for a temporary workaround, you can use an inline policy with jsonencode interpolation to get multi-SNS topic permissions working in some cases. FWIW I think what made this work in my setup (compared to using a aws_iam_policy_document) was the queue policy's requirement for an "Id": "sqspolicy" field on the container, which I couldn't specify on aws_iam_policy_document.

Here's what we have:

resource "aws_sqs_queue_policy" "sns_to_sqs" {
  queue_url = "${var.queue_url}"

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Id": "sqspolicy",
  "Statement": [
    {
      "Sid": "First",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "${var.queue_arn}",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": ${jsonencode(var.topic_arns)}
        }
      }
    }
  ]
}
POLICY
}

[xpolb01]

@bosgood This worker well for me. Any fix for this so far?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.