aws-sns: When using filterPolicy, stack fails with: FilterPolicyScope: Invalid value [null]. Please use either MessageBody or MessageAttributes

[ksenia-exe]

Describe the bug

After updating cdk lib to 2.68.0, we are unable to deploy stacks that use SNS subscriptions with the filterPolicy parameter. Deployment errors out with

FilterPolicyScope: Invalid value [null]. Please use either MessageBody or MessageAttributes

Deployments were successful before resolving to the new version of the library. There is also no way to specify to use MessageAttributes in aws-cdk-lib/aws-sns-subscriptions or aws-cdk-lib/aws-sns lib.

Expected Behavior

The expected behavior is to successfully deploy the stack when using SqsSubscription with filterPolicy prop in v2.68.0 (or later) of aws-cdk-lib.

Current Behavior

The deployment errors out with:

FilterPolicyScope: Invalid value [null]. Please use either MessageBody or MessageAttributes (Service: AmazonSNS; Status Code: 400; Error Code: InvalidParameter; Request ID: 42450475-089d-58ca-a91a-58eec215835a; Proxy: null)

In the cdk.out template, it indeed doesn't have the scope param

"constructName": {
   "Type": "AWS::SNS::Subscription",
   "Properties": {
    "Protocol": "sqs",
    "TopicArn": "---",
    "Endpoint": {
     "Fn::GetAtt": [
      "----",
      "Arn"
     ]
    },
    "FilterPolicy": {
     "name": [
      "myName"
     ]
    },
    "Region": "us-west-2"
   },
   "Metadata": {
    "aws:cdk:path": "----"
   }
  }

Reproduction Steps

Here is how the constructs are set up. I can provide more details if needed.

   import { SqsSubscription } from "aws-cdk-lib/aws-sns-subscriptions";
   import { SubscriptionFilter } from "aws-cdk-lib/aws-sns";
   ....
 
    const queue = new Queue(this, "myQueue", { queueName: queueName });
    const topic = Topic.fromTopicArn(
      this,
      `myTopic`,
      TOPIC_ARN,
    );
    topic.addSubscription(
        new SqsSubscription(queue, {
          filterPolicy: { name: SubscriptionFilter.stringFilter({ allowlist: ["myName"] }) },
        }),
      );

We tried completely rebuilding everything from scratch and cleaning npm but the stack still fails.

Possible Solution

Replace undefined with "MessageAttributes" here.

Additional Information/Context

No response

CDK CLI Version

2.68.0

Framework Version

No response

Node.js Version

16

OS

macOS 12.6

Language

Typescript

Language Version

No response

Other information

No response

[pahud]

Hi

Unfortunately I can't reproduce your error and was able to deploy it successfully with cdk 2.70.0 with the code below

 const queue = new sqs.Queue(this, "myQueue", { queueName: 'myQueue' });
 const topic = new sns.Topic(this, 'Topic')
 const importedtopic = sns.Topic.fromTopicArn(
  this,
  `myTopic`,
  topic.topicArn,
);
importedtopic.addSubscription(
     new SqsSubscription(queue, {
       filterPolicy: { name: SubscriptionFilter.stringFilter({ allowlist: ["myName"] }) },
     }),
);

Can you check if the error still exists in 2.70.0 ?

[curtisshaffer]

We saw this error as well.

Steps:

1. Attempt to deploy a stack specifying a SqsSubscription with a filterPolicyWithMessageBody.
2. Have the stack fail creation and attempt to rollback (to a state without SqsSubscription with a filterPolicyWithMessageBody).

We saw the same with:

1. Successfully deploy a stack specifying a SqsSubscription with a filterPolicyWithMessageBody.
2. Then attempt to deploy the stack to remove the filterPolicyWithMessageBody from the SqsSubscription.

Stack had the above mentioned error and it seemed unresolvable without deleting the stack and recreating.
Invalid parameter: FilterPolicyScope: Invalid value [null]. Please use either MessageBody or MessageAttributes

We had no other FilterPolicyScope defined AFAIK.

We were on 2.67.0.

[curtisshaffer]

I've confirmed that the above flows are reproducible in 2.70.0 of aws-cdk.

[curtisshaffer]

@pahud FYI on above comments ^^^

[MrArnoldPalmer]

Hey @curtisshaffer, when you look at the template output in cdk.out once the stack update fails, are the subscription props correct? It seems like this may be an issue with cloudformation where the service is unable to make an update to the subscription resource that should be valid.

[curtisshaffer]

Apologies @MrArnoldPalmer , I'm bogged down and trying to get an example that is simple and doesn't have any of the other connected work I have to show you this.

[curtisshaffer]

@MrArnoldPalmer

It seems like this may be an issue with cloudformation where the service is unable to make an update

This was definitely the issue. Cloudtrail logs indicated that it was trying to:

1. Update FilterPolicy to {}
2. Update FilterPolicyScope to be null

The first update succeeds, but the latter update fails. Does not appear to be a cdk issue, but a cloudformation issue.

[MrArnoldPalmer]

@curtisshaffer thank you, I'll try to ask around and see what can be done about this.

[MrArnoldPalmer]

@curtisshaffer I confirmed that this is a service issue btw, I'm wondering for workaround purposes, if you do

new SqsSubscription(queue, {
  filterPolicy: {},
}),

If this works?

[gerarar]

@MrArnoldPalmer

@curtisshaffer I confirmed that this is a service issue btw, I'm wondering for workaround purposes, if you do

new SqsSubscription(queue, {
  filterPolicy: {},
}),

If this works?

Hi, we had just come across this issue. We were trying to convert a subscription from filtering on MessageBody to MessageAttributes (filterPolicyWithMessageBody to filterPolicy).

We have tried setting filterPolicy to just {}, but it still errors with "Invalid parameter: FilterPolicyScope: Invalid value [null]. Please use either MessageBody or MessageAttributes (Service: AmazonSNS; Status Code: 400; Error Code: InvalidParameter; ..." while updating the stack.

As a workaround, we just commented out the sns subscription resource (so it can get deleted) and then redeployed with the correct subscription and filter to recreate it. Avoids a stack deletion as suggested above.

[MrArnoldPalmer]

@gerarar yeah in most cases it seems like removing a subscription and replacing it should be fairly safe anyway. That likely is the best workaround for now.

[pabloizquierdoalegra]

Seems like this problem was fixed. I was having this issue and I was removing and re-adding the subscription as a workaround. But it appears to be fixed. Can anyone confirm?

[MrArnoldPalmer]

Yeah this has been fixed on the service side, closing but please do reopen if you see this again.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[akshatbharadwaj]

@MrArnoldPalmer
Was this ever fixed? I tried removing an SNS filter policy from an existing subscription today, and got the same error. I am on CDK version 2.81.

[cj-christoph-gysin]

I also encountered this recently (using aws-cdk 2.110.0)

[Himank17Gupta]

+1, encountered same issue while removing an sns filter, (using aws-cdk 2.130.0)

[KingsleyBawuah]

+1, encountered this issue when trying to add a filter to subscription that didn't have one. (using aws-cdk 2.125.0)

[barrett-schonefeld]

Encountered this today adding an SNS filter policy

[SimplyKnownAsG]

Also encountered this. Though, the description is not exactly what I encountered. I believe the issue is a two-step issue:

1. Create a subscription using filterPolicyWithMessageBody.
2. Modify the existing subscription to change to use filterPolicy.

The issue appears to be with CloudFormation (though could be fixed from CDK). The change results in a AWS::SNS::Subscription from having FilterPolicyScope: MessageAttributes to no longer having a FilterPolicyScope. Per CloudFormation, FilterPolicyScope does not need to be specified.

There is a corresponding CloudFormation issue here: aws-cloudformation/cloudformation-coverage-roadmap#1840