@aws-cdk/aws-lambda-event-sources: Lambda event filtering #17874

fengkaijia · 2021-12-07T02:35:04Z

Description

Support the newly released event filtering feature for Amazon SQS, Amazon DynamoDB, and Amazon Kinesis.

https://aws.amazon.com/about-aws/whats-new/2021/11/aws-lambda-event-filtering-amazon-sqs-dynamodb-kinesis-sources/
https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html

Use Case

For filtering event source.

Proposed Solution

Add a filters to SqsEventSourceProps, KinesisEventSourceProps and DynamoEventSourceProps.

Other information

No response

Acknowledge

I may be able to implement this feature request
This feature might incur a breaking change

The text was updated successfully, but these errors were encountered:

dgoetsch · 2021-12-08T20:13:46Z

In addition to the classes up above, do we also need to add a field to EventSourceMapping?

Additionally, do we need an api designed for filter patterns? I don't really want to inline the json into an event source - I would much rather prefer to be able to express something like "give me all record creation events from on the ddb stream" natively in my chosen cdk laguage; as opposed to "apply this filter json to my ddb stream".

zehsor · 2021-12-10T20:26:34Z

As long as this issue is open:
CloudFormation natively supports this.
You can always write an escape hatch for that.

I wrote a blog post on how to do this for filtering lambda events.

@dgoetsch I think so.
I really like your idea. However i think it should only be additionally given the amount of flexibility you have with all the possible filter criteria. Do you think cdk should also validate the filter json?

kcliffe · 2022-01-31T21:36:27Z

@zehsor Can i ask why the Medium article seems to contain repeated (and non related) code samples? I thought i was because i was not a member - so i joined and no luck?

zehsor · 2022-02-01T11:54:58Z

@kcliffe I don't really understand what you mean by non related code samples. I just added all the different steps that i did in order to set up a complete cdk sample project. The filtering part takes places in the cdk stack (Building the stack is the headline).
If you have questions about my article i invite you to switch over to medium and comment on the article, since this issue is not about my medium article :)

kcliffe · 2022-02-01T21:18:02Z

@zehsor Sorry, i'm not trying to hijack this issue, it's just that the blog article in it's current form is completely useless as a reference to what you did since none of the code samples are visible? Here is a png of what i'm seeing FYI but i take your point - the issue is probably with Medium so i'll try to discuss it with them...

dgoetsch · 2022-02-03T21:04:10Z

Hey @zehsor ; I dropped the ball and never responded to you.

I think you're point above is fair: an expressive, fluent api is definitely preceded by a more literal integration.
I also think having strong compile time guarantees on the correctness of filters format would be very helpful for developer experience - this is one of the main motivations for using cdk imo.

In that case, I think the first step is probably just to implement syntax for Filters (i.e. a Filter object with specific fields vs some arbitrary json).

Dilski · 2022-05-02T14:55:22Z

While it may be nice, I'm not sure a fluent api or similar is really necessary for this feature to be implemented.

According to the lambda event filtering docs

For filter rules, Lambda supports the same set of syntax and rules as Amazon EventBridge. For more information, see Amazon EventBridge event patterns in the Amazon EventBridge User Guide.

There is an EventBridge event pattern interface in the CDK, typed to { [string]: any }, and without any special validation or api see cdk docs here.

I propose event filtering for lambda is implemented in the same way, and I am happy to contribute those changes?

owenmorgan · 2022-05-12T11:06:13Z

+1

kaizencc · 2022-05-26T14:51:59Z

@Dilski I am happy to talk through any proposed APIs here. My suggestion is to look at sns subscription filters as prior art for something similar: https://github.com/aws/aws-cdk/tree/master/packages/%40aws-cdk/aws-sns#filter-policy.

Regarding fluent APIs, I think as long as we allow users to drop down to basic json, we should be able to try and model a higher level API as well. If we don't want to start with everything at once, we can also try something like a LambdaFilter class with only a static LambdaFilter.json() class. That way, we can always add additional abstractions to the LambdaFilter class later.

kaizencc · 2022-05-26T21:35:39Z

@iRoachie i'm responding to your comment here so we consolidate things in one place. i'd be happy to take a community contribution on this and a potential PR should have an API similar to the sns subscription filter linked above. I think that API roughly follows what we're trying to achieve here. I'm not advocating for the eventbridge API typed that's typed { [string]: any } because that API was actually introduced 4!! years ago and I think we can improve on it.

So something like this:

fn.addEventSource(new sources.DynamoEventSource(table, {
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  batchSize: 5,
  filterPatterns: [{
    price: lambda.SourceFilter.numericFilter({
      between: { start: 100, stop: 200 },
    }),
  }, {
    size: lambda.SourceFilter.stringFilter({
       denylist: ['small', 'medium'], // becomes 'anything-but'
    }),
  }, {
    whatever: new lambda.SourceFilter(/* can put in your custom json here */),
  }],
})

I think we give @wtho another week to indicate that they intend to contribute this. After that, this feature is fair game :).

iRoachie · 2022-05-26T21:44:59Z

Alright sounds good. I'll check back in a week then. Happened to need this at work so that's why I'm interested 😄

wtho · 2022-05-28T07:39:24Z

@iRoachie feel free to take over. I won't have too much time over the next few weeks to tackle this.
If you want I can have a look at your PR as I did quite some research about it.

peterwoodworth · 2022-07-06T23:58:09Z

@iRoachie @wtho are either of you planning on picking this back up soon? If not maybe someone else can pick this up, this feature seems to be in pretty high demand!

Dilski · 2022-07-14T13:32:17Z

I can give this a go this week

Dilski · 2022-07-15T16:08:59Z

@kaizencc @wtho I've been playing around with this as the API. Would love to hear thoughts

const patternExample: FilterPattern = {
    "arbitrary": {
        "depth": {
            "of-keys": {
                "nullField": FilterExpression.isNull(),
                "emptyField": FilterExpression.isEmpty(),
                "equalsField": FilterExpression.equals("some value"),
                "equalsFieldNullSupported": FilterExpression.equals(null),
                "equalsAnyField": FilterExpression.equalsAnyOf("valuea", "valueb"),
                "orField": FilterExpression.or(
                    FilterExpression.equals("valuea"),
                    FilterExpression.equals("valueb")
                ),
                "notEqualsField": FilterExpression.not(
                    FilterExpression.equals("my value")
                ),
                "notEqualsAnyOfField": FilterExpression.not(
                    FilterExpression.equalsAnyOf("not valuea", "not valueb")
                ),
                "numericField": FilterExpression.numeric([{operator: NumericOperator.EQ, number: 4.20}]),
                "existsField": FilterExpression.exists(),
                "beginsWithField": FilterExpression.beginsWith("begins-"),
                "moreComplexOrField": FilterExpression.or(
                    FilterExpression.equals(170),
                    FilterExpression.numeric([{operator: NumericOperator.LT, number: 20}])
                ),

            }
        }
    }
}

Which outputs the filter expression:

{
  "arbitrary": {
    "depth": {
      "of-keys": {
        "nullField": [null],
        "emptyField": [""],
        "equalsField": ["some value"],
        "equalsFieldNullSupported": [null],
        "equalsAnyField": ["valuea", "valueb"],
        "orField": ["valuea", "valueb"],
        "notEqualsField": [{"anything-but": ["my value"]}],
        "notEqualsAnyOfField": [{"anything-but": ["not valuea", "not valueb"]}],
        "numericField": [{"numeric": ["=", 4.2]}],
        "existsField": [{"exists": true}],
        "beginsWithField": [{"prefix": "begins-"}],
        "moreComplexOrField": [ 170, {"numeric": ["<", 20]}]
      }
    }
  }
}

The filter pattern is typed like type FilterPattern = { [key: string]: FilterExpression | FilterPattern}. I think the combination of that typing with the and and or feels more fluid and flexible - but I'm not sure if it's too flexible?

adham-123 · 2022-08-10T08:37:37Z

Any updates?

Dilski · 2022-08-10T12:27:15Z

I'm was hoping to get some feedback on my above comment before doing more

wtho · 2022-08-16T10:33:05Z

@Dilski I think it's better to discuss it in a PR in detail, but as far as I remember the keys are not as flexible in jsii.

Try to construct these objects in C# or Java and you'll quickly realize the type system there is not so good in building these flexible definitions.

But I'm sure @kaizencc can give better feedback on how to work in the limitations jsii. Look at the other aws-cdk APIs to learn more and embrace "idiomatic jsii".

Dilski · 2022-09-05T12:14:42Z

I no longer have time to contribute this, but it looks like @marciocadev 's pull request introduces a similar API which looks good

… event sources (#21917) ## Description Adds the ability to create filters for SQS, DynamoDB and Kinesis, enabling filter criteria settings for event sources ## Use Cases With this PR will be possible, for example, to filter events from a DynamoDB Stream allowing only INSERT events to be transmitted as shown in the example below ```typescript const fn = new NodejsFunction(this, 'Fn'); const table = new dynamodb.Table(this, 'T', { partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING, }, stream: dynamodb.StreamViewType.NEW_IMAGE, }); fn.addEventSource(new sources.DynamoEventSource(table, { startingPosition: lambda.StartingPosition.LATEST, filters: [ lambda.FilterCriteria.filter({ eventName: FilterRule.isEqual('INSERT'), }), ], })); ``` Closes #17874 ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [x] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [x] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

github-actions · 2022-09-09T01:50:25Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

… event sources (aws#21917) ## Description Adds the ability to create filters for SQS, DynamoDB and Kinesis, enabling filter criteria settings for event sources ## Use Cases With this PR will be possible, for example, to filter events from a DynamoDB Stream allowing only INSERT events to be transmitted as shown in the example below ```typescript const fn = new NodejsFunction(this, 'Fn'); const table = new dynamodb.Table(this, 'T', { partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING, }, stream: dynamodb.StreamViewType.NEW_IMAGE, }); fn.addEventSource(new sources.DynamoEventSource(table, { startingPosition: lambda.StartingPosition.LATEST, filters: [ lambda.FilterCriteria.filter({ eventName: FilterRule.isEqual('INSERT'), }), ], })); ``` Closes aws#17874 ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [x] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [x] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

fengkaijia added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Dec 7, 2021

github-actions bot added the @aws-cdk/aws-lambda Related to AWS Lambda label Dec 7, 2021

github-actions bot assigned kaizencc Dec 7, 2021

kaizencc added effort/small Small work item – less than a day of effort p2 p1 and removed needs-triage This issue or PR still needs to be triaged. p2 labels Dec 7, 2021

kaizencc removed their assignment Dec 7, 2021

HenrikFricke mentioned this issue Dec 17, 2021

Clean up messaging lab superluminar-io/serverless-workshop#291

Closed

fwang mentioned this issue Dec 18, 2021

Table: support Lambda event filter sst/sst#1170

Closed

wtho mentioned this issue Jan 14, 2022

feat(lambda-event-sources): lambda event filter #18437

Closed

kaizencc added effort/medium Medium work item – several days of effort and removed effort/small Small work item – less than a day of effort labels Jan 26, 2022

douglasnaphas mentioned this issue Mar 20, 2022

Update the roster automatically when someone joins the user's current Seder douglasnaphas/madliberation#250

Closed

kaizencc mentioned this issue May 17, 2022

(aws-lambda-event-sources): FilterCriteria for SQS eventsources #20338

Closed

2 tasks

fwang mentioned this issue Jul 4, 2022

Add support for Lambda event filtering for Kinesis Stream sst/sst#1407

Open

marciocadev mentioned this issue Sep 5, 2022

feat(lambda-event-sources): add filters to SQS, DynamoDB, and Kinesis event sources #21917

Merged

4 tasks

mergify bot closed this as completed in #21917 Sep 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

@aws-cdk/aws-lambda-event-sources: Lambda event filtering #17874

@aws-cdk/aws-lambda-event-sources: Lambda event filtering #17874

fengkaijia commented Dec 7, 2021 •

edited

Loading

dgoetsch commented Dec 8, 2021

zehsor commented Dec 10, 2021

kcliffe commented Jan 31, 2022

zehsor commented Feb 1, 2022 •

edited

Loading

kcliffe commented Feb 1, 2022

dgoetsch commented Feb 3, 2022

Dilski commented May 2, 2022

owenmorgan commented May 12, 2022

kaizencc commented May 26, 2022

kaizencc commented May 26, 2022

iRoachie commented May 26, 2022

wtho commented May 28, 2022

peterwoodworth commented Jul 6, 2022 •

edited

Loading

Dilski commented Jul 14, 2022

Dilski commented Jul 15, 2022

adham-123 commented Aug 10, 2022

Dilski commented Aug 10, 2022

wtho commented Aug 16, 2022

Dilski commented Sep 5, 2022

github-actions bot commented Sep 9, 2022

@aws-cdk/aws-lambda-event-sources: Lambda event filtering #17874

@aws-cdk/aws-lambda-event-sources: Lambda event filtering #17874

Comments

fengkaijia commented Dec 7, 2021 • edited Loading

Description

Use Case

Proposed Solution

Other information

Acknowledge

dgoetsch commented Dec 8, 2021

zehsor commented Dec 10, 2021

kcliffe commented Jan 31, 2022

zehsor commented Feb 1, 2022 • edited Loading

kcliffe commented Feb 1, 2022

dgoetsch commented Feb 3, 2022

Dilski commented May 2, 2022

owenmorgan commented May 12, 2022

kaizencc commented May 26, 2022

kaizencc commented May 26, 2022

iRoachie commented May 26, 2022

wtho commented May 28, 2022

peterwoodworth commented Jul 6, 2022 • edited Loading

Dilski commented Jul 14, 2022

Dilski commented Jul 15, 2022

adham-123 commented Aug 10, 2022

Dilski commented Aug 10, 2022

wtho commented Aug 16, 2022

Dilski commented Sep 5, 2022

github-actions bot commented Sep 9, 2022

⚠️COMMENT VISIBILITY WARNING⚠️

fengkaijia commented Dec 7, 2021 •

edited

Loading

zehsor commented Feb 1, 2022 •

edited

Loading

peterwoodworth commented Jul 6, 2022 •

edited

Loading