New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(AWS Stream): Add support for custom checkpoint #9056
Conversation
Feat/ddb custom checkpointing
Codecov Report
@@ Coverage Diff @@
## master #9056 +/- ##
==========================================
+ Coverage 87.04% 87.17% +0.12%
==========================================
Files 287 288 +1
Lines 10986 11068 +82
==========================================
+ Hits 9563 9648 +85
+ Misses 1423 1420 -3
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @imewish - awesome job 🙌 I have a few small comments, please let me know what do you think
@@ -572,6 +578,40 @@ describe('AwsCompileStreamEvents', () => { | |||
awsCompileStreamEvents.serverless.service.functions.first.events[4].stream.destinations | |||
.onFailure | |||
); | |||
|
|||
// event 6 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All new functionality should be tested using runServerless
approach - we're currently in the process of migrating away from these old tests as they're fragile and makes refactorings harder to do. Could you please instead of expanding these, introduce a new test in AwsCompileStreamEvents #2
describe in this module?
For details about approach to testing, please check out: https://github.com/serverless/serverless/blob/master/test/README.md
docs/providers/aws/events/streams.md
Outdated
events: | ||
- stream: | ||
arn: arn:aws:dynamodb:region:XXXXXX:table/foo/stream/1970-01-01T00:00:00.000 | ||
functionResponseTypes: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know that underneath it's FunctionResponseTypes
, but I think for the Framework for now we should go with simple functionResponseType: ReportBatchItemFailures
setting. I know that AWS might expand that in the future, but I think it's very likely that always at most one functionResponseType
will be accepted - currently the FunctionResponseTypes
is limited to always have at most 1 item.
If AWS ever introduces the option to set multiple types, we will migrate to functionResponseTypes
and in the meantime, let's use a simpler approach - what do you think?
Btw - we already followed similar approaches for EFS integration for example.
@@ -32,6 +32,10 @@ class AwsCompileStreamEvents { | |||
maximumRecordAgeInSeconds: { | |||
anyOf: [{ const: -1 }, { type: 'integer', minimum: 60, maximum: 604800 }], | |||
}, | |||
functionResponseTypes: { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's add that property to https://github.com/serverless/serverless/blob/master/docs/providers/aws/guide/serverless.yml.md as well
Thanks for the feedback @pgrzesik. Let me work on it and will update them soon |
@pgrzesik pushed some changes as per your comments. Let me know if I'm missing anything. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks @imewish 🙇 I have a few minor comments, please let me know what do you think. Additionally, it would be good to extend the serverless.yml.md
file with that property: https://github.com/serverless/serverless/blob/master/docs/providers/aws/guide/serverless.yml.md
@@ -32,6 +32,10 @@ class AwsCompileStreamEvents { | |||
maximumRecordAgeInSeconds: { | |||
anyOf: [{ const: -1 }, { type: 'integer', minimum: 60, maximum: 604800 }], | |||
}, | |||
functionResponseType: { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's expect a string here instead of an array
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
noted, thought of it will be useful in future if aws add new item :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will definitely be useful, but I think we can adjust it at that point - I've made a small mistake with my suggestion to change the type to string - following other similar examples, I think we should go with such definition:
functionResponseType: { enum: ['ReportBatchItemFailures'] }
@@ -233,6 +237,10 @@ class AwsCompileStreamEvents { | |||
event.stream.bisectBatchOnFunctionError; | |||
} | |||
|
|||
if (event.stream.functionResponseType != null) { | |||
streamResource.Properties.FunctionResponseTypes = event.stream.functionResponseType; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After reconfiguring schema to expect a string, we will have to wrap it to be an array here I believe
@@ -375,6 +375,7 @@ describe('AwsCompileStreamEvents', () => { | |||
arn: 'arn:aws:dynamodb:region:account:table/bar/stream/2', | |||
batchWindow: 15, | |||
maximumRetryAttempts: 4, | |||
functionResponseType: ['ReportBatchItemFailures'], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As mentioned in my previous message, let's not extend these old tests, but instead extend or introduce a new one based on runServerless
utility.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry about that. Need to look at the docs and see how it works.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No worries - let me know if you have any questions around that 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pgrzesik Do we have a new folder to place the new tests? or is it under the same folder?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We already have a few of those in a separate describe in this file - please see:
describe('AwsCompileStreamEvents #2', () => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks :)
Feat/ddb custom checkpointing
Hi @pgrzesik I'm again :), Pushed the changes as per your feedback. https://github.com/serverless/serverless/pull/9056/files. Awaiting feedback. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well done 🙌 Just two very minor comments and we're ready to merge 👍
@@ -32,6 +32,10 @@ class AwsCompileStreamEvents { | |||
maximumRecordAgeInSeconds: { | |||
anyOf: [{ const: -1 }, { type: 'integer', minimum: 60, maximum: 604800 }], | |||
}, | |||
functionResponseType: { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will definitely be useful, but I think we can adjust it at that point - I've made a small mistake with my suggestion to change the type to string - following other similar examples, I think we should go with such definition:
functionResponseType: { enum: ['ReportBatchItemFailures'] }
@@ -233,6 +237,12 @@ class AwsCompileStreamEvents { | |||
event.stream.bisectBatchOnFunctionError; | |||
} | |||
|
|||
if (event.stream.functionResponseType != null) { | |||
streamResource.Properties.FunctionResponseTypes = []; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's maybe go with streamResource.Properties.FunctionResponseType = [event.stream.functionResponseType];
to be more succinct here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure thing. Will be pushing the changes soon
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pgrzesik Done :)
refactor as per review comments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks awesome, thank you @imewish 🙇
Thanks a lot @pgrzesik 🙌 |
Closes: #9030