feat: sns to lambda stage

[malachi-constant]

Relates

• typescript conversion: SnsToLambda stage #250

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[malachi-constant]

Integration test passing.

[anmolsgandhi]

Continuing the conversation on redrive, i believe users can have multiple ways to redrive, most common i have seen is having a lambda to reprocess, that is also a pattern we have in SDLF. So essentially DLQ --> Lambda --> SNS event/Lambda logic --> SNS

[LeonLuttenberger]

Continuing the conversation on redrive, i believe users can have multiple ways to redrive, most common i have seen is having a lambda to reprocess, that is also a pattern we have in SDLF. So essentially DLQ --> Lambda --> SNS event/Lambda logic --> SNS

A Lambda could solve this, yes. The reason I think an SQS queue is still preferable is that it comes with built-in redrive functionality. As this is a relatively recent addition (December 2021), the Lambda pattern in SDLF was designed prior to this.

Another advantage to this is that a Lambda can only redrive 10 messages at a time per each call to SQS. This redrive feature in SQS allows you to redrive thousands of messages in a matter of seconds by just pressing the "redrive" button.

[anmolsgandhi]

Continuing the conversation on redrive, i believe users can have multiple ways to redrive, most common i have seen is having a lambda to reprocess, that is also a pattern we have in SDLF. So essentially DLQ --> Lambda --> SNS event/Lambda logic --> SNS

A Lambda could solve this, yes. The reason I think an SQS queue is still preferable is that it comes with built-in redrive functionality. As this is a relatively recent addition (December 2021), the Lambda pattern in SDLF was designed prior to this.

Another advantage to this is that a Lambda can only redrive 10 messages at a time per each call to SQS. This redrive feature in SQS allows you to redrive thousands of messages in a matter of seconds by just pressing the "redrive" button.

Yes that does make sense, if we move towards a SNS->SQS->LAMBDA stage then we completely dont provide for the SNS->LAMBDA usecase. thoughts ??

[malachi-constant]

Continuing the conversation on redrive, i believe users can have multiple ways to redrive, most common i have seen is having a lambda to reprocess, that is also a pattern we have in SDLF. So essentially DLQ --> Lambda --> SNS event/Lambda logic --> SNS

A Lambda could solve this, yes. The reason I think an SQS queue is still preferable is that it comes with built-in redrive functionality. As this is a relatively recent addition (December 2021), the Lambda pattern in SDLF was designed prior to this.
Another advantage to this is that a Lambda can only redrive 10 messages at a time per each call to SQS. This redrive feature in SQS allows you to redrive thousands of messages in a matter of seconds by just pressing the "redrive" button.

Yes that does make sense, if we move towards a SNS->SQS->LAMBDA stage then we completely dont provide for the SNS->LAMBDA usecase. thoughts ??

My vote would be to add SNS as an optional feature to our existing sqs to lambda stage or just inherit from the existing stage with the additional functionality and event target.

[kukushking]

if we move towards a SNS->SQS->LAMBDA stage then we completely dont provide for the SNS->LAMBDA usecase.

My 2 cents: using SNS + SQS together is a very common scalable pattern and it is able to cover same requirements as direct SNS -> lambda and SQS + lambda. You can scale consumers and producers independently and add more queues if running into lambda concurrency limits.

The reason I think an SQS queue is still preferable is that it comes with built-in redrive functionality. As this is a relatively recent addition (December 2021), the Lambda pattern in SDLF was designed prior to this.

Another advantage to this is that a Lambda can only redrive 10 messages at a time per each call to SQS. This redrive feature in SQS allows you to redrive thousands of messages in a matter of seconds by just pressing the "redrive" button.

Neat, didn't know about this addition. It was always a pain to only be able to redrive by 10 messages at a time.