Feature request: Parser, SNS envelope: Add messageId to parsed event

[Twonky-Chris]

Use case

The Parser returns only the parsed Message. It would be very helpful to also have the messageId in hand, for example, for error handling, especially as item identifier for batchItemFailures.

Solution/User Experience

maybe like this:

.use(
parser({ schema: validator, envelope: SnsSqsEnvelope, options: {returnMessageId: true })
)

Alternative solutions

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #typescript channel on our Powertools for AWS Lambda Discord: Invite link

[dreamorosi]

Hi @Twonky-Chris, thank you for opening this issue.

Just to clarify your use case, in your code snippet validator is a custom Zod schema that you're using to specify how we should parse the message field in each SNS message, is that correct?

[Twonky-Chris]

Hi @Twonky-Chris, thank you for opening this issue.

Just to clarify your use case, in your code snippet validator is a custom Zod schema that you're using to specify how we should parse the message field in each SNS message, is that correct?

Yes, correct. My suggestion was just about the "options: {returnMessageId: true }".

[everett1992]

I think there's two usecases to consider and we could handle them with different methods. The first is that you just want to access the envelop's content, and the second is that you want the evelope and content.

const event = { Records: [ { messageId: "...", body: "{\"foo\": 1}" } ] }
const schema = z.object({foo: z.string()})
type Schema = z.infer<typeof schema>

// Just get the array of parsed message bodies
SqsSnsEnvelope.parseInner(event, schema) satisfies Schema[]
 => [ { foo: 1 } ]

// get the array of parsed message bodies and keep the envelope.
// Not sure what the type should be 
// - should we keep the Records wrapper, or is there no value there?
// - should we keep the unparsed body so we still have a SQSRecord, or replace it?
SqsSnsEnvelope.parseAll(event, schema) satisfies { Records: (SqsSnsMetadata & Schema)[] }
 => { Records: [ { messageid: "...", body: { foo: 1 } }] }

[dreamorosi]

Thank you both for the context.

I agree that right now, for certain/most of the built-in schemas and envelopes there's this grey area between just using the built-in schema and using the corresponding built-in envelope.

Right now you can either use the schema, which gives you access to the entire parsed event but without transforming the payload, or you use the envelope which does transform the payload into something usable but locks you out of all other fields.

Our initial assumption was that most customers would only be interested in the payload (i.e. message, body, etc.) so envelopes fit the requirements, and for all other customers who needed more fields, schemas were available. This is probably not 100% accurate.

We have already started discussing how to solve this, and generally speaking we are not likely going to add extra options like options: {returnMessageId: true }. If we did that, next week or next month another customer could come and ask us to add a new option for another field, and eventually we could end up redefining the entire schema in the options - which is not where we want to go.

Adding new custom methods on the envelope like parseInner() and parseAll() is an option. In the design for envelopes we tried to stay as close as possible to the Zod API instead of adding new methods. This helps with familiarity but also keeps the door open to add support for other current or future Zod features like parseAsync().

We have some ideas and @am29d is working on this - so we expect to be able to share more in the coming weeks.