Feature request: Add schema parsing issues when using the Batch processor with parser

[sdangol]

Use case

When there's a parsing error when using the Parser with Batch Processor, I only get the error message "Failed to parse record" but there is no information regarding why the parsing failed.

Solution/User Experience

It would be nice to get the error details on why the parsing failed.

Alternative solutions

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[dreamorosi]

After our discussion offline we settled on walking back some of the API we introduced as there was too much complexity as evidenced by the challenge in making the internal logic observable.

Because of this, we'll settle on a slightly simpler API that is however more explicit in what it does.

Specifically, we're going to change the input object of the BatchProcessor to accept two sets of options:

either

{
  schema: StandardSchemaV1;
  transform: never;
  innerschema: never;
  logger?: GenericLogger
}

or

{
  schema: never;
  transform?: 'json' | 'base64' | 'unmarshall'  // undefined means no transformer
  innerschema: z.ZodTypeAny; // (review this type because I am not sure it's the right one)
  logger?: GenericLogger
}

This allows customers to have an easy turn-key option in which they only pass the schema for the inner payload and we extend the base schema + apply the optional transform or have full control over the schema.

With this API we also simplify greatly the internal logic and now only do at most one parsing, which makes it easier to also emit logs and provide meaningful errors.

As discussed, since we're aiming at releasing tomorrow, if we are unable to complete the implementation in time, let's just modify the type for the BatchProcessor and remove this option object we introduced, so we have more time to work on it in the next days.

[powertools-for-aws-oss-automation]

Warning

This issue is now closed. Please be mindful that future comments are hard for our team to see.
If you need more assistance, please either reopen the issue, or open a new issue referencing this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.