Feature request: allow SqsFifoPartialProcessor to continue processing other group IDs upon failure

[dreamorosi]

Use case

Currently the SqsFifoPartialProcessor part of the Batch Processing utility short-circuits by throwing an error as soon as an item in a batch fails to process. This is done to prevent out-of-order processing.

By default, we will stop processing at the first failure and mark unprocessed messages as failed to preserve ordering. However, this behavior may not be optimal for customers who wish to proceed with processing messages from a different group ID.

We should look into adding this feature.

Solution/User Experience

Customers should be able to pass a skipGroupOnError option

This could be done by either accepting an option object when instantiating the processor, like:

import {
  SqsFifoPartialProcessor,
  processPartialResponseSync,
} from '@aws-lambda-powertools/batch';
import type {
CallableFunction
} from '@aws-lambda-powertools/batch/types';

declare recordHandler = CallableFunction;

const processor = new SqsFifoPartialProcessor({ skipGroupOnError: true });  // <-- New option

export const handler = async (
  event,
  context
) => {
  return processPartialResponseSync(event, recordHandler, processor, {
    context,
  });
};

Or by accepting a new option in the already existing option object accepted by the processPartialResponseSync function, like:

import {
  SqsFifoPartialProcessor,
  processPartialResponseSync,
} from '@aws-lambda-powertools/batch';
import type {
CallableFunction
} from '@aws-lambda-powertools/batch/types';

declare recordHandler = CallableFunction;

const processor = new SqsFifoPartialProcessor();

export const handler = async (
  event,
  context
) => {
  return processPartialResponseSync(event, recordHandler, processor, {
    context,
    skipGroupOnError: true // <-- New option
  });
};

The second one is probably more organic and allows customers additional customizability as they can define the value for skipGroupOnError in the context of a specific invocation, however it will likely require some TypeScript generic type so that it's accepted only when the processor parameter is of type SqsFifoPartialProcessor.

Either way, the value for skipGroupOnError should be saved in the SqsFifoPartialProcessor instance and then taken in account when calling the shortCircuitProcessing method to decide whether to throw or not.

Alternative solutions

N/A

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[arnabrahman]

@dreamorosi , thanks for the nicely detailed instructions. I would like to take a look at this issue.

[dreamorosi]

Hi @arnabrahman, sounds good!

If you have any questions please let me know 😊

[dreamorosi]

I didn't link it in the issue, but if it helps you can also take a look at the implementation of this feature in the Python version which is in this file here.

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under v2.2.0 version!