Feature request: ability to not throw error on full batch failure

[leandrodamascena]

Use case

Currently when using the Batch Processor utility, if all the records in a batch are marked as failed the utility throws a BatchProcessingError.

Taking into consideration that the utility is supposed to be used with partial failure reporting, a Lambda function that throws an error is functionally equal to a partial failure that reports all items as failed in the sense that all the items in that batch are retried as a result.

While we initially implemented this as an error to reflect the full batch failure in the operational metrics (i.e. function runtime errors), there are cases such as when processing small batches that this behavior can skew the metrics due to higher chances of a full batch to fail.

To accommodate these use cases, as well as those customers who simply want to avoid throwing an error, we should add a new raise_on_batch_fail (or similar) option to the utility that allows customers to opt out of the error throwing mechanism.

More information here: aws-powertools/powertools-lambda-typescript#2122 (comment)

Solution/User Experience

from aws_lambda_powertools import Logger, Tracer
from aws_lambda_powertools.utilities.batch import (
    BatchProcessor,
    EventType,
    process_partial_response,
)
from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord
from aws_lambda_powertools.utilities.typing import LambdaContext

processor = BatchProcessor(event_type=EventType.SQS, raise_on_batch_fail= False)  
tracer = Tracer()
logger = Logger()


@tracer.capture_method
def record_handler(record: SQSRecord):  
    payload: str = record.json_body  # if json string data, otherwise record.body for str
    logger.info(payload)


@logger.inject_lambda_context
@tracer.capture_lambda_handler
def lambda_handler(event, context: LambdaContext):
    return process_partial_response(  
        event=event,
        record_handler=record_handler,
        processor=processor,
        context=context,
    )

Alternative solutions

No response

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under 2.41.0 version!

[github-actions]

This is now released under 2.42.0 version!