Enhancement: Make kinesis_streams MAX_EVENT_SIZE configurable or support >1MB records

[JoaoSousa00]

Is your feature request related to a problem? Please describe.

The plugin out_kinesis_streams has a hard-coded maximum record size:

• MAX_EVENT_SIZE = 1048556

This causes records larger than 1MB to be discarted with a warning like:

• Discarding record which is larger than max size allowed by Firehose, %s

This behavior limits use cases where larger records are valid (especially since AWS Kinesis Data Streams now supports up to 10MB record sizes when the stream supports “large records”).

According to AWS’s official announcement, Amazon Kinesis Data Streams now supports record sizes up to 10 MiB, a tenfold increase over the previous 1 MiB limit.

Describe the solution you'd like

Please consider the following improvement:

• Make the maximum record size configurable (e.g., via a plugin parameter), so that users can align the plugin behavior with the actual stream configuration and avoid discarding valid records.

Describe alternatives you've considered

Please consider the following improvement:

• Make the maximum record size bigger, so that users can avoid discarding valid records.

Additional context
N/A

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.