out_kinesis_streams: increase PUT_RECORDS_PAYLOAD_SIZE to 10MB

[ShelbyZ]

• Increase size from 5MB -> 10MB to match recent changes with PutRecords API

Kinesis Streams PutRecords API change in 2025 to support events up to 10MB (ref) while our defaults reflect the older value 1MB.

Add runtime tests to validate the new limit:

event_size_near_limit: Validates events just under 10MB are accepted
event_size_at_aws_max: Validates events near AWS max are discarded (existing behavior)
event_truncation_with_backslash: Validates backslash handling at truncation boundary

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• [N/A] Example configuration file for the change
• [N/A] Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

Logs: https://gist.github.com/ShelbyZ/0c8d005d39ddc8759cf6b77d7dc90dc5

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [N/A] Documentation required for this feature

Backporting

• [N/A] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features
  • Increased Kinesis output plugin maximum payload size from 5 MB to 10 MB.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b5d606f0-5737-407e-8e84-505ff3f5a545

📥 Commits

Reviewing files that changed from the base of the PR and between c2fec52 and cf8e1b4.

📒 Files selected for processing (1)

• tests/runtime/out_kinesis.c

---

📝 Walkthrough

Walkthrough

This PR increases the Kinesis plugin's maximum PutRecords payload size from 5 MiB to 10 MiB and adds three runtime tests that validate event-size boundary behavior: an event just under the limit, an oversized event expected to be discarded, and an oversized event with backslash escapes to exercise special-character handling.

Changes

Kinesis Payload Size Increase and Event-Size Boundary Tests

Layer / File(s)	Summary
Kinesis PutRecords payload size constant update plugins/out_kinesis_streams/kinesis_api.h	The compile-time constant PUT_RECORDS_PAYLOAD_SIZE is doubled from 5,242,880 (5 MiB) to 10,485,760 (10 MiB), raising the maximum payload for PutRecords API operations.
Kinesis event-size test infrastructure tests/runtime/out_kinesis.c	Add Kinesis API header include to expose the payload size constant. Define create_large_log_event helper to allocate and build JSON records of a requested size, and run_kinesis_test_with_data runner that initializes a Kinesis output, pushes test data, and tears down the engine context.
Kinesis event-size boundary test cases tests/runtime/out_kinesis.c	Add three runtime tests covering payload boundaries: flb_test_kinesis_event_size_at_limit for an event just under the maximum, flb_test_kinesis_event_size_over_limit for an oversized event, and flb_test_kinesis_event_size_with_backslash for an oversized event with regular and boundary-adjacent backslash escape sequences. Register all three in TEST_LIST.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related issues

• Enhancement: Make kinesis_streams MAX_EVENT_SIZE configurable or support >1MB records #11524: Addresses the Kinesis plugin's hard-coded maximum record/payload size constraint by raising PUT_RECORDS_PAYLOAD_SIZE to 10 MiB, directly related to the request for support for >1 MB or configurable maximum sizes.

Suggested reviewers

• edsiper
• cosmo0920

Poem

🐰 From five to ten the limits grow,
Kinesis streams now free to flow,
With tests that dance on boundaries near,
Backslashes skip without a fear,
A doubling done, the code runs clear! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: increasing PUT_RECORDS_PAYLOAD_SIZE from 5MB to 10MB in the out_kinesis_streams plugin.
Docstring Coverage	✅ Passed	Docstring coverage is 84.62% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/runtime/out_firehose.c`:
- Around line 789-793: The variable declaration "size_t boundary" must be moved
from mid-block into the top of the function’s declaration section; declare
"size_t boundary = MAX_EVENT_SIZE - 1;" alongside the other local variable
declarations at the start of the function, then use that variable in the
existing logic that sets large_json[boundary] and large_json[boundary + 1] when
checking "if (boundary + 1 < total_len - suffix_len)". Ensure no other
references rely on a narrower scope so compilation and behavior remain
unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a2b7b2f1-1a61-4a5a-88ec-61e525b12e75

📥 Commits

Reviewing files that changed from the base of the PR and between f6126eb and 673f5b2.

📒 Files selected for processing (2)

• plugins/out_kinesis_streams/kinesis_api.h
• tests/runtime/out_firehose.c

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 673f5b201e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".