out_s3: Address underflow of buffer size on startup - #12266
Conversation
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe S3 output now uses atomic buffer and quarantine accounting, restores usage for existing files, exposes atomic file-size reads, and handles quota-aware cleanup. Runtime tests cover near-limit appends and restart recovery. ChangesS3 storage accounting
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bc8eef071e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
plugins/out_s3/s3_store.h (1)
36-36: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winParenthesize the negative macro value.
S3_STORE_QUARANTINE_FULLexpands to a bare-2. Any use in a larger expression, for examplex - S3_STORE_QUARANTINE_FULL, changes meaning. Wrap the value in parentheses.♻️ Proposed change
-#define S3_STORE_QUARANTINE_FULL -2 +#define S3_STORE_QUARANTINE_FULL (-2)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/out_s3/s3_store.h` at line 36, Update the S3_STORE_QUARANTINE_FULL macro to expand to the parenthesized value (-2), preserving its numeric value while ensuring safe use in larger expressions.plugins/out_s3/s3_store.c (1)
99-102: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAlign the limit comparison with
quarantine_size_reserve.
buffer_size_reserverejects whenupdated_size >= ctx->store_dir_limit_size.quarantine_size_reserverejects only whenupdated_size > ctx->quarantine_dir_limit_size. The two helpers therefore treat an exactly-at-limit write differently, andstore_dir_limit_sizerejects a write that would land exactly on the configured limit.If the strict
>=is intentional to preserve the previous out_s3 behavior, add a short/* ... */comment. Otherwise use>in both helpers.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/out_s3/s3_store.c` around lines 99 - 102, Align the boundary behavior in buffer_size_reserve with quarantine_size_reserve by changing the store_dir_limit_size comparison to reject only when updated_size is greater than the limit; if retaining the existing >= behavior, add a brief inline comment documenting that intentional difference.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/out_s3/s3_store.c`:
- Around line 335-343: Update the near-full limit check in buffer_chunk so the
warning after a successful append does not return -1 or otherwise signal retry.
Preserve the warning and return success after the buffer-size update, or move
the check before the append while ensuring the event is not appended twice.
---
Nitpick comments:
In `@plugins/out_s3/s3_store.c`:
- Around line 99-102: Align the boundary behavior in buffer_size_reserve with
quarantine_size_reserve by changing the store_dir_limit_size comparison to
reject only when updated_size is greater than the limit; if retaining the
existing >= behavior, add a brief inline comment documenting that intentional
difference.
In `@plugins/out_s3/s3_store.h`:
- Line 36: Update the S3_STORE_QUARANTINE_FULL macro to expand to the
parenthesized value (-2), preserving its numeric value while ensuring safe use
in larger expressions.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: ee17859e-d50b-4909-a68d-2c1b6fd0ff60
📒 Files selected for processing (5)
plugins/out_s3/s3.cplugins/out_s3/s3.hplugins/out_s3/s3_store.cplugins/out_s3/s3_store.htests/runtime/out_s3.c
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/runtime/out_s3.c`:
- Around line 1097-1142: The near-full test around s3_store_buffer_put must also
cover an append exceeding the 100-byte store_dir_limit_size. After the
successful 96-byte append, call s3_store_buffer_put with an additional payload
that crosses the limit, assert it fails, and verify s3_ctx->current_buffer_size
and s3_store_file_size_get(s3_file) both remain 96 bytes.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7877f431-2816-4b45-9340-8490595c6d4f
📒 Files selected for processing (2)
plugins/out_s3/s3_store.ctests/runtime/out_s3.c
🚧 Files skipped from review as they are similar to previous changes (1)
- plugins/out_s3/s3_store.c
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
92f597a to
063e7cd
Compare
cosmo0920
changed the title
2 participants
Implemented the complete fix. This is an S3 accounting underflow, not ordinary backpressure.
Key changes:
current_buffer_sizeduring startup.2^64 − N.Verification passed:
cmake --build build -j8ctest --test-dir build -R '^flb-rt-out_s3$' --output-on-failuretests/integration/.venv/bin/python -m pytest tests/integration/scenarios/out_s3/tests/test_out_s3_001.py -q— 10 passed, 2 skippedLEAKS=1 LEAKS_STRICT=1 tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/out_s3/tests/test_out_s3_001.py -q— 10 passed, 2 skipped; strict macOS Leaks cleanCloses #12164 #12149.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
Bug Fixes
Tests