LOG-9386: fix(buffers): replace protobuf nesting-limit fix with merged upstream #26099 - #293
Conversation
…d upstream vectordotdev#26099 The fork adopted vectordotdev#25417 (commit 6873155) early, before the Vector team approved it. That PR was ultimately closed in favor of the better solution merged as vectordotdev#26099. This replaces our vectordotdev#25417 port with vectordotdev#26099, adapted onto v0.54.0's buffer base (which predates upstream's TryWriteOutcome refactor, so writer.rs and the disk backend are left untouched). What vectordotdev#26099 fixes over vectordotdev#25417: - State-independent routing: an over-nested event no longer takes a different path based on disk occupancy. Previously it was pruned while the disk had room but forwarded whole once the disk reported full. The overflow decision now lives in BufferSender (via the new Bufferable::is_fully_encodable) and is made before the item reaches any backend, so the same item is treated the same at 99% and 100% full. The disk backend filter is now unconditional. - Backend-aware overflow: SenderAdapter::requires_encodable_items() gates the divert so only wire-format-constrained stages (disk) trigger it. A memory overflow stage keeps arbitrarily nested items instead of having them pruned for a disk write that was never the final destination. - Single safe budget: the two per-path limits (99 for Log.fields/Trace.fields, 96 for metadata) collapse to one MAX_VALUE_NESTING_FRAMES = 96, the highest limit safe on every wire path. Validation no longer depends on event type or destination field, closing the gap where a non-object Log.value root could pass the gate at cost 97-99 yet fail prost decode on the receiver. Adds regression tests for the near-full, already-full, and unconstrained-base cases, plus the tightest-wire-path budget check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@vparfonov: This pull request references LOG-9386 which is a valid jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: vparfonov The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/assign @jcantrill |
|
@vparfonov: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
3 participants
That PR was ultimately closed in favor of the better solution merged as vectordotdev#26099. This replaces our vectordotdev#25417 port with vectordotdev#26099, adapted onto v0.54.0's buffer base (which predates upstream's TryWriteOutcome refactor, so writer.rs and the disk backend are left untouched).
What vectordotdev#26099 fixes over vectordotdev#25417:
State-independent routing: an over-nested event no longer takes a different path based on disk occupancy. Previously it was pruned while the disk had room but forwarded whole once the disk reported full. The overflow decision now lives in BufferSender (via the new Bufferable::is_fully_encodable) and is made before the item reaches any backend, so the same item is treated the same at 99% and 100% full. The disk backend filter is now unconditional.
Backend-aware overflow: SenderAdapter::requires_encodable_items() gates the divert so only wire-format-constrained stages (disk) trigger it. A memory overflow stage keeps arbitrarily nested items instead of having them pruned for a disk write that was never the final destination.
Single safe budget: the two per-path limits (99 for Log.fields/Trace.fields, 96 for metadata) collapse to one MAX_VALUE_NESTING_FRAMES = 96, the highest limit safe on every wire path. Validation no longer depends on event type or destination field, closing the gap where a non-object Log.value root could pass the gate at cost 97-99 yet fail prost decode on the receiver.
Adds regression tests for the near-full, already-full, and unconstrained-base cases, plus the tightest-wire-path budget check.