[fix](auto partition) keep load row metrics monotonic for auto partition#64109
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Review result: no blocking issues found in this PR.
Critical checkpoint conclusions:
- Goal and proof: The PR aims to keep global BE load row/byte metrics monotonic for auto-partition batching while preserving RuntimeState progress compensation. The code accomplishes this by removing negative global metric deltas when rows are moved into the batching block and by skipping global metric increments while replaying the internal batched block. No targeted test is added, so the remaining risk is regression coverage for this metric-only behavior.
- Scope and clarity: The change is small and focused on the three relevant BE writer/distribution locations.
- Concurrency: The modified code does not introduce new shared state or new concurrency. The
_deal_batchedflag is used in the existing writer/replay flow, and the new local snapshot is taken after_send_new_partition_batch(), which preserves the intended distinction between recursive replay writes and normal source writes. - Lifecycle/static initialization: No new lifecycle ownership, static initialization, or release-order behavior is introduced.
- Configuration: No configuration item is added.
- Compatibility: No wire protocol, storage format, or function symbol compatibility impact was found.
- Parallel paths: Both
VTabletWriterandVTabletWriterV2are updated consistently. I also checked the row-distribution path where the negative delta was previously emitted. - Conditional checks: The new condition only guards global DorisMetrics increments and follows the existing
_deal_batchedreplay state. RuntimeState row/byte accounting remains unchanged. - Tests: No tests are added or modified. Given this is metric accounting, a targeted auto-partition load metric regression would reduce risk, but I did not find a functional blocker in the implementation.
- Observability: The change directly corrects observability metrics and does not require additional logs or metrics.
- Transaction/persistence/data writes: The PR does not change transaction visibility, rowset writes, persistence, or data routing semantics.
- FE/BE variable passing: No new FE/BE variables or thrift fields are added.
- Performance: The added boolean snapshot and conditional metric increments are negligible; no performance issue found.
- Other issues: No duplicate existing review threads were present, and no additional user-provided focus points were specified.
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
zclllyybb
approved these changes
Jun 4, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
TPC-H: Total hot run time: 29427 ms |
Contributor
TPC-DS: Total hot run time: 170258 ms |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
github-actions Bot
pushed a commit
that referenced
this pull request
Jun 5, 2026
…ion (#64109) ### What problem does this PR solve? When loading into an auto partition table, BE may temporarily batch rows whose target partitions do not exist yet. The previous logic first incremented `doris_be_load_rows` when receiving the input block, then decremented it when those rows were moved into the auto-partition batching block, and finally incremented it again when the batched rows were replayed after partition creation. Because `doris_be_load_rows` is exposed as a counter, the negative adjustment made the metric non-monotonic and could show row count drops during auto partition loads. This PR keeps the existing `RuntimeState` row compensation logic unchanged, but stops applying negative deltas to the global BE load row/byte metrics. It also skips global metric increments when replaying the internal batched block, so each source row is counted once.
github-actions Bot
pushed a commit
that referenced
this pull request
Jun 5, 2026
…ion (#64109) ### What problem does this PR solve? When loading into an auto partition table, BE may temporarily batch rows whose target partitions do not exist yet. The previous logic first incremented `doris_be_load_rows` when receiving the input block, then decremented it when those rows were moved into the auto-partition batching block, and finally incremented it again when the batched rows were replayed after partition creation. Because `doris_be_load_rows` is exposed as a counter, the negative adjustment made the metric non-monotonic and could show row count drops during auto partition loads. This PR keeps the existing `RuntimeState` row compensation logic unchanged, but stops applying negative deltas to the global BE load row/byte metrics. It also skips global metric increments when replaying the internal batched block, so each source row is counted once.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
When loading into an auto partition table, BE may temporarily batch rows whose target partitions do not exist yet. The previous logic first incremented
doris_be_load_rowswhen receiving the input block, then decremented it when those rows were moved into the auto-partition batching block, and finally incremented it again when the batched rows were replayed after partition creation.Because
doris_be_load_rowsis exposed as a counter, the negative adjustment made the metric non-monotonic and could show row count drops during auto partition loads.This PR keeps the existing
RuntimeStaterow compensation logic unchanged, but stops applying negative deltas to the global BE load row/byte metrics. It also skips global metric increments when replaying the internal batched block, so each source row is counted once.