Skip to content

[refine](exec) simplify DataQueue eos handling#65324

Open
Mryange wants to merge 1 commit into
apache:masterfrom
Mryange:refine-data-queue-eos
Open

[refine](exec) simplify DataQueue eos handling#65324
Mryange wants to merge 1 commit into
apache:masterfrom
Mryange:refine-data-queue-eos

Conversation

@Mryange

@Mryange Mryange commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

DataQueue required callers to push data, manually mark child finish, and call remaining_has_data() to advance the internal child cursor before reading. This made eos handling spread across sink/source operators and exposed DataQueue internals to callers.

This change lets push_block() carry eos, makes get_block_from_queue() return a DataQueueBlock with the queue-level eos result, keeps per-child free lists internal to DataQueueBlock, and moves round-robin child selection into DataQueue. Union and cache operators now consume the unified queue result directly.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Mryange

Mryange commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed for PR #65324.

I did not find a substantiated blocking issue to raise inline.

Checkpoint conclusions:

  • Goal and scope: the PR centralizes DataQueue EOS handling and updates union/cache source and sink callers consistently.
  • Data correctness / EOS: final data blocks and empty EOS notifications are preserved through push_block(..., eos), get_block_from_queue(), union source, and cache source.
  • Concurrency: multiple sink writers still use per-subqueue locks and atomics; the source-side round-robin cursor is single-reader state for the per-task DataQueue shared state used by union/cache split pipelines.
  • Lifecycle: terminate() marks children finished, clears queued/free blocks, resets the total counter, and wakes the source dependency; no new static/global lifecycle concern was found.
  • Config / compatibility / persistence: no new config, wire format, storage, or edit-log compatibility surface was introduced.
  • Parallel paths: union pass-through/materialized paths and cache miss/hit paths were checked; no distinct missing propagation path was found.
  • Tests: DataQueueTest was updated for EOS-only and unified DataQueueBlock behavior. I attempted ./run-be-ut.sh --run --filter=DataQueueTest.*, but stopped it after it expanded into third-party dependency bootstrap in this checkout, so no BE UT result is claimed. git diff --check passed.

Subagent conclusions:

  • optimizer-rewrite: no candidate findings; convergence round 1 returned NO_NEW_VALUABLE_FINDINGS.
  • tests-session-config: proposed TSC-001 about _flag_queue_idx concurrency. I dismissed it with code evidence because union/cache DataQueue shared state is per source pipeline task in the reviewed split-operator wiring, leaving one source reader per queue. Convergence round 1 returned NO_NEW_VALUABLE_FINDINGS.

User focus: no additional user-provided review focus was supplied.

@Mryange

Mryange commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants