Skip to content

[chore](format) remove the batch size in parquet/orc reader#62761

Closed
zhangstar333 wants to merge 3 commits into
apache:masterfrom
zhangstar333:rm_batch_size
Closed

[chore](format) remove the batch size in parquet/orc reader#62761
zhangstar333 wants to merge 3 commits into
apache:masterfrom
zhangstar333:rm_batch_size

Conversation

@zhangstar333
Copy link
Copy Markdown
Contributor

@zhangstar333 zhangstar333 commented Apr 23, 2026

What problem does this PR solve?

Problem Summary:

before the reader use _MIN_BATCH_SIZE as batch size,
now use the batch size from runtime state.
so OrcReader/ParquetReader c'tor pass the runtime_state object, and remove the batch_size, ctz params.
Unified constructor order as
(profile, state, params, range, io_ctx, meta_cache=null, enable_lazy_mat=true, [subclass-specific parameters])

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhangstar333
Copy link
Copy Markdown
Contributor Author

run buildall

@zhangstar333
Copy link
Copy Markdown
Contributor Author

/review

@zhangstar333
Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

OpenCode automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/24877329588

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen
Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 62.79% (54/86) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.49% (26794/37480)
Line Coverage 53.88% (279873/519450)
Region Coverage 47.27% (215399/455698)
Branch Coverage 50.56% (97423/192698)

Comment thread be/src/service/internal_service.cpp Outdated
Comment thread be/src/service/internal_service.cpp Outdated
Comment thread be/src/format/table/transactional_hive_reader.h
Comment thread be/src/format/table/table_format_reader.h
@zhangstar333
Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 33.66% (34/101) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.39% (20430/38268)
Line Coverage 36.93% (192559/521441)
Region Coverage 33.24% (149786/450647)
Branch Coverage 34.37% (65552/190746)

@hello-stephen
Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 90.10% (91/101) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.86% (27684/37481)
Line Coverage 57.62% (299515/519844)
Region Coverage 54.71% (248837/454804)
Branch Coverage 56.36% (107821/191319)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants