Skip to content

[client] Implement BatchScanner with limit-based scan#515

Open
Arnav-panjla wants to merge 3 commits into
apache:mainfrom
Arnav-panjla:feat/batch-scanner-316
Open

[client] Implement BatchScanner with limit-based scan#515
Arnav-panjla wants to merge 3 commits into
apache:mainfrom
Arnav-panjla:feat/batch-scanner-316

Conversation

@Arnav-panjla
Copy link
Copy Markdown
Contributor

@Arnav-panjla Arnav-panjla commented Apr 26, 2026

Purpose

Fixes #316

Implement one-shot BatchScanner using LimitScanRequest.

Brief change log

  • Added limit to TableScan
  • Added create_batch_scanner
  • Implemented BatchScanner (eager RPC, single poll_batch)
  • Added Arrow IPC (log) and KV → RecordBatch (PK) decoding
  • Added projection support

Tests

  • All tests pass (cargo test)
  • Verified with cargo check, clippy, fmt

API and Format

API Change: Yes

  • New BatchScanner, TableScan::limit, create_batch_scanner

Storage Format: No

Documentation

No

fresh-borzoni
fresh-borzoni reviewed Apr 30, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@fresh-borzoni fresh-borzoni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Arnav-panjla Ty for the PR, left some comments, PTAL

Comment thread Cargo.lock
Comment thread crates/fluss/src/client/table/batch_scanner.rs
Comment thread crates/fluss/src/client/table/batch_scanner.rs Outdated
@fresh-borzoni
Copy link
Copy Markdown
Member

fresh-borzoni commented May 9, 2026

@Arnav-panjla Do you need any help, will you be able to address comments in the review?

@fresh-borzoni
Copy link
Copy Markdown
Member

fresh-borzoni commented May 13, 2026

@Arnav-panjla Let us know if you wish to continue working on this PR, or we can take it over to finish

@Arnav-panjla
Copy link
Copy Markdown
Contributor Author

Arnav-panjla commented May 13, 2026

very sry ,
had my exams going on
I’ll update you with the next iteration within a day.

@fresh-borzoni
Copy link
Copy Markdown
Member

fresh-borzoni commented May 13, 2026

very sry , had my exams going on I’ll update you with the next iteration within a day.

@Arnav-panjla No rush, I just need to understand that you wish to continue working on this.
Good luck with the exams!

@Arnav-panjla Arnav-panjla force-pushed the feat/batch-scanner-316 branch from e43866c to 9ca191f Compare May 23, 2026 10:20
@Arnav-panjla @fresh-borzoni
[client] Implement BatchScanner with limit-based scan
4fc2135 
Implements a one-shot bounded BatchScanner backed by a single
LimitScanRequest RPC (fixes apache#316):
- adds TableScan::limit and create_batch_scanner
- eager RPC with leader resolution (mirrors Lookuper)
- Arrow IPC (log) and KV -> RecordBatch (PK) decoding
- projection support

Squashed from PR apache#515.
@fresh-borzoni fresh-borzoni force-pushed the feat/batch-scanner-316 branch 2 times, most recently from 1c3290c to a7b994f Compare May 27, 2026 13:02
@fresh-borzoni
Copy link
Copy Markdown
Member

fresh-borzoni commented May 27, 2026 •
edited
Loading

@Arnav-panjla Ty for the changes, LGTM overall
I added some code to address schema evolution and kv wiring, hope you don't mind my help with it, PTAL

@charlesdong1991 @leekeiabstraction Can you take a look as well?

@fresh-borzoni fresh-borzoni force-pushed the feat/batch-scanner-316 branch from a7b994f to 6521684 Compare May 27, 2026 15:27
charlesdong1991
charlesdong1991 reviewed May 27, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@charlesdong1991 charlesdong1991 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice pr! just a couple minor comments/questions

Comment thread crates/fluss/src/client/table/scanner.rs
Comment thread crates/fluss/tests/integration/batch_scanner.rs Outdated
Comment thread crates/fluss/src/client/table/batch_scanner.rs Outdated
Comment thread crates/fluss/src/client/table/batch_scanner.rs
@fresh-borzoni
Copy link
Copy Markdown
Member

fresh-borzoni commented May 27, 2026

@charlesdong1991 Ty for the review, addressed, PTAL 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fresh-borzoni fresh-borzoni fresh-borzoni left review comments

+1 more reviewer

@charlesdong1991 charlesdong1991 charlesdong1991 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Core BatchScanner implementation

3 participants

@Arnav-panjla @fresh-borzoni @charlesdong1991