[feature](lance) Add Rust-based Lance format reader for AI-native dat… by tomz-alt · Pull Request #62182 · apache/doris

tomz-alt · 2026-04-07T17:01:50Z

Summary

Add native Lance format support to Doris via Rust FFI integration, enabling SQL queries over AI-native Lance datasets from local disk and S3.

Lance is a columnar format designed for vector search, multimodal data (images, embeddings), and fast random access -- widely used in AI/ML pipelines.

Quick Examples

Read from S3:

SELECT * FROM s3(
    "uri" = "s3://bucket/embeddings.lance/data/fragment.lance",
    "format" = "lance",
    "s3.access_key" = "...",
    "s3.secret_key" = "...",
    "s3.region" = "us-east-1",
    "s3.endpoint" = "https://s3.us-east-1.amazonaws.com"
) ORDER BY id LIMIT 10;

Read from local disk (for testing):

-- Get backend_id from: SHOW BACKENDS;
SELECT * FROM local(
    "file_path" = "data/my_dataset.lance/data/fragment.lance",
    "backend_id" = "<backend_id from SHOW BACKENDS>",
    "format" = "lance"
) ORDER BY id LIMIT 10;

Aggregation across multi-fragment dataset:

SELECT count(*), min(id), max(id) FROM s3(
    "uri" = "s3://bucket/large.lance/data/fragment.lance",
    "format" = "lance",
    "s3.access_key" = "...", "s3.secret_key" = "...",
    "s3.region" = "us-east-1",
    "s3.endpoint" = "https://s3.us-east-1.amazonaws.com"
);

Architecture

Data exchange: Arrow C Data Interface (zero-copy between Rust and C++)
Async containment: block_on() with single-threaded tokio runtime (zero extra OS threads)
Build gating: BUILD_RUST_READERS=OFF by default, zero impact on existing builds

What Works (Verified on Live Cluster)

Feature	Status
SELECT * / column projection	Tested
WHERE filter / LIMIT / COUNT(*)	Tested
SUM() / AVG() aggregation	Tested
Multi-fragment datasets (3 fragments, 15 rows)	Tested
S3 access with AWS credentials	Tested
Schema inference (fetch_table_schema)	Tested
Time travel version (config wired)	Config ready
Vector ANN search / FTS / filter pushdown	Config ready

Known Limitations

TVF path only: No CREATE CATALOG support yet. Must use local() or s3() TVF
Directory-based format workaround: Lance datasets are directories. The TVF file_path must point to a single .lance data file inside the dataset; the reader auto-strips the path back to the dataset root and reads all fragments. If the TVF glob matches multiple .lance files (multi-fragment dataset), each scan range reopens the full dataset causing duplicate rows. Workaround: ensure the file_path glob matches exactly one data file per dataset
No Doris data cache integration: Lance reads bypass BlockFileCache. S3 reads are not cached on local SSD
No filter/vector pushdown from FE: The Rust config supports filter, vector_search, full_text_search but the FE planner does not populate them yet
BUILD_RUST_READERS=OFF default: Requires explicit opt-in and Rust toolchain
Binary size: Rust static lib ~430MB (.a), adds ~50-80MB to final doris_be after LTO

How to Build

# Rust tests only:
cd be/src/rust/doris-native && cargo test

# Full BE with Lance:
BUILD_RUST_READERS=ON ./build.sh --be

# Regression test:
./run-regression-test.sh --run -s test_lance_tvf

Changes

Thrift (2 files): FORMAT_LANCE = 19, TLanceFileDesc, enable_rust_lance_reader

FE (4 files): LanceFileFormatProperties, FileFormatProperties factory, FileFormatConstants, SessionVariable

BE - Rust (be/src/rust/doris-native/, 6 files): error.rs, lance_reader.rs (LanceReaderConfig with S3/version/vector/FTS support), ffi.rs (extern C functions), lib.rs

BE - C++ (5 files): lance_ffi.h, lance_rust_reader.h/cpp (GenericReader with Arrow import), file_scanner.cpp (FORMAT_LANCE dispatch), internal_service.cpp (fetch_table_schema)

Build (3 files): rust.cmake (Corrosion v0.5), CMakeLists.txt (BUILD_RUST_READERS option), format/CMakeLists.txt

Tests (4 files): 24 Rust tests, C++ GTest (7), C++ standalone (8), Groovy regression (9)

Future Work

Lance Catalog: CREATE CATALOG for dataset discovery (eliminates TVF path limitations and backend_id requirement)
Fragment-level scan ranges: FE lists fragments and creates one scan range per fragment with fragment ID, avoiding duplicate reads
FE filter/vector pushdown: Pass WHERE predicates and ANN queries to lance-rs scanner
Doris BlockFileCache integration: Route lance I/O through Doris cache for S3 data caching
Lance Session cache: Shared IndexCache for vector/FTS index reuse across queries
Time travel SQL: FOR VERSION AS OF N via LanceMvccSnapshot

Thearas · 2026-04-07T17:01:55Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2026-04-07T18:17:20Z

run buildall

hello-stephen · 2026-04-07T19:41:12Z

FE UT Coverage Report

Increment line coverage 18.18% (2/11) 🎉
Increment coverage report
Complete coverage report

tomz-alt · 2026-04-07T21:00:26Z

we need to install cargo toolchain to make rust ffi compile

hello-stephen · 2026-04-07T22:19:06Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.98% (20105/37947)
Line Coverage	36.53% (188940/517204)
Region Coverage	32.78% (146566/447093)
Branch Coverage	33.93% (64209/189233)

hello-stephen · 2026-04-07T22:26:00Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.69% (27386/37162)
Line Coverage	57.31% (295496/515622)
Region Coverage	54.51% (245979/451226)
Branch Coverage	56.18% (106642/189815)

hello-stephen · 2026-04-08T08:15:42Z

run buildall

hello-stephen · 2026-04-08T09:16:57Z

TeamCity pipeline will use regression-test/pipeline/common/custom_env.sh when compiling, please set BUILD_RUST_READERS=on in it for test.

hello-stephen · 2026-04-08T10:26:03Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.00% (20112/37947)
Line Coverage	36.56% (189072/517225)
Region Coverage	32.81% (146716/447111)
Branch Coverage	33.94% (64233/189251)

hello-stephen · 2026-04-08T10:46:05Z

BE Regression && UT Coverage Report

Increment line coverage 89.87% (71/79) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.72% (27397/37162)
Line Coverage	57.36% (295782/515643)
Region Coverage	54.61% (246412/451244)
Branch Coverage	56.29% (106852/189833)

hello-stephen · 2026-04-12T14:32:45Z

run buildall

hello-stephen · 2026-04-12T16:02:58Z

FE UT Coverage Report

Increment line coverage 18.18% (2/11) 🎉
Increment coverage report
Complete coverage report

morningman · 2026-04-13T17:39:47Z

run buildall

tomz-alt · 2026-04-13T18:58:56Z

run buildall

hello-stephen · 2026-04-13T19:35:27Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.48% (1798/2291)
Line Coverage	64.17% (32304/50345)
Region Coverage	65.13% (16260/24967)
Branch Coverage	55.63% (8689/15620)

hello-stephen · 2026-04-13T20:23:06Z

FE UT Coverage Report

Increment line coverage 18.18% (2/11) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-19T05:54:25Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.48% (1798/2291)
Line Coverage	64.10% (32273/50345)
Region Coverage	65.06% (16244/24967)
Branch Coverage	55.58% (8681/15620)

hello-stephen · 2026-04-19T09:01:52Z

FE Regression Coverage Report

Increment line coverage 90.91% (10/11) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-19T09:13:08Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.27% (20296/38100)
Line Coverage	36.76% (191118/519960)
Region Coverage	33.04% (148446/449349)
Branch Coverage	34.13% (64891/190108)

hello-stephen · 2026-04-19T09:18:57Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.68% (27490/37309)
Line Coverage	57.41% (297600/518344)
Region Coverage	54.59% (247557/453460)
Branch Coverage	56.18% (107131/190678)

Revert review-bot-driven changes that broke local dev ergonomics and CI: - Restore CMake auto-detect of pre-built libdoris_ffi.a (no env var required for dev builds) - Restore rust.cmake PARENT_SCOPE (its include() context had already set the cache variable; PARENT_SCOPE is the correct and safe form) - Remove enable_rust_lance_reader session variable guard in file_scanner.cpp - Revert case-insensitive column matching and per-unit timestamp scale in lance_rust_reader.cpp Keep the multi-fragment duplicate-row fix: lance_rust_reader.cpp extracts fragment_file from the scan range path and passes it through the JSON config; doris-ffi filters by matching data-file path so each scan range reads exactly one fragment (not the whole dataset). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

tomz-alt · 2026-04-20T18:45:34Z

run buildall

hello-stephen · 2026-04-20T20:17:52Z

FE UT Coverage Report

Increment line coverage 18.18% (2/11) 🎉
Increment coverage report
Complete coverage report

tomz-alt · 2026-04-20T20:51:25Z

run buildall

Upstream master refactored GenericReader virtual methods: get_next_block -> _do_get_next_block (protected hook) get_columns(name_to_type, missing_cols) -> _get_columns_impl(name_to_type) Update LanceRustReader's overrides to match the new signatures so CI (which builds our PR on top of current master) compiles. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…o lance-support

morningman · 2026-04-20T22:48:10Z

run buildall

tomz-alt · 2026-04-21T06:52:43Z

run buildall

suxiaogang223 · 2026-04-21T10:42:05Z

run buildall

morningman · 2026-04-21T15:41:26Z

run buildall

tomz-alt · 2026-04-21T18:09:55Z

so many PR per day....

hello-stephen · 2026-04-21T19:31:20Z

FE Regression Coverage Report

Increment line coverage 90.91% (10/11) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-21T19:51:36Z

BE Regression && UT Coverage Report

Increment line coverage 0.68% (2/296) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.71% (26833/37418)
Line Coverage	54.95% (285305/519212)
Region Coverage	51.76% (235098/454223)
Branch Coverage	53.27% (101803/191106)

morningman · 2026-04-07T18:04:44Z

+            fuzzy = true,
+            description = {"使用 Rust Lance 读取器读取 Lance 格式数据",
+                    "Use Rust-based Lance reader for Lance format data"})
+    private boolean enableRustLanceReader = false;


Do we really need this variable?

github-actions · 2026-04-21T20:09:48Z

PR approved by at least one committer and no changes requested.

dismiss

github-actions · 2026-04-21T20:22:49Z

PR approved by anyone and no changes requested.

tomz-alt requested review from BiteTheDDDDt and zclllyybb as code owners April 7, 2026 17:01

morningman self-assigned this Apr 7, 2026

tomz-alt force-pushed the lance-support branch 5 times, most recently from 9825d0e to 6f655a4 Compare April 7, 2026 18:15

tomz-alt force-pushed the lance-support branch from 6f655a4 to 2b57665 Compare April 7, 2026 22:45

tomz-alt force-pushed the lance-support branch 3 times, most recently from a02a47a to 9dcf75e Compare April 13, 2026 06:19

tomz-alt force-pushed the lance-support branch 2 times, most recently from d306c78 to f0bad39 Compare April 13, 2026 18:41

Merge branch 'master' into lance-support

88b0fa7

tom zhang and others added 3 commits April 20, 2026 15:42

Merge remote-tracking branch 'upstream/master' into lance-support

4025e89

Merge branch 'lance-support' of https://github.com/tomz-alt/doris int…

daf1e42

…o lance-support

morningman added the kind/need-document label Apr 21, 2026

Merge branch 'master' into lance-support

f12a8bb

fix conflict

5b01233

morningman approved these changes Apr 21, 2026

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Apr 21, 2026

github-actions Bot added the reviewed label Apr 21, 2026

BiteTheDDDDt approved these changes Apr 22, 2026

View reviewed changes

BiteTheDDDDt merged commit b613c37 into apache:master Apr 22, 2026
30 of 33 checks passed

Conversation

tomz-alt commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Quick Examples

Architecture

What Works (Verified on Live Cluster)

Known Limitations

How to Build

Changes

Future Work

Uh oh!

Thearas commented Apr 7, 2026

Uh oh!

morningman commented Apr 7, 2026

Uh oh!

hello-stephen commented Apr 7, 2026

FE UT Coverage Report

Uh oh!

tomz-alt commented Apr 7, 2026

Uh oh!

hello-stephen commented Apr 7, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 7, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Apr 8, 2026

Uh oh!

hello-stephen commented Apr 8, 2026

Uh oh!

hello-stephen commented Apr 8, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 8, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Apr 12, 2026

Uh oh!

hello-stephen commented Apr 12, 2026

FE UT Coverage Report

Uh oh!

morningman commented Apr 13, 2026

Uh oh!

tomz-alt commented Apr 13, 2026

Uh oh!

hello-stephen commented Apr 13, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Apr 13, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Apr 19, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Apr 19, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Apr 19, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 19, 2026

BE Regression && UT Coverage Report

Uh oh!

tomz-alt commented Apr 20, 2026

Uh oh!

hello-stephen commented Apr 20, 2026

FE UT Coverage Report

Uh oh!

tomz-alt commented Apr 20, 2026

Uh oh!

morningman commented Apr 20, 2026

Uh oh!

tomz-alt commented Apr 21, 2026

Uh oh!

suxiaogang223 commented Apr 21, 2026

Uh oh!

morningman commented Apr 21, 2026

Uh oh!

tomz-alt commented Apr 21, 2026

tomz-alt commented Apr 7, 2026 •

edited

Loading