Search before asking
Motivation
The Rust fixed-schema log scanner currently aligns historical Arrow batches with the scanner schema by Arrow field name.
This differs from the KV FixedSchemaDecoder, which uses stable Fluss column IDs through index_mapping.
Name-based alignment is not robust for schema evolution:
- A column with the same ID but a different name may be treated as missing.
- A newly added column that reuses an old column name but has a different ID
may incorrectly read the old column value.
Although the Fluss server does not currently support RENAME COLUMN, the Rust fixed-schema decoding path should consistently use column IDs as the stable column identity.
Solution
Use Fluss column IDs when aligning historical Arrow batches with the fixed target schema:
- Preserve values when the source and target columns have the same ID.
- Fill the target column with NULL when its ID does not exist in the source
schema.
- Do not match columns that only have the same name but different IDs.
- Apply the same behavior to local and remote log reads.
- Reuse the existing
index_mapping logic used by FixedSchemaDecoder.
Anything else?
This improvement does not change the public API or storage format.
Willingness to contribute
Search before asking
Motivation
The Rust fixed-schema log scanner currently aligns historical Arrow batches with the scanner schema by Arrow field name.
This differs from the KV
FixedSchemaDecoder, which uses stable Fluss column IDs throughindex_mapping.Name-based alignment is not robust for schema evolution:
may incorrectly read the old column value.
Although the Fluss server does not currently support
RENAME COLUMN, the Rust fixed-schema decoding path should consistently use column IDs as the stable column identity.Solution
Use Fluss column IDs when aligning historical Arrow batches with the fixed target schema:
schema.
index_mappinglogic used byFixedSchemaDecoder.Anything else?
This improvement does not change the public API or storage format.
Willingness to contribute