Skip to content

[Improvement][rust] Align fixed-schema log reads by column ID #3770

Description

@slfan1989

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

The Rust fixed-schema log scanner currently aligns historical Arrow batches with the scanner schema by Arrow field name.

This differs from the KV FixedSchemaDecoder, which uses stable Fluss column IDs through index_mapping.

Name-based alignment is not robust for schema evolution:

  • A column with the same ID but a different name may be treated as missing.
  • A newly added column that reuses an old column name but has a different ID
    may incorrectly read the old column value.

Although the Fluss server does not currently support RENAME COLUMN, the Rust fixed-schema decoding path should consistently use column IDs as the stable column identity.

Solution

Use Fluss column IDs when aligning historical Arrow batches with the fixed target schema:

  • Preserve values when the source and target columns have the same ID.
  • Fill the target column with NULL when its ID does not exist in the source
    schema.
  • Do not match columns that only have the same name but different IDs.
  • Apply the same behavior to local and remote log reads.
  • Reuse the existing index_mapping logic used by FixedSchemaDecoder.

Anything else?

This improvement does not change the public API or storage format.

Willingness to contribute

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions