Skip to content

fix: field name mismatch when alter table rename caused by FieldMappingReader#274

Merged
lxy-9602 merged 4 commits into
alibaba:mainfrom
duanyyyyyyy:duanyan/fix_schema_evolution
May 12, 2026
Merged

fix: field name mismatch when alter table rename caused by FieldMappingReader#274
lxy-9602 merged 4 commits into
alibaba:mainfrom
duanyyyyyyy:duanyan/fix_schema_evolution

Conversation

@duanyyyyyyy
Copy link
Copy Markdown
Contributor

Purpose

The check does not consider column name change. When a user does pure ALTER TABLE RENAME COLUMN with no other schema change:

  • non_partition_data_schema[i].Name() = file's physical column name (pre-rename, e.g. age)
  • non_partition_read_schema[i].Name() = query's logical column name (post-rename, e.g. years)
  • Field ids are identical (RENAME doesn't change ids); idx mapping is identity; types unchanged; no non-exist fields

All four flag conditions evaluate to false → need_mapping_ = false → PASSTHRU.

The inner reader (e.g. paimon-cpp builtin parquet reader) emits the batch with the file's physical column name (age). The consumer (e.g. StarRocks BE) then looks up columns by their post-rename logical name
(years) using RecordBatch::GetColumnByName and gets a null shared_ptr. A subsequent member access on the null array dereferences nullptr → SIGSEGV.

This was missed by the existing TestAppendReadWithSchemaEvolution integration tests because they always combine RENAME with ADD COLUMN, which independently forces need_mapping_ = true via
non_exist_field_info_.

Tests

Added FieldMappingReaderTest.TestReadWithSchemaEvolutionPureRename:

  • File schema: (f0:utf8, f1:int32) with field ids (0, 1)
  • Read schema: (name_new:utf8, age_new:int32) with same field ids (0, 1) (pure RENAME)
  • Asserts the result ChunkedArray matches the expected one — arrow::ChunkedArray::Equals compares both data values and schema field names, so a passthrough of the file's old names would fail the assertion.

Without the fix this test fails (Equals returns false due to field name mismatch).

API and Format

Documentation

Generative AI tooling

opus 4.7

@duanyyyyyyy duanyyyyyyy force-pushed the duanyan/fix_schema_evolution branch from 7210f68 to d3389f9 Compare May 12, 2026 04:07
@lxy-9602 lxy-9602 changed the title Fix FieldMappingReader skipping mapping when only column names differ fix: field name mismatch when alter table rename caused by FieldMappingReader May 12, 2026
Copy link
Copy Markdown
Collaborator

@lxy-9602 lxy-9602 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602 lxy-9602 merged commit 4bf7ae6 into alibaba:main May 12, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants