Skip to content

[core][python][spark] Optimize data evolution write column metadata - #9574

Merged
JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/optimize-data-evolution-write-cols
Sep 3, 2026
Merged

[core][python][spark] Optimize data evolution write column metadata#9574
JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/optimize-data-evolution-write-cols

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Purpose

Reduce DataFileMeta size for wide data-evolution tables with dedicated BLOB or vector fields.

This adds the disabled-by-default data-evolution.write-cols-optimization.enabled option. When enabled, normal files containing every non-dedicated field omit the redundant writeCols list, while dedicated files keep explicit columns.

Readers in Java and Python resolve omitted metadata from the historical table schema. The change also covers stats evolution, conflict detection and rewrite, streaming column discovery, global-index refresh, Daft reads, and Spark copy-files schema rebinding.

For compatibility, persist the option with CREATE TABLE or ALTER TABLE so file schema versions record the compact representation, and upgrade all readers before enabling it.

Tests

  • mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=none -Dtest=TableSchemaTest,BlobTableTest#testOmitWriteColsForAllNonDedicatedColumns,VectorStoreTableTest#testOmitWriteColsForAllNonDedicatedColumns,RowIdColumnConflictCheckerTest,FilesTableTest#testReadStatsWithOmittedNonDedicatedWriteCols,DataEvolutionUtilsTest test
  • mvn -pl paimon-spark/paimon-spark-common -am -Pfast-build -Pspark3 -DfailIfNoTests=false -DwildcardSuites=none -Dtest=CopyFilesUtilTest test
  • python3 -m pytest -q pypaimon/tests/table_schema_test.py pypaimon/tests/blob_table_test.py pypaimon/tests/write/conflict_detection_test.py pypaimon/tests/ray_row_id_conflict_rewriter_test.py pypaimon/tests/vector_table_test.py::VectorTableWriteReadTest::test_vector_table_partial_update_non_vector_column
  • python3 -m compileall -q pypaimon
  • git diff --check

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the persisted CREATE/ALTER workflow across Java, PyPaimon, and Spark, including historical-schema resolution, BLOB/vector dedicated files, stats, conflict detection, compaction, streaming discovery, and copy-files schema rebinding. Targeted Python tests passed, and the targeted Java tests passed except for a local CodeGenerator service-discovery failure unrelated to this change. No blocking issue found.

@JingsongLi
JingsongLi merged commit 0ce6a13 into apache:master Sep 3, 2026
21 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants