Skip to content

[core] fix stats evolution for data evolution tables#7797

Merged
JingsongLi merged 3 commits into
apache:masterfrom
steFaiz:fix_files_system_table
May 11, 2026
Merged

[core] fix stats evolution for data evolution tables#7797
JingsongLi merged 3 commits into
apache:masterfrom
steFaiz:fix_files_system_table

Conversation

@steFaiz
Copy link
Copy Markdown
Contributor

@steFaiz steFaiz commented May 9, 2026

Purpose

Currently stats evolution in Files System table do not consider data evolution, which may cause:

Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
	at org.apache.paimon.memory.MemorySegmentUtils.getLongMultiSegments(MemorySegmentUtils.java:779) ~[paimon-flink-1.18-1.4-beta-20260415.042901-14.jar:1.4-beta-SNAPSHOT]
	at org.apache.paimon.memory.MemorySegmentUtils.getLong(MemorySegmentUtils.java:769) ~[paimon-flink-1.18-1.4-beta-20260415.042901-14.jar:1.4-beta-SNAPSHOT]
	at org.apache.paimon.data.BinaryArray.getLong(BinaryArray.java:160) ~[paimon-flink-1.18-1.4-beta-20260415.042901-14.jar:1.4-beta-SNAPSHOT]
	at org.apache.paimon.stats.SimpleStatsEvolution$NullCountsEvoArray.getLong(SimpleStatsEvolution.java:215) ~[paimon-flink-1.18-1.4-beta-20260415.042901-14.jar:1.4-beta-SNAPSHOT]
...

This PR fixes this by passing the writeCols on creation.

Tests

See org.apache.paimon.table.system.FilesTableTest

Copy link
Copy Markdown
Contributor

@leaves12138 leaves12138 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The change correctly scopes SimpleStatsEvolution by both schema id and writeCols, so the stats row/array is first interpreted in the data-evolution file's written-column layout and then evolved back to the current table schema. This should avoid the out-of-bounds/null-count mapping issue in the $files table, and the new regression test covers partial-column data-evolution files across schema evolution.

@JingsongLi
Copy link
Copy Markdown
Contributor

+1

@JingsongLi JingsongLi merged commit d121dd3 into apache:master May 11, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants