Skip to content

[WIP] [format] Support narrowing Parquet reads -- do not merge this!#8505

Draft
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:codex/parquet-int64-int32-read-compat
Draft

[WIP] [format] Support narrowing Parquet reads -- do not merge this!#8505
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:codex/parquet-int64-int32-read-compat

Conversation

@leaves12138

Copy link
Copy Markdown
Contributor

Purpose

Some externally produced Parquet files can store values with a wider physical type than the Paimon table schema expects. For example, an INT field may be stored as Parquet INT64 and a FLOAT field may be stored as Parquet DOUBLE. When such columns are dictionary encoded, the vectorized reader currently selects the updater only from the Paimon logical type and calls methods such as decodeToInt, which fails for dictionaries such as PlainLongDictionary.

Changes

  • Use a narrowing updater when reading Paimon INT from Parquet INT64.
  • Use a narrowing updater when reading Paimon FLOAT from Parquet DOUBLE.
  • Keep overflow checks for INT64 -> INT via Math.toIntExact.
  • Add regression tests for INT64 -> INT and DOUBLE -> FLOAT reads.

Tests

  • mvn -pl paimon-format -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=none -Dtest=FileTypeNotMatchReadTypeTest test

@leaves12138 leaves12138 changed the title [format] Support narrowing Parquet reads [WIP] [format] Support narrowing Parquet reads -- do not merge this! Jul 8, 2026
@leaves12138 leaves12138 marked this pull request as draft July 8, 2026 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant