Skip to content

[format] Follow Parquet list backward-compatibility rules in schema clipping #9550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
juntaozhang wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[format] Follow Parquet list backward-compatibility rules in schema clipping #9550

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetReaderFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@

import static org.apache.paimon.data.columnar.ColumnVectorUtils.createParquetWritableColumnVector;
import static org.apache.paimon.format.parquet.ParquetSchemaConverter.PAIMON_SCHEMA;
import static org.apache.paimon.format.parquet.ParquetSchemaConverter.isThreeLevelList;
import static org.apache.paimon.format.parquet.ParquetSchemaConverter.parquetListElementType;
import static org.apache.paimon.format.parquet.ParquetSchemaConverter.parquetMapKeyValueType;
import static org.apache.paimon.format.parquet.reader.ParquetReaderUtil.buildFieldsList;
Expand Down Expand Up @@ -336,11 +337,10 @@ private Type clipParquetType(DataType readType, Type parquetType) {
// There are two representations for array type in parquet.
// See link:
// https://impala.apache.org/docs/build/html/topics/impala_parquet_array_resolution.html.
int level = arrayGroup.getType(0) instanceof GroupType ? 3 : 2;
Type elementType =
clipParquetType(elementReadType, parquetListElementType(arrayGroup));

if (level == 3) {
if (isThreeLevelList(arrayGroup)) {

@JingsongLi JingsongLi Sep 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve the original Rule 2 interpretation after clipping

Rule 2 is identified from the original two-field repeated group here, but after recursive projection the rebuilt requested schema may contain only one field. ParquetReaderUtil.getArrayElementColumn then reclassifies that clipped group as a Rule 5 wrapper and unwraps it. For LIST<STRUCT<x,y>> read as ARRAY<ROW>, reader construction pairs the RowType with a PrimitiveColumnIO and throws ClassCastException. Please carry the original list interpretation into field construction, or otherwise keep Rule 2 unambiguous after projection, and add a one-field projection read test.

// In case that the name in middle level is not "list".
Type groupMiddle =
new GroupType(
Expand Down
112 changes: 101 additions & 11 deletions paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSchemaConverter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@
import java.util.List;
import java.util.stream.Collectors;

import static org.apache.paimon.utils.Preconditions.checkArgument;
import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY;
import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT32;
import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT64;
Expand All @@ -63,6 +64,8 @@ public class ParquetSchemaConverter {
public static final String MAP_KEY_NAME = "key";
public static final String MAP_VALUE_NAME = "value";
public static final String LIST_ELEMENT_NAME = "element";
private static final String LIST_WRAPPER_NAME = "list";
private static final String LEGACY_LIST_ARRAY_NAME = "array";

/**
* Whether {@code type} is an unsigned integer. Such a column stores a signed value whose bits
Expand Down Expand Up @@ -490,20 +493,107 @@ public static DataField convertToPaimonField(Type parquetType) {
return new DataField(parquetType.getId().intValue(), parquetType.getName(), paimonDataType);
}

/** Returns true if the given group is annotated as a Parquet LIST logical type. */
public static boolean isList(GroupType listType) {
return listType.getLogicalTypeAnnotation()
instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation;
}

/**
* Returns true if the given group is a three-level Parquet list.
*
* <p>In a three-level list the immediate repeated child is a wrapper group whose single
* non-repeated child is the actual element type. This covers the canonical layout ({@code list
* -> element}) as well as legacy wrappers such as Hive's {@code bag} layout.
*
* <p>This corresponds to the Parquet spec's backward-compatibility <b>Rule 5</b>: a repeated
* group that contains exactly one non-repeated child is a wrapper, unless it matches one of
* Rules 1-4.
*
* <p>The compatibility encodings that are <em>not</em> three-level are:
*
* <ul>
* <li><b>Rule 1</b>: the repeated field is a primitive and is itself the element type.
* <li><b>Rule 2</b>: the repeated field is a group with multiple fields and is itself the
* element type.
* <li><b>Rule 3</b>: the repeated field is a group whose single child is also repeated; the
* group itself is the element type.
* <li><b>Rule 4</b>: the repeated field is a group named {@code "array"} or {@code
* "<list>_tuple"} with a single child; the group itself is the element type.
* </ul>
*
* <p>See the Parquet spec: <a
* href="https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules">LogicalTypes#Backward-compatibility-rules</a>
*/
public static boolean isThreeLevelList(GroupType listType) {
if (!isList(listType)) {
return false;
}

// A list must have exactly one repeated child (the middle level).
if (listType.getFieldCount() != 1) {
return false;
}
Type middle = listType.getType(0);
if (middle.isPrimitive() || middle.getRepetition() != Type.Repetition.REPEATED) {
return false;
}
GroupType repeatedGroup = middle.asGroupType();

// Rule 5: the repeated group is a wrapper containing exactly one non-repeated child.
if (repeatedGroup.getFieldCount() != 1
|| repeatedGroup.getType(0).getRepetition() == Type.Repetition.REPEATED) {

@JingsongLi JingsongLi Sep 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Complete Rule 3 handling beyond element selection

This correctly keeps the repeated group as the element, but downstream schema conversion still treats its repeated child as a scalar. With the Rule 3 shape added in the tests and two inner values [8, 9], the inferred type returns only 8; using the semantically correct nested array type instead fails during clipping. Please convert the nested repeated child as an array and align clipping and ColumnIO traversal, with an end-to-end multi-value read test.

return false;
}

// Rule 4: legacy "array" and "<list>_tuple" encodings are not wrappers; the repeated
// group itself is the element type.
return !LEGACY_LIST_ARRAY_NAME.equals(repeatedGroup.getName())
&& !(listType.getName() + "_tuple").equals(repeatedGroup.getName());
}

/**
* Returns true if the given group follows the canonical three-level Parquet list layout ({@code
* list -> element}).
*
* <p>The canonical layout is described in the Parquet spec: <a
* href="https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists">LogicalTypes#Lists</a>
*/
public static boolean isCanonicalList(Type type) {
if (type.isPrimitive()) {
return false;
}

GroupType listGroup = type.asGroupType();
if (!isThreeLevelList(listGroup)) {
return false;
}

Type middle = listGroup.getType(0);
Type element = middle.asGroupType().getType(0);
return LIST_WRAPPER_NAME.equals(middle.getName())
&& LIST_ELEMENT_NAME.equals(element.getName());
}

/**
* Returns the element type of the given LIST-annotated group according to the Parquet spec's
* backward-compatibility rules for lists.
*
* <p>For a three-level list (Rule 5) the returned type is the single child of the repeated
* wrapper. For Rules 1-4 the repeated field itself is returned because it is the element type.
*/
public static Type parquetListElementType(GroupType listType) {
int level = listType.getType(0) instanceof GroupType ? 3 : 2;
if (level == 3) {
// Level 3 representation of list type.
// List type should only have one middle group type, which is repeated, and one element
// type, which is optional.
checkArgument(
listType.getLogicalTypeAnnotation()
instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation,
"Expected LIST-annotated group but got: %s",
listType);

if (isThreeLevelList(listType)) {
return listType.getType(0).asGroupType().getType(0);
} else if (level == 2) {
// Level 2 representation of list type
return listType.getType(0);
} else {
throw new UnsupportedOperationException(
"Parquet list type only have two level representation and three level representation.");
}

return listType.getType(0);

@JingsongLi JingsongLi Sep 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve required element nullability for Rules 1-4

The returned element node still has REPEATED repetition, while convertToPaimonField only applies notNull() to REQUIRED nodes. As a result, Rules 1-4 are converted to nullable Paimon elements even though the Parquet compatibility rules define them as required. Please force the converted element DataType to non-null for non-Rule-5 layouts, without changing the physical repeated Type used by clipping, and add conversion-level nullability assertions.

}

public static Pair<Type, Type> parquetMapKeyValueType(GroupType mapType) {
Expand Down
47 changes: 1 addition & 46 deletions paimon-format/src/main/java/org/apache/paimon/format/parquet/VariantShreddingTypePruner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@
import org.apache.paimon.types.RowType;

import org.apache.parquet.schema.GroupType;
import org.apache.parquet.schema.LogicalTypeAnnotation;
import org.apache.parquet.schema.Type;

import javax.annotation.Nullable;
Expand All @@ -38,6 +37,7 @@
import java.util.Map;
import java.util.Set;

import static org.apache.paimon.format.parquet.ParquetSchemaConverter.isCanonicalList;
import static org.apache.paimon.format.parquet.ParquetSchemaConverter.parquetListElementType;
import static org.apache.paimon.utils.Preconditions.checkArgument;

Expand All @@ -53,9 +53,6 @@
* objectSchemaMap}.
*/
public class VariantShreddingTypePruner {
private static final String LIST_WRAPPER_NAME = "list";
private static final String LIST_ELEMENT_NAME = "element";

@Nullable private final PathNode root;

VariantShreddingTypePruner(RowType variantRowType) {
Expand Down Expand Up @@ -206,48 +203,6 @@ private GroupType clipListShreddingRow(GroupType group, PathNode node, List<Type
return group.withNewFields(newFields);
}

/**
* Returns true if the given group follows the canonical three-level Parquet list layout.
*
* <p>The canonical layout is described in the Parquet spec: <a
* href="https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists">LogicalTypes#Lists</a>
*/
private static boolean isCanonicalList(Type type) {
if (type.isPrimitive()) {
return false;
}

GroupType listGroup = type.asGroupType();
// 1. Must be a LIST logical type.
if (!(listGroup.getLogicalTypeAnnotation()
instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation)) {
return false;
}

// 2. LIST group must have exactly one child named "list".
if (listGroup.getFieldCount() != 1) {
return false;
}
Type middle = listGroup.getType(0);
if (!LIST_WRAPPER_NAME.equals(middle.getName())) {
return false;
}

// 3. The child must be a repeated group.
if (middle.isPrimitive() || middle.getRepetition() != Type.Repetition.REPEATED) {
return false;
}
GroupType repeatedWrapper = middle.asGroupType();

// 4. The repeated wrapper must contain exactly one child named "element".
if (repeatedWrapper.getFieldCount() != 1) {
return false;
}

Type element = repeatedWrapper.getType(0);
return LIST_ELEMENT_NAME.equals(element.getName());
}

/** Returns true if the given group is a plain struct (not a Parquet list or map). */
private static boolean isObjectGroup(Type type) {
if (type.isPrimitive()) {
Expand Down
Loading
Loading