Skip to content

update_schema().update_column() does not update required property for list elements #2786

@mukul-mpac

Description

@mukul-mpac

Bug Report

Description

When using update_schema().update_column() to update the required property of list elements, the field type updates successfully but the required property does not change.

Environment

  • PyIceberg version: 0.10.0
  • Python version: 3.12.12
  • Platform: macOS

Root Cause

Located in pyiceberg/table/update/schema.py:112 in the _ApplyChanges.list() method:

return ListType(element_id=list_type.element_id, element=element_type, 
                element_required=list_type.element_required)  # Always uses original value

The method always uses the original element_required value instead of checking if the element field has been updated in self._updates.

Minimal Reproduction

from pyiceberg.types import ListType, StringType

# Create table with list field where elements are optional (element_required=False)
with iceberg_table.update_schema() as update_schema:
    update_schema.add_column(
        path="test_list_primitive",
        field_type=ListType(element_type=StringType(), element_required=False),
    )

# Try to update element to be required
with iceberg_table.update_schema(allow_incompatible_changes=True) as update_schema:
    update_schema.update_column(
        path=("test_list_primitive", "element"),
        required=True,  # This doesn't work
    )

# Check schema - element_required is still False
field = iceberg_table.schema().find_field("test_list_primitive")
print(field.field_type.element_required)  # Still False, should be True

Expected Behavior

The element_required property should be updated to True.

Actual Behavior

The element_required property remains False (unchanged).

Proposed Fix

def list(self, list_type: ListType) -> ListType:
    element_type = self._visit_type(list_type.element_type)
    element_required = list_type.element_required
    
    # Check if element field has been updated
    if update := self._updates.get(list_type.element_id):
        element_required = update.required
    
    return ListType(
        element_id=list_type.element_id,
        element=element_type,
        element_required=element_required
    )

Workaround

Drop and recreate the list field:

# Step 1: Drop
with iceberg_table.update_schema(allow_incompatible_changes=True) as update_schema:
    update_schema.delete_column("test_list_primitive")

# Step 2: Recreate with element_required=True
with iceberg_table.update_schema(allow_incompatible_changes=True) as update_schema:
    update_schema.add_column(
        path="test_list_primitive",
        field_type=ListType(element_type=StringType(), element_required=True),
    )

Additional Notes

A similar issue likely exists for MapType.value_required in the map() method of the same class.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions