-
Notifications
You must be signed in to change notification settings - Fork 405
Open
Description
Bug Report
Description
When using update_schema().update_column() to update the required property of list elements, the field type updates successfully but the required property does not change.
Environment
- PyIceberg version: 0.10.0
- Python version: 3.12.12
- Platform: macOS
Root Cause
Located in pyiceberg/table/update/schema.py:112 in the _ApplyChanges.list() method:
return ListType(element_id=list_type.element_id, element=element_type,
element_required=list_type.element_required) # Always uses original valueThe method always uses the original element_required value instead of checking if the element field has been updated in self._updates.
Minimal Reproduction
from pyiceberg.types import ListType, StringType
# Create table with list field where elements are optional (element_required=False)
with iceberg_table.update_schema() as update_schema:
update_schema.add_column(
path="test_list_primitive",
field_type=ListType(element_type=StringType(), element_required=False),
)
# Try to update element to be required
with iceberg_table.update_schema(allow_incompatible_changes=True) as update_schema:
update_schema.update_column(
path=("test_list_primitive", "element"),
required=True, # This doesn't work
)
# Check schema - element_required is still False
field = iceberg_table.schema().find_field("test_list_primitive")
print(field.field_type.element_required) # Still False, should be TrueExpected Behavior
The element_required property should be updated to True.
Actual Behavior
The element_required property remains False (unchanged).
Proposed Fix
def list(self, list_type: ListType) -> ListType:
element_type = self._visit_type(list_type.element_type)
element_required = list_type.element_required
# Check if element field has been updated
if update := self._updates.get(list_type.element_id):
element_required = update.required
return ListType(
element_id=list_type.element_id,
element=element_type,
element_required=element_required
)Workaround
Drop and recreate the list field:
# Step 1: Drop
with iceberg_table.update_schema(allow_incompatible_changes=True) as update_schema:
update_schema.delete_column("test_list_primitive")
# Step 2: Recreate with element_required=True
with iceberg_table.update_schema(allow_incompatible_changes=True) as update_schema:
update_schema.add_column(
path="test_list_primitive",
field_type=ListType(element_type=StringType(), element_required=True),
)Additional Notes
A similar issue likely exists for MapType.value_required in the map() method of the same class.
Metadata
Metadata
Assignees
Labels
No labels