Fail to read V1 Iceberg Spec

[yingying-chen-cko]

Apache Iceberg version

0.9.0 (latest release)

Please describe the bug 🐞

I have an Iceberg V1 table and I am trying to load the table using pyiceberg.table.StaticTable:

from pyiceberg.table import StaticTable

t = StaticTable.from_metadata(
    "gs://<project_id>/<table-name>/metadata/v1.metadata.json"
)

When I run t.inspect.manifests(), it gives the following error

ResolveError: 504: added_files_count: required int is non-optional, and not part of the file schema

I believe this is because in pyiceberg.manifest.DEFAULT_READ_VERSION is set to 2 but my table is V1. So I patch this to manifest.DEFAULT_READ_VERSION = 1 and this gives me another error:

AttributeError: 'pyiceberg.manifest.ManifestFile' object has no attribute 'content'

I managed to resolve this error temporary by adding the content attribute to pyiceberg.manifest.MANIFEST_LIST_FILE_SCHEMAS[1]. And more errors are raised as I keep resolving:

For pyiceberg.manifest.MANIFEST_ENTRY_SCHEMAS[1]:

AttributeError: 'pyiceberg.manifest.ManifestEntry' object has no attribute 'sequence_number'

When running t.inspect.files(), there an error generated from pyiceberg.manifest.DATA_FILE_TYPE[1]:

AttributeError: 'pyiceberg.manifest.DataFile' object has no attribute 'content'

I am able to load the table after adding all the above missing attributes, but is there a way to parse V1 table or is this a bug when loading V1 table?

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[Fokko]

@yingying-chen-cko Thanks for raising this, and this looks like a bug indeed.

I believe this is because in pyiceberg.manifest.DEFAULT_READ_VERSION is set to 2

This is intentional since we handle all the metadata internally as V2. This also explains the error that it can't find content, since we set it to DATA for all the V1 entries.

Could you share the full stacktrace?

[yingying-chen-cko]

No problem, this is the stacktrace when I run t.inspect.manifests()

---------------------------------------------------------------------------
ResolveError                              Traceback (most recent call last)
Cell In[2], line 1
----> 1 m = t.inspect.manifests()
      2 mf = t.current_snapshot().manifests(t.io)[0]
      3 mfe = mf.fetch_manifest_entry(t.io)

File ~/cdp/adhoc_scripts/iceberg/.venv/lib/python3.13/site-packages/pyiceberg/table/inspect.py:457, in InspectTable.manifests(self)
    456 def manifests(self) -> "pa.Table":
--> 457     return self._generate_manifests_table(self.tbl.current_snapshot())

File ~/cdp/adhoc_scripts/iceberg/.venv/lib/python3.13/site-packages/pyiceberg/table/inspect.py:428, in InspectTable._generate_manifests_table(self, snapshot, is_all_manifests_table)
    426 manifests = []
    427 if snapshot:
--> 428     for manifest in snapshot.manifests(self.tbl.io):
    429         is_data_file = manifest.content == ManifestContent.DATA
    430         is_delete_file = manifest.content == ManifestContent.DELETES

File ~/cdp/adhoc_scripts/iceberg/.venv/lib/python3.13/site-packages/pyiceberg/table/snapshots.py:257, in Snapshot.manifests(self, io)
    255 def manifests(self, io: FileIO) -> List[ManifestFile]:
    256     """Return the manifests for the given snapshot."""
--> 257     return list(_manifests(io, self.manifest_list))

File ~/cdp/adhoc_scripts/iceberg/.venv/lib/python3.13/site-packages/cachetools/_decorators.py:117, in _cached_unlocked.<locals>.wrapper(*args, **kwargs)
    115 except KeyError:
    116     pass  # key not found
--> 117 v = func(*args, **kwargs)
    118 try:
    119     cache[k] = v

File ~/cdp/adhoc_scripts/iceberg/.venv/lib/python3.13/site-packages/pyiceberg/manifest.py:644, in _manifests(io, manifest_list)
    642 """Read and cache manifests from the given manifest list, returning a tuple to prevent modification."""
    643 file = io.new_input(manifest_list)
--> 644 return tuple(read_manifest_list(file))

File ~/cdp/adhoc_scripts/iceberg/.venv/lib/python3.13/site-packages/pyiceberg/manifest.py:657, in read_manifest_list(input_file)
    647 def read_manifest_list(input_file: InputFile) -> Iterator[ManifestFile]:
    648     """
    649     Read the manifests from the manifest list.
    650 
   (...)    655         An iterator of ManifestFiles that are part of the list.
    656     """
--> 657     with AvroFile[ManifestFile](
    658         input_file,
    659         MANIFEST_LIST_FILE_SCHEMAS[DEFAULT_READ_VERSION],
    660         read_types={-1: ManifestFile, 508: PartitionFieldSummary},
    661         read_enums={517: ManifestContent},
    662     ) as reader:
    663         yield from reader

File ~/cdp/adhoc_scripts/iceberg/.venv/lib/python3.13/site-packages/pyiceberg/avro/file.py:177, in AvroFile.__enter__(self)
    174 if not self.read_schema:
    175     self.read_schema = self.schema
--> 177 self.reader = resolve_reader(self.schema, self.read_schema, self.read_types, self.read_enums)
    179 return self

File ~/cdp/adhoc_scripts/iceberg/.venv/lib/python3.13/site-packages/pyiceberg/avro/resolver.py:235, in resolve_reader(file_schema, read_schema, read_types, read_enums)
    218 def resolve_reader(
    219     file_schema: Union[Schema, IcebergType],
    220     read_schema: Union[Schema, IcebergType],
    221     read_types: Dict[int, Callable[..., StructProtocol]] = EMPTY_DICT,
    222     read_enums: Dict[int, Callable[..., Enum]] = EMPTY_DICT,
    223 ) -> Reader:
    224     """Resolve the file and read schema to produce a reader.
    225 
    226     Args:
   (...)    233         NotImplementedError: If attempting to resolve an unrecognized object type.
    234     """
--> 235     return visit_with_partner(file_schema, read_schema, ReadSchemaResolver(read_types, read_enums), SchemaPartnerAccessor())

File /opt/homebrew/Cellar/python@3.13/3.13.3/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934, in singledispatch.<locals>.wrapper(*args, **kw)
    931 if not args:
    932     raise TypeError(f'{funcname} requires at least '
    933                     '1 positional argument')
--> 934 return dispatch(args[0].__class__)(*args, **kw)

File ~/cdp/adhoc_scripts/iceberg/.venv/lib/python3.13/site-packages/pyiceberg/schema.py:626, in _(schema, partner, visitor, accessor)
    623 @visit_with_partner.register(Schema)
    624 def _(schema: Schema, partner: P, visitor: SchemaWithPartnerVisitor[P, T], accessor: PartnerAccessor[P]) -> T:
    625     struct_partner = accessor.schema_partner(partner)
--> 626     return visitor.schema(schema, partner, visit_with_partner(schema.as_struct(), struct_partner, visitor, accessor))

File /opt/homebrew/Cellar/python@3.13/3.13.3/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934, in singledispatch.<locals>.wrapper(*args, **kw)
    931 if not args:
    932     raise TypeError(f'{funcname} requires at least '
    933                     '1 positional argument')
--> 934 return dispatch(args[0].__class__)(*args, **kw)

File ~/cdp/adhoc_scripts/iceberg/.venv/lib/python3.13/site-packages/pyiceberg/schema.py:641, in _(struct, partner, visitor, accessor)
    638     finally:
    639         visitor.after_field(field, field_partner)
--> 641 return visitor.struct(struct, partner, field_results)

File ~/cdp/adhoc_scripts/iceberg/.venv/lib/python3.13/site-packages/pyiceberg/avro/resolver.py:399, in ReadSchemaResolver.struct(self, struct, expected_struct, field_readers)
    397     results.append((pos, DefaultReader(read_field.initial_default)))
    398 elif read_field.required:
--> 399     raise ResolveError(f"{read_field} is non-optional, and not part of the file schema")
    400 else:
    401     # Just set the new field to None
    402     results.append((pos, NoneReader()))

ResolveError: 504: added_files_count: required int is non-optional, and not part of the file schema

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'