Skip to content

fix: require pyarrow>=18.0.0 for native UUID type support#3606

Merged
Fokko merged 1 commit into
apache:mainfrom
anxkhn:fix/bump-pyarrow-floor-uuid-support
Jul 6, 2026
Merged

fix: require pyarrow>=18.0.0 for native UUID type support#3606
Fokko merged 1 commit into
apache:mainfrom
anxkhn:fix/bump-pyarrow-floor-uuid-support

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

PyIceberg's PyArrow UUID handling depends on the native UUID APIs pyarrow.uuid()
and pyarrow.UuidType, but the pyarrow, pandas, duckdb, and ray extras
still pin pyarrow>=17.0.0.

  • On write, PyArrowSchemaVisitor.visit_uuid returns pa.uuid()
    (pyiceberg/io/pyarrow.py:806).
  • On read, _ConvertToIceberg.primitive branches on
    isinstance(primitive, pa.UuidType) (pyiceberg/io/pyarrow.py:1497).

Both pyarrow.uuid() and pyarrow.UuidType were introduced in Apache Arrow
18.0.0 (released 2024-10-28) and do not exist in 17.x. So an environment that
resolves pyarrow==17.x under the current floor raises
AttributeError: module 'pyarrow' has no attribute 'uuid' (or 'UuidType') as
soon as a UUID-typed column is written, or an Arrow schema containing a UUID is
converted back to Iceberg.

This usage was added in #2007 ("Fix UUID support") without raising the floor, and
the earlier Bump PyArrow to 18.0.0 change only moved the tested/locked version,
not the declared lower bound. This PR bumps the floor to pyarrow>=18.0.0 for the
four affected extras so the declared dependency matches what the code actually
requires. See also #1174 (Minimum required pyarrow version).

Are these changes tested?

This is a dependency-floor bump with no code change, so the real guard is CI
resolving the raised floor across the supported Python matrix; uv lock --check
confirms uv.lock stays in sync with pyproject.toml.

Locally:

  • uv lock --check passes (lock in sync).
  • make lint (ruff, ruff-format, mypy, license, uv-lock, ...) all pass.
  • Existing UUID coverage still passes and exercises both affected code paths:
    tests/io/test_pyarrow_visitor.py (60 passed),
    tests/utils/test_schema_conversion.py -k UUID (2 passed), and the UUID cases in
    tests/test_conversions.py (2 passed). A manual round-trip
    UUIDType -> pa.uuid() -> pa.UuidType -> UUIDType runs clean.

No new test is added because there is no new code path to cover: the existing UUID
conversion tests already exercise visit_uuid and _ConvertToIceberg, and the
floor bump only makes the declared dependency honest. A heavier alternative would
be to version-gate pa.uuid()/pa.UuidType at runtime (as done for
AzureFileSystem via MIN_PYARROW_VERSION_SUPPORTING_AZURE_FS in
pyiceberg/io/pyarrow.py), but that keeps UUID unsupported on 17.x rather than
requiring the version the feature needs; happy to go that route instead if
preferred.

Are there any user-facing changes?

Yes, indirectly: the minimum supported pyarrow for the pyarrow, pandas,
duckdb, and ray extras rises from 17.0.0 to 18.0.0. This only affects
environments currently resolving pyarrow 17.x, where UUID columns are already
broken; it prevents that unsupported install rather than changing any working
behavior. (Adding the changelog label if this warrants a changelog entry - I
don't have permission to set labels as an external contributor.)

PyIceberg's PyArrow UUID handling relies on APIs that only exist in
Apache Arrow 18.0.0 and later, but the pyarrow, pandas, duckdb, and ray
extras still pin pyarrow>=17.0.0.

On write, PyArrowSchemaVisitor.visit_uuid returns pa.uuid(); on read,
_ConvertToIceberg branches on isinstance(primitive, pa.UuidType). Both
pyarrow.uuid() and pyarrow.UuidType were introduced in pyarrow 18.0.0
and do not exist on 17.x, so an install that resolves pyarrow==17.x
raises AttributeError as soon as a UUID-typed column is written or an
Arrow schema containing a UUID is converted back to Iceberg.

Bump the floor to pyarrow>=18.0.0 for the affected extras so the
declared dependency matches what the code actually requires, and keep
uv.lock in sync.

@rambleraptor rambleraptor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like a proper change to make. The alternative is to check pyarrow's version on UUID columns + send a proper error message there.

I don't like that alternative, since runtime errors are never ideal.

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is reasonable. You could always override to Arrow 17, but then you're on your own.

@Fokko Fokko merged commit 1a87aa3 into apache:main Jul 6, 2026
17 checks passed
@Fokko

Fokko commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Thanks @anxkhn and thanks @rambleraptor for the review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants