Skip to content

fix: table UUID partitions not working - #2916

Open
JosephLenton wants to merge 4 commits into
apache:mainfrom
JosephLenton:fix-uuid-partitions-fail
Open

fix: table UUID partitions not working#2916
JosephLenton wants to merge 4 commits into
apache:mainfrom
JosephLenton:fix-uuid-partitions-fail

Conversation

@JosephLenton

Copy link
Copy Markdown

Which issue does this PR close?

Table Partitions work on writing with UUID values.

What changes are included in this PR?

  • PrimitiveLiteral::UInt128 will now convert to a RawLiteralEnum::String, when the type is UUID.
  • A similar case is added to RawLiteralEnum for UUID deserialisation.

Are these changes tested?

I added:

  • an integration test to confirm the UUID partition works.
  • a second test to confirm writing all of the primitive types work.

@JosephLenton JosephLenton changed the title Fix UUID partitions fail fix: table UUID partitions not working Jul 27, 2026
@JosephLenton

Copy link
Copy Markdown
Author

A few notes ... Avro has an avro::Value::Uuid type so I did look into adding a RawLiteralEnum::Uuid type. Then the types internally could all match up. However this doesn't play well with Serde + apache_avro::to_value. Uuid will only serialise to bytes or a string.

The other thing I found is with the integration tests. I found when I had multiple tests in the same file it would fail with a Reqwest + Tokio error. I believe in opendal-core there is a global Reqwest client, and this will keep sessions alive across multiple tests within the same program (i.e. a single integration test). It doesn't do this if the endpoint changes (this is relevant later).

As a workaround I put my second test into a second file.

I'd be happy to try to solve this in a followup ticket if there is interest, and any other work to make test simpler. The solution is technically straight forward, but would be a chunk of code. We are talking a custom S3 storage layer, changing OpenDal core, or moving to use test-containers (as the endpoint will be different across tests). So I've left that out.

@JosephLenton
JosephLenton force-pushed the fix-uuid-partitions-fail branch from f92395e to de83553 Compare July 27, 2026 19:44

@blackmwk blackmwk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @JosephLenton for this fix!

Comment thread crates/integration_tests/src/lib.rs
Comment thread crates/iceberg/src/spec/values/serde.rs
@JosephLenton
JosephLenton force-pushed the fix-uuid-partitions-fail branch from 3fb8540 to 5f0221f Compare July 29, 2026 11:00
@JosephLenton
JosephLenton force-pushed the fix-uuid-partitions-fail branch from 5f0221f to 6ca9fa2 Compare July 29, 2026 11:01
@JosephLenton

JosephLenton commented Jul 29, 2026

Copy link
Copy Markdown
Author

Hey @blackmwk those changes are done. Please take another look when you have time!

Btw whilst working on this I found a few items which I suspect to be bugs. I can add them as tickets later when I have time:

  • I think UUID fields are missing the metadata ARROW:extension:metadata=arrow.uuid, such as here on my PR: https://github.com/apache/iceberg-rust/pull/2916/changes#diff-78166ca072bfb69d2fbd40e6e7c1bbf337c85e52f14abd742efa6a03f898d8c0R1013
  • You cannot insert a UUID as a string. i.e. INSERT ... VALUES ('aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee') fails. You must use a hex literal instead.
  • I managed to get the error called Result::unwrap()on anErr value: Execution("type mismatch and can't cast to got Utf8 and FixedSizeBinary(16)"), and this has a typo in it (the cast to got ... part).
  • When inserting data into a partitioned table, if the source data is a select statement without a from, then insertion panics. i.e. INSERT INTO my_table SELECT 123, 456; fails if the table has a partition.
  • arrow_schema_to_schema ignores the Arrow extension types, and so turns an Arrow UUID into a FixedSize(16) and not an Iceberg UUID type.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Partitions with type UUID fail to create transactions

2 participants