Skip to content

TableCreation::format_version is ignored when creating a table using the Iceberg REST catalog interface #2923

Description

@JosephLenton

Apache Iceberg Rust version

None

Describe the bug

When using the Rest catalog, the TableCreation::format_version is silently ignored and never used. The result is you can't set a table to be V3 when using format_version and a REST catalog.

Steps to reproduce

  • Spin up a Rest catalog (say Polaris), and use that.
  • Make a new table.
  • Set it to use format_version(FormatVersion::V3)
  • Then add a V3 column, and try to use the table.
  • Outcome: the table is in V2 format (not v3).

The table building code would be something like this:

let table = TableCreation::builder()
    .name(table_name)
    .schema(table_schema)
    .format_version(FormatVersion::V3) // <-- this call here is ignored
    .build();

Context

In the Rest catalog.rs here the tables .properties are passed, and format_version is never used: https://github.com/apache/iceberg-rust/blob/main/crates/catalog/rest/src/catalog.rs#L794-L815

A fix could be to either add it to the properties here, or add it to the properties upstream.

Workaround

Set the table properties to use V3 by hand like so:

let table = TableCreation::builder()
    .name(table_name)
    .schema(table_schema)
    .partition_spec(table_partition_spec)
    .properties(HashMap::from([
        // Format v3 is needed as we are using nanosecond times.
        (
            TableProperties::PROPERTY_FORMAT_VERSION.to_string(),
            "3".to_string(),
        ),
    ]))
    .build();

To Reproduce

No response

Expected behavior

  • The format version set by format_version should be used, when it is set on it's own.

I'm not 100% sure what should happen when setting the version by both format_version and properties, or if we even care about having precise behaviour there.

Willingness to contribute

I can contribute a fix for this bug independently

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions