Add support for partition evolution in Iceberg.

[bitsondatadev]

Now that we've updated to 0.11.0 for Iceberg, we should consider adding support for partition evolution. I spoke with @phd3 and he said there may be some support for reading partition evolved tables but currently we don't support the DDL for updating the table properties after creation.

Docs: https://iceberg.apache.org/evolution/#partition-evolution

[bitsondatadev]

One thing we'll need to discuss is adding syntax for ALTER TABLE.

Looking at the postgresql syntax for CREATE TABLE and ALTER TABLE, we would likely want to add SET ( storage_parameter [= value] [, ... ] ) and RESET ( storage_parameter [, ... ] ).

For this table creation

CREATE TABLE distributors (
    did     integer,
    name    varchar(40),
    UNIQUE(name) WITH (fillfactor=70)
)
WITH (
 location='s3a://a/b/c/d',
 partitioning = ARRAY['identity(name)']
);

We could update with

ALTER TABLE distributors SET ( partitioning = ARRAY['day(name)', 'bucket(did, 64)']);
ALTER TABLE distributors RESET (location);

[tuppimax]

@bitsondatadev @phd3 when you say "there may be some support for reading partition evolved tables" are there any known issues with reading partition evolved tables ? I would be happy help out.

We use Trino only reads while all the writes happen through Spark. Trying to understand the gaps which affect our use case.

Thanks in advance

[phd3]

@tuppimax Currently, there isn’t test coverage around tables with partition spec evolution, so adding them would give us confidence on the read support. However, Trino uses iceberg library itself to generate splits based on provided filters, so it should already be partition spec aware. In addition, partition pruning optimization and delete operation requires that the filters match entire partitions. This code is also aware of partition spec evolution currently. But need to have tests to make sure everything works.

[phd3]

The tests would need to use Spark to evolve the spec (See io.trino.tests.iceberg.TestSparkCompatibility), since Trino doesn’t have support right now. IIRC the spec evolution syntax requires iceberg-spark3-runtime version 0.11.0, if so we need another release of trinodb/docker-images to have that.

[Kyo91]

@tuppimax See #8284 for an example of an issue with reading partition evolved tables.

[alexjo2144]

Sorry, didn't see this older thread so I made a duplicate #12174 but updating the partitioning on a table using ALTER TABLE ... SET PROPERTIES is in this PR: #12259