lower/upper bounds for timestamps are written incorrectly

[alsugiliazova]

Wrong behavior when use_iceberg_partition_pruning is set to 1. On antalya and upstream starting from 25.4 it is 1 by default.

SELECT *
FROM `default.iceberg_5d3aa3af_9a0c_11f0_bf29_e0c26496f172`

   ┌───────────────────datetime─┐
1. │ 2024-12-31 23:00:00.000000 │
   └────────────────────────────┘

SELECT *
FROM `default.iceberg_5d3aa3af_9a0c_11f0_bf29_e0c26496f172`
WHERE datetime = '2024-12-31 23:00:00.000000'

Query id: 097b4ebb-a03a-4156-bf0c-1b9ea0ff55df

Ok.

0 rows in set. Elapsed: 0.014 sec. 

Repro:

<clickhouse>
    <named_collections>
        <s3_conn>
            <access_key_id>admin</access_key_id>
            <secret_access_key>password</secret_access_key>
            <url>http://minio:9000/warehouse/</url>
        </s3_conn>
    </named_collections>
</clickhouse>

CREATE TABLE s3_5d240441_9a0c_11f0_ad18_e0c26496f172 (datetime DateTime)
ENGINE = S3(s3_conn, filename='s3_5d240441_9a0c_11f0_ad18_e0c26496f172', format=Parquet)

INSERT INTO s3_5d240441_9a0c_11f0_ad18_e0c26496f172 VALUES ('2025-01-01 00:00:00')

ice insert default.iceberg_5d3aa3af_9a0c_11f0_bf29_e0c26496f172 -p s3://warehouse/s3_5d240441_9a0c_11f0_ad18_e0c26496f172

Same data insert with PyIceberg works fine.

[alsugiliazova]

Thanks to Misha, we found the issue: the ice tool writes lower_bounds and upper_bounds for timestamps in milliseconds, but the Iceberg spec requires microseconds (UTC). Because of this mismatch, ClickHouse interprets the stats incorrectly and prunes files that actually contain matching rows. PyIceberg writes bounds in microseconds as required, so queries work correctly there.

[subkanthi]

The Metrics object is part of iceberg 1.8.1, there doesnt seem to be any additional logic in ice

[shyiko]

The issue is with us handling TimestampType(isAdjustedToUTC=1, unit=TimeUnit(MILLIS=MilliSeconds(), MICROS=<null>, NANOS=<null>)). Metrics are gathered in milliseconds in this case while timestamps in iceberg (pre v3) are stored in microseconds per https://iceberg.apache.org/spec/#parquet. Basically we cannot use ParquetUtil.footerMetrics(metadata, Stream.empty(), metricsConfig, nameMapping); as-is to create data files when there are timestamp columns with millisecond precision.

[shyiko]

Addressed in 0.8.0.