DateTime64(3) turns into DateTime64(3, UTC) with native parquet reader v3

[Selfeer]

Company or project name

No response

Describe what's wrong

I have a parquet file with:

physical_type: INT64
logical_type: Timestamp(isAdjustedToUTC=false, timeUnit=milliseconds, is_from_converted_type=false, force_set_converted_type=false)

There is a mismatch when I read it with and without input_format_parquet_use_native_reader_v3

input_format_parquet_use_native_reader_v3=0

DESCRIBE TABLE file('standalone_timestamp.parquet', 'Parquet')
SETTINGS input_format_parquet_use_native_reader_v3 = 0
   ┌─name─┬─type────────────────────┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐
1. │ ts   │ Nullable(DateTime64(3)) │              │                    │         │                  │                │
   └──────┴─────────────────────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘

input_format_parquet_use_native_reader_v3=1

DESCRIBE TABLE file('standalone_timestamp.parquet', 'Parquet')
SETTINGS input_format_parquet_use_native_reader_v3 = 1
   ┌─name─┬─type───────────────────────────┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐
1. │ ts   │ Nullable(DateTime64(3, 'UTC')) │              │                    │         │                  │                │
   └──────┴────────────────────────────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘

Does it reproduce on the most recent release?

Yes

How to reproduce

ClickHouse version: 25.8.4.13
Settings: input_format_parquet_use_native_reader_v3=1

I wasn't able to get steps strictly with ClickHouse but here I could generate a repro file with pyarrow:

import pyarrow as pa
import pyarrow.parquet as pq
import datetime
import os

OUT_PATH = "test_decimal/standalone_timestamp.parquet"
os.makedirs(os.path.dirname(OUT_PATH), exist_ok=True)

timestamp_type = pa.timestamp('ms', tz=None)

values = [datetime.datetime(2023, 1, 1, 12, 0, 0), None, datetime.datetime(2023, 1, 2, 13, 0, 0)]
arr = pa.array(values, type=timestamp_type)
table = pa.Table.from_arrays([arr], schema=pa.schema([pa.field("ts", timestamp_type, nullable=True)]))

pq.write_table(table, OUT_PATH, version="1.0", compression=None, use_dictionary=False)

Expected behavior

No response

Error message and/or stacktrace

No response

Additional context

No response

[al13n321]

AFAIU, neither behavior is actually correct, and the new behavior seems less incorrect.

https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#local-semantics-timestamps-not-normalized-to-utc says:

A TIMESTAMP with isAdjustedToUTC=false represents year, month, day, hour, minute, second and subsecond fields in a local timezone, regardless of what specific time zone is considered local.

ClickHouse doesn't have a data type like that. Both DateTime64(3) and DateTime64(3, 'UTC') are the number of milliseconds since Unix epoch, representing a specific point in time. The timezone parameter only affects parsing and formatting.

DateTime64(3, 'UTC') seems more correct: if converted to string, it'll show the correct "year, month, day, hour, minute, second and subsecond" that the parquet file had in mind. Like in this paragraph in parquet documentation:

Another way to get to the same definition is to treat the local timestamp value as if it were in UTC and store it as an instant. For example, if we treat the local timestamp 1970-01-03 00:00:00 as if it were the instant 1970-01-03 00:00:00 UTC, we can store it as 172800000. When reading 172800000 back, we can reconstruct the instant 1970-01-03 00:00:00 UTC and convert it to a local timestamp as if we were in the UTC time zone, resulting in 1970-01-03 00:00:00. In Java, this can be achieved by calling LocalDateTime.ofEpochSecond(172800, 0, ZoneOffset.UTC).

#87872 makes DateTime64(3, 'UTC') the default behavior for both readers and adds setting input_format_parquet_local_time_as_utc=0 to go back to DateTime64(3).