[Python][C++] Rowgroup statistics for pd.NaT array ill defined

[asfimport]

When initialising an array with NaT only values the row group statistic is corrupt returning either random values or raises integer out of bound exceptions.

import io
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

df = pd.DataFrame({"t": pd.Series([pd.NaT], dtype="datetime64[ns]")})
buf = pa.BufferOutputStream()
pq.write_table(pa.Table.from_pandas(df), buf, version="2.0")
buf = io.BytesIO(buf.getvalue().to_pybytes())
parquet_file = pq.ParquetFile(buf)
# Asserting behaviour is difficult since it is random and the state is ill defined. 
# After a few iterations an exception is raised.
while True:
    parquet_file.metadata.row_group(0).column(0).statistics.max

Reporter: Florian Jetter / @fjetter
Assignee: Uwe Korn / @xhochy

PRs and other links:

• GitHub Pull Request #5180
• GitHub Pull Request #5403

_{Note: This issue was originally created as ARROW-6339. Please see the migration documentation for further details.}

[asfimport]

Florian Jetter / @fjetter:
The same is true for other null values, e.g.

df = pd.DataFrame({
    "t": pd.Series([pd.NaT], dtype="datetime64[ns]"),
    "f": [pd.np.nan],
    "o": [None],
})

The statistics are mostly initialised to zero but not all the times.

[asfimport]

Antoine Pitrou / @pitrou:
@jorisvandenbossche

[asfimport]

Uwe Korn / @xhochy:
The problem here is that parquet_file.metadata.row_group(0).column(0).statistics.has_min_max is False and thus .max should never be accessed. Instead of returning undefined data, we should raise an exception.

[asfimport]

Krisztian Szucs / @kszucs:
Issue resolved by pull request 5403
#5403