Don't truncate min/max for iceberg delete files #12671
Conversation
|
I used metric FULL_METRIC_CONFIG for parquet to allow storing full paths in min/max. It was impossible to do it this way for ORC so I decided to put hard limit there. |
homar
requested review from
findepi and
alexjo2144
and removed request for
findepi
homar
changed the title
homar
changed the title
| case ORC: | ||
| return createOrcWriter(FULL_METRICS_CONFIG, outputPath, POSITION_DELETE_SCHEMA, jobConf, session, storageProperties); | ||
| return createOrcWriter(FULL_METRICS_CONFIG, outputPath, POSITION_DELETE_SCHEMA, jobConf, session, storageProperties, stringStatisticsLimit); |
There was a problem hiding this comment.
We never want to truncate the stats path, right? So could we do
DataSize.ofBytes(Long.MAX_VALUE) here?
There was a problem hiding this comment.
hmm that makes sense, I will change it
There was a problem hiding this comment.
Actually it has to be Integer.MAX_VALUE
| case ORC: | ||
| return createOrcWriter(metricsConfig, outputPath, icebergSchema, jobConf, session, storageProperties); | ||
| return createOrcWriter(metricsConfig, outputPath, icebergSchema, jobConf, session, storageProperties, Optional.empty()); |
There was a problem hiding this comment.
Maybe cleaner to not have the stats limit be optional in createOrcWriter and instead get it from the session here, so we always have a value
| @@ -132,9 +133,9 @@ public IcebergFileWriter createDataFileWriter( | |||
| switch (fileFormat) { | |||
| case PARQUET: | |||
| // TODO use metricsConfig | |||
| return createParquetWriter(outputPath, icebergSchema, jobConf, session, hdfsContext); | |||
| return createParquetWriter(MetricsConfig.getDefault(), outputPath, icebergSchema, jobConf, session, hdfsContext); | |||
There was a problem hiding this comment.
Is there a GH Issue to support metrics mode for Parquet? If not, can you make one
Either way, link it here
homar
force-pushed
the
homar/cleanup_unused_delete_files_during_iceberg_optimize
branch
2 times, most recently
from
6168f68
to
c242315
Compare
homar
force-pushed
the
homar/cleanup_unused_delete_files_during_iceberg_optimize
branch
from
c242315
to
582a1b1
Compare
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergFileWriterFactory.java
Show resolved
Hide resolved
Description
Previously min/max were truncated to very short numbers which may have had very bad impact on performance.
Related issues, pull requests, and links
Documentation
( ) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.
Release notes
( ) No release notes entries required.
( ) Release notes entries required with the following suggested text: