HIVE-28765: Iceberg: Incorrect partition statistics on time travel + partition evolution - #5748
Conversation
c33a0d6 to
7b1b1b8
Compare
7b1b1b8 to
6caf105
Compare
6caf105 to
01a4117
Compare
…partition evolution
01a4117 to
e11c40d
Compare
| } | ||
| Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable()); | ||
| Snapshot snapshot = IcebergTableUtil.getTableSnapshot(table, hmsTable); | ||
| boolean isTimeTravel = snapshot != null && table.currentSnapshot() != null && |
There was a problem hiding this comment.
@okumin, could we use equals here?
public boolean isPartitioned(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
if (!hmsTable.getTTable().isSetId()) {
return false;
}
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
Snapshot snapshot = IcebergTableUtil.getTableSnapshot(table, hmsTable);
boolean isTimeTravelOrBranch = snapshot != null && snapshot.equals(table.currentSnapshot());
if (isTimeTravelOrBranch && hasUndergonePartitionEvolution(table)) {
return false;
}
return table.spec().isPartitioned();
}
There was a problem hiding this comment.
I agree that it is simpler and tolerant with unexpected snapshots, such as having the same snapshot id but different timestamps.
https://github.com/apache/iceberg/blob/c338323b2e9cd8a862fdf328ceacc1b59ea6275a/core/src/main/java/org/apache/iceberg/BaseSnapshot.java#L320-L332
I updated it. I kept the variable name, isTimeTravel, assuming reading a different snapshot using tag/branch sounds like a time-travel query.
There was a problem hiding this comment.
forked branch could be ahead of main, so I wouldn't call it time-travel
There was a problem hiding this comment.
I grabbed your point. I applied a slight modification to the variable name.
1ac16b2
|
|
@deniskuzZ Thanks for reviewing this pull request! |



What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/HIVE-28765
We use not partition stats but table stats when we access a complicated table with tagging or branching.
Why are the changes needed?
We would like to improve the query plan for a time-travel read with tagging and branching.
Does this PR introduce any user-facing change?
No
Is the change a dependency upgrade?
No
How was this patch tested?
Updated an existing query file.