You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
During Compaction, HoodieCompactedRecordScanner merges log records in log files and feeds them to MergeHandle. MergeHandle merges these records with the old columnar data (in base-file). While doing the second level merge, if the mergeHandle chooses the record coming from log, it re-tags hoodie metadata for that record with compaction commit time (using HoodieStorageWriter.writeAvroWithMetadata). For incremental pull, this could cause same update to be returned at different commits. Hence, we need to ensure commit-time for records appended to log-files gets preserved after compaction
The text was updated successfully, but these errors were encountered:
During Compaction, HoodieCompactedRecordScanner merges log records in log files and feeds them to MergeHandle. MergeHandle merges these records with the old columnar data (in base-file). While doing the second level merge, if the mergeHandle chooses the record coming from log, it re-tags hoodie metadata for that record with compaction commit time (using HoodieStorageWriter.writeAvroWithMetadata). For incremental pull, this could cause same update to be returned at different commits. Hence, we need to ensure commit-time for records appended to log-files gets preserved after compaction
The text was updated successfully, but these errors were encountered: