Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compaction must preserve commit timestamps of merged records #376

Closed
bvaradar opened this issue Apr 12, 2018 · 2 comments
Closed

Compaction must preserve commit timestamps of merged records #376

bvaradar opened this issue Apr 12, 2018 · 2 comments

Comments

@bvaradar
Copy link
Contributor

During Compaction, HoodieCompactedRecordScanner merges log records in log files and feeds them to MergeHandle. MergeHandle merges these records with the old columnar data (in base-file). While doing the second level merge, if the mergeHandle chooses the record coming from log, it re-tags hoodie metadata for that record with compaction commit time (using HoodieStorageWriter.writeAvroWithMetadata). For incremental pull, this could cause same update to be returned at different commits. Hence, we need to ensure commit-time for records appended to log-files gets preserved after compaction

@bvaradar
Copy link
Contributor Author

@n3nash @vinothchandar : FYI

@vinothchandar
Copy link
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants