-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[HUDI-7921] Making HoodieTable closeable #11494
base: master
Are you sure you want to change the base?
Conversation
throw new HoodieLogCompactException(String.format("Cannot log compact since a compaction instant with greater " | ||
+ "timestamp exists. Instant details %s", compactionInstantWithGreaterTimestamp.get())); | ||
} | ||
try { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NTR: only change in this code block is the closing of HoodieTable.
HoodieTable table = createTable(config, hadoopConf); | ||
resolveWriteConflict(table, metadata, this.pendingInflightAndRequestedInstants); | ||
HoodieTable<?, I, ?, T> table = createTable(config, hadoopConf); | ||
try { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NTR: Even though HoodieTable implements AutoCloseable, I could not use try with resources construct. I wanna throw something specific if close of HoodieTable fails vs something fails within try block. hence had to take this route.
if (shouldComplete && compactionMetadata.getCommitMetadata().isPresent()) { | ||
completeCompaction(compactionMetadata.getCommitMetadata().get(), table, compactionInstantTime); | ||
try { | ||
HoodieTimeline pendingCompactionTimeline = table.getActiveTimeline().filterPendingCompactionTimeline(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NTR: only change in this code block is the closing of HoodieTable.
} else { | ||
throw new HoodieClusteringException("Non clustering replace-commit inflight at timestamp " + clusteringInstant); | ||
try { | ||
HoodieTimeline pendingClusteringTimeline = table.getActiveTimeline().filterPendingReplaceTimeline(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NTR: only change in this code block is the closing of HoodieTable.
table.getMetaClient().reloadActiveTimeline(); | ||
return true; | ||
try { | ||
HoodieTimeline pendingClusteringTimeline = table.getActiveTimeline().filterPendingReplaceTimeline(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NTR: entire set of changes in this patch is about closing HoodieTables
Change Logs
Making HoodieTable Closeable. We have few resources initialized in HoodieTable file FileSysteViewManager and metadata. So introducing close in HoodieTable and closing out all resources instantiated.
Impact
Avoids memory leaks with WriteClients.
Risk level (write none, low medium or high below)
low.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist