[lake/tiering] add table dropped handling and nullable write results#2920
[lake/tiering] add table dropped handling and nullable write results#2920beryllw wants to merge 2 commits intoapache:mainfrom
Conversation
c74237d to
ef27034
Compare
ef27034 to
2e0a904
Compare
leekeiabstraction
left a comment
There was a problem hiding this comment.
TY for the PR, left a comment.
| */ | ||
| public class TableBucketWriteResult<WriteResult> implements Serializable { | ||
|
|
||
| private static final long serialVersionUID = 1L; |
There was a problem hiding this comment.
As canceled field is added, shouldn't the serialVersionUID be bumped as well? Similarly the serialiser needs to handle each version of TableBucketWriteResult differently.
luoyuxia
left a comment
There was a problem hiding this comment.
Haven't finish review, but left some comments
| return forceCompleteDroppedTable(); | ||
| } | ||
|
|
||
| checkSplitOrStartNext(); |
There was a problem hiding this comment.
if move to a split, and then the split is for the table dropped, will getOrMoveToTable cause exception and then fail the job. Maybe we can protect this case .
| ScanRecords scanRecords; | ||
| try { | ||
| scanRecords = currentLogScanner.poll(pollTimeout); | ||
| } catch (TableNotExistException e) { |
There was a problem hiding this comment.
IIUC, we don't need to check this exception, just wait it to be forced complete in next fetch.
Also, as disscuss before, poll won't throw TableNotExistException.
Purpose
Linked issue: close #2498
This PR improves Lake Tiering's handling of dropped tables and adds cancellation support for tiering operations.
Brief change log
Dropped Table Handling
in-progress splits with empty results instead of failing
Cancellation Support
Tests
API and Format
Documentation