fix(ilp): fix data loss causes by concurrent column drop transactions #3583
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fuzz test that drops column concurrently with ILP ingestion found a problem. It is a race condition in ILP, which can lead to data loss. More specifically, column name is double-cached in ILP, once for UTF8 and second time UTF16. These maps can diverge and column index begins to return negative column types that ILP doesn't know what to do with.
To help the review, i replaced "if" clause inside loop that is trying to determine if column has be added, with "switch" statement. It was not clear that there are three branches. I feel "switch" makes it easier to understand what's going on. I also renamed
columnIndex
tocolumnWriterIndex
which is what this variable is.Log is below: