Add table level lock for segment upload #6165
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xiangfu0
force-pushed
the
add_table_lock_for_segment_upload
branch
from
f3a94cc
to
9ba9f4e
Compare
mcvsubbu
reviewed
Oct 20, 2020
...ntroller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
Outdated
Show resolved
Hide resolved
Codecov Report
@@ Coverage Diff @@
## master #6165 +/- ##
==========================================
- Coverage 66.44% 64.05% -2.40%
==========================================
Files 1075 1237 +162
Lines 54773 58482 +3709
Branches 8168 8655 +487
==========================================
+ Hits 36396 37462 +1066
- Misses 15700 18284 +2584
- Partials 2677 2736 +59
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
xiangfu0
force-pushed
the
add_table_lock_for_segment_upload
branch
3 times, most recently
from
8a02f09
to
145a314
Compare
xiangfu0
requested review from
Jackie-Jiang,
kishoreg,
mayankshriv and
siddharthteotia
| @@ -1505,10 +1507,19 @@ public void deleteOfflineTable(String tableName) { | |||
| LOGGER.info("Deleting table {}: Removed table config", offlineTableName); | |||
|
|
|||
| // Remove instance partitions | |||
| final String rawTableName = TableNameBuilder.extractRawTableName(tableName); | |||
There was a problem hiding this comment.
(nit) we don't use final for local variable
...ntroller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
Outdated
Show resolved
Hide resolved
...ntroller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
Outdated
Show resolved
Hide resolved
...ntroller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
Outdated
Show resolved
Hide resolved
...ntroller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
Outdated
Show resolved
Hide resolved
|
Discussed offline and suggested:
|
xiangfu0
force-pushed
the
add_table_lock_for_segment_upload
branch
3 times, most recently
from
1b9d2de
to
a17ea4c
Compare
xiangfu0
force-pushed
the
add_table_lock_for_segment_upload
branch
from
a17ea4c
to
7a10126
Compare
mcvsubbu
reviewed
Oct 22, 2020
pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java
Outdated
Show resolved
Hide resolved
...ntroller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
Outdated
Show resolved
Hide resolved
mcvsubbu
approved these changes
Oct 23, 2020
pinot-spi/src/main/java/org/apache/pinot/spi/utils/retry/RandomDelayRetryPolicy.java
Outdated
Show resolved
Hide resolved
| /** | ||
| * Delay policy with random delay between attempts. | ||
| */ | ||
| public class RandomDelayRetryPolicy extends BaseRetryPolicy { |
There was a problem hiding this comment.
Thanks for adding this. This will be useful in other places as well.
mayankshriv
approved these changes
Oct 23, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add table level lock for segment upload to avoid unnecessary race conditions to cause idealstates update failure when pushing new segments with high parallelism.