Fix the replication in segment assignment strategy #9816
Conversation
|
@mcvsubbu @sajjad-moradi @jugomezv @snleee : please review |
| @@ -51,6 +51,10 @@ public void init(HelixManager helixManager, TableConfig tableConfig) { | |||
| SegmentsValidationAndRetentionConfig validationAndRetentionConfig = tableConfig.getValidationConfig(); | |||
| Preconditions.checkState(validationAndRetentionConfig != null, "Validation Config is null"); | |||
| _replication = validationAndRetentionConfig.getReplicationNumber(); | |||
| // Number of replicas per partition of low-level consumers check is for the real time tables only | |||
| if (validationAndRetentionConfig.getReplicasPerPartition() != null) { | |||
There was a problem hiding this comment.
We need to check if the table is offline or real-time and then decide where to pull the replication data.
There was a problem hiding this comment.
Is it possible to wrap all the logic for whether to use replication or replicasPerPartition within the TableConfig itself? That way, tableConfig.getReplicationNumber should just return the correct value depending on whether it is offline or realtime table.
There was a problem hiding this comment.
we saw real time tables using balanced strategy in Production. @jugomezv can you please confirm
There was a problem hiding this comment.
@navina We can wire that in table config but it would also mean cleaning up at all the other places we use this. I would so keep that out of scope for this fix
There was a problem hiding this comment.
Please tag this PR with the (open) issue #8804.
Since we have a problem in production, we should fix in some way (add TODOs as needed), but fix the overall replication/replicasPerPartition as a part of that issue.
Codecov Report
@@ Coverage Diff @@
## master #9816 +/- ##
============================================
+ Coverage 60.58% 70.26% +9.68%
+ Complexity 5281 4998 -283
============================================
Files 1949 1964 +15
Lines 104632 105033 +401
Branches 15847 15896 +49
============================================
+ Hits 63389 73803 +10414
+ Misses 36540 26096 -10444
- Partials 4703 5134 +431
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
Can we also add the testing case to cover this? We should add the test where we set |
| @@ -51,6 +52,10 @@ public void init(HelixManager helixManager, TableConfig tableConfig) { | |||
| SegmentsValidationAndRetentionConfig validationAndRetentionConfig = tableConfig.getValidationConfig(); | |||
| Preconditions.checkState(validationAndRetentionConfig != null, "Validation Config is null"); | |||
| _replication = validationAndRetentionConfig.getReplicationNumber(); | |||
| // Number of replicas per partition of low-level consumers check is for the real time tables only | |||
| if (tableConfig.getTableType() == TableType.REALTIME) { | |||
There was a problem hiding this comment.
Let's add the TODO comment that we can clean this up once the table config has the new API that picks up the correct replication depending on the table type.
There was a problem hiding this comment.
added a todo and a unit test.
| _replication = validationAndRetentionConfig.getReplicationNumber(); | ||
| // Number of replicas per partition of low-level consumers check is for the real time tables only | ||
| if (tableConfig.getTableType() == TableType.REALTIME) { | ||
| _replication = validationAndRetentionConfig.getReplicasPerPartitionNumber(); | ||
| } |
There was a problem hiding this comment.
nit:
_replication = tableConfig.getTableType() == TableType.REALTIME
? validationAndRetentionConfig.getReplicasPerPartitionNumber()
: validationAndRetentionConfig.getReplicationNumber();
GSharayu
force-pushed
the
new_master
branch
3 times, most recently
from
9edbf3f
to
ee3aab0
Compare
The Balanced and Replica group segment assignment strategy would assume replication to be fetched from only table config as tableConfig.getValidationConfig().getReplicationNumber() irrespective of whether its a realtime table or offline table
But for real time tables, if we have tableConfig.getValidationConfig().getReplicasPerPartitionNumber(), this should be preferred over table replication
No additional changes are required
Reference Issue: #9309
Also, issue #8804