Fix upsert replace

[Jackie-Jiang]

• Fix upsert replace to
  • Correctly replace record location without removing the valid docs from the replaced segment
  • Correctly track and remove the remaining valid docs from the replaced segment
  • Track replaced segment so that the docs do not need to be removed again
• Use segment lock to prevent adding/replacing/removing the same segment at same time which can cause race condition
• Track an unexpected case where the primary keys show up in the wrong segment using meter UPSERT_KEYS_IN_WRONG_SEGMENT

[codecov-commenter]

Codecov Report

Merging #9132 (393ef29) into master (396f37d) will decrease coverage by 5.11%.
The diff coverage is 60.00%.

@@             Coverage Diff              @@
##             master    #9132      +/-   ##
============================================
- Coverage     68.71%   63.60%   -5.12%     
+ Complexity     4740     4734       -6     
============================================
  Files          1838     1791      -47     
  Lines         97436    95357    -2079     
  Branches      14699    14467     -232     
============================================
- Hits          66958    60652    -6306     
- Misses        25757    30326    +4569     
+ Partials       4721     4379     -342     

Flag	Coverage Δ
integration1	?
unittests1	66.98% <60.00%> (+0.04%)	⬆️
unittests2	15.32% <0.00%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ata/manager/realtime/RealtimeTableDataManager.java	11.32% <0.00%> (-29.44%)	⬇️
...apache/pinot/segment/local/utils/SegmentLocks.java	100.00% <ø> (ø)
...t/local/upsert/PartitionUpsertMetadataManager.java	63.50% <59.74%> (+4.01%)	⬆️
...a/org/apache/pinot/common/metrics/ServerMeter.java	98.41% <100.00%> (-1.59%)	⬇️
...va/org/apache/pinot/core/routing/RoutingTable.java	0.00% <0.00%> (-100.00%)	⬇️
...va/org/apache/pinot/common/config/NettyConfig.java	0.00% <0.00%> (-100.00%)	⬇️
...a/org/apache/pinot/common/metrics/MinionMeter.java	0.00% <0.00%> (-100.00%)	⬇️
...g/apache/pinot/common/metrics/ControllerMeter.java	0.00% <0.00%> (-100.00%)	⬇️
.../apache/pinot/common/metrics/BrokerQueryPhase.java	0.00% <0.00%> (-100.00%)	⬇️
.../apache/pinot/common/metrics/MinionQueryPhase.java	0.00% <0.00%> (-100.00%)	⬇️
... and 360 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

[KKcorps]

LGTM!