Tiered storage

[npawar]

Description

Issue: #5553
Tiered storage support in Pinot - Phase 1.
This phase supports default tag based instance assignments only, which are not persisted in zk.

Phase 2 will introduce instanceAssignmentConfig for tiers, which will allow us to support replica groups for tiers and also let us persist the InstancePartitions

Example:
This example show how to configure segments older than 15 days move to tier_b_OFFLINE and segments older than 7 days move to tier_a_OFFLINE

{
  "tableName": "myTable",
  "tableType": ...,
  ...
  "tierConfigs": [{
    "name": "tierA",
    "segmentSelectorType": "time",
    "segmentAge": "7d",
    "storageType": "pinot_server",
    "serverTag": "tier_a_OFFLINE"
  }, {
    "name": "tierB",
    "segmentSelectorType": "TIME",
    "segmentAge": "15d",
    "storageType": "PINOT_SERVER",
    "serverTag": "tier_b_OFFLINE"
  }] 
}

Release Notes

Tiered storage phase 1 - default tag based instance assignment for tiers

[Jackie-Jiang]

Well done.
Most comments are minor (comments in OfflineSegmentAssignment also applies to RealtimeSegmentAssignment).
Please merge the 2 relocators into one.

[npawar]

Well done.
Most comments are minor (comments in OfflineSegmentAssignment also applies to RealtimeSegmentAssignment).
Please merge the 2 relocators into one.

Thanks for the review. I've merged the 2 relocators into SegmentRelocator. I introduced controller confs for controller.segment.relocator.frequency, and deprecated the confs for .controller.realtime.segment.relocator.frequency. For some releases, we'll carry both around.

[Jackie-Jiang]

LGTM otherwise. One critical comment in SegmentRelocator

[npawar]

LGTM otherwise. One critical comment in SegmentRelocator

Made one small change not part of the review. Converted the "segmentSelectorType" and "storageType" to enums. If you want to look again.

[Jackie-Jiang]

Made one small change not part of the review. Converted the "segmentSelectorType" and "storageType" to enums. If you want to look again.

Since we introduced the enum, try to use enum over string for these 2 fields

[xiangfu0]

Shall we support multiple tags for a tier?

[npawar]

Yes that's a good idea. I can add that for phase 2 (where I'll be handling advanced instance assignments for tiers)

[xiangfu0]

thinking of adding a method like int getPriority()?
For time based tiers we can use internal age for comparison if priority is the same.

[npawar]

I thought I'll add that when the requirements demand it.. As of now, I didn't see it being requirwd

[npawar]

Made one small change not part of the review. Converted the "segmentSelectorType" and "storageType" to enums. If you want to look again.

Since we introduced the enum, try to use enum over string for these 2 fields

The reason I kept it string is that if we want people to plug in their own strategies, they can do so without needing to add it to the enum

[Jackie-Jiang]

Made one small change not part of the review. Converted the "segmentSelectorType" and "storageType" to enums. If you want to look again.

Since we introduced the enum, try to use enum over string for these 2 fields

The reason I kept it string is that if we want people to plug in their own strategies, they can do so without needing to add it to the enum

In that case, IMO we should not introduce the enum

[npawar]

Made one small change not part of the review. Converted the "segmentSelectorType" and "storageType" to enums. If you want to look again.

Since we introduced the enum, try to use enum over string for these 2 fields

The reason I kept it string is that if we want people to plug in their own strategies, they can do so without needing to add it to the enum

In that case, IMO we should not introduce the enum

Hmm, agreed. Reverted enum change.