Potential Race Condition against Segment Tier during Reload All Segments

[J-HowHuang]

Description

We fetch the IndexLoadingConfig once here:

pinot/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java

Line 625 in 50e061f

reloadSegments(segmentDataManagers, fetchIndexLoadingConfig(), forceDownload, reloadJobId);

Then we submit tasks that reload individual segments to the reload/refresh executor, where each task do set the tier to the shared IndexLoadingConfig object based on the segment itself:

pinot/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java

Lines 1002 to 1006 in 50e061f

	public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig, SegmentZKMetadata zkMetadata,
	SegmentMetadata localMetadata, boolean forceDownload)
	throws Exception {
	String segmentTier = getSegmentCurrentTier(segmentName);
	indexLoadingConfig.setSegmentTier(segmentTier);

Impact

This can cause problem where segments would be preprocessed with a wrong config if tier overwrite is present in table config, when the instance has segments on different tiers.