SegmentStatusChecker emits incorrect percentsegmentsavailable metrics

[shuturmurgh]

• SegmentStatusChecker periodic task occasionally emits incorrect percentsegmentsavailable metric by marking segments as OFFLINE when they have already transitioned to CONSUMING state.
• This issue has only been observed for large tables (40k-100k segments)
• No signs of ZK replica lag or memory bottleneck
• Increasing controller.statuschecker.waitForPushTimePeriod from 10min (default) to 20 min hasn't resolved the issue since the lag can be over 30 mins which is unacceptable

Example Timeline

Timestamp	Component	Event
02:45:57	Broker	Received new segment table__28__2014__20251207T2115Z via EV update
02:46:12	Server	Segment transitioned OFFLINE → CONSUMING
03:13:50	Controller	SegmentStatusChecker reports segment has no ONLINE/CONSUMING replica

[bluemeaford]

Thanks for the clear timeline — a couple of clarifying questions that might help narrow this down:
1. Pinot version + deployment mode (and Helix/ZK versions)? Also: single controller or HA/multi-controller?
2. Are these segments REALTIME only, or mixed with OFFLINE?
3. When the checker reports “no ONLINE/CONSUMING replica”, does Helix/ExternalView still show the segment as CONSUMING on a server, or is EV actually missing it at that moment?
4. Any related controller logs around the check cycle (e.g., “missing replicas” / “no instance for partition” / “idealstate mismatch”)?

If EV still shows CONSUMING, one possibility is a controller-side view/cache/race in SegmentStatusChecker; if EV is missing, it may be Helix state propagation lag or a transient EV inconsistency under load.

[shuturmurgh]

1. Versions: Pinot 1.3, Zookeeper: 3.8.4. Controller Mode: Dual.
2. Only REALTIME
3. ZK EV shows the segment present
4. Couldn't find any other relevant Controller log. Since Table table_REALTIME has 1 segments without ONLINE/CONSUMING replica: [table__1__12853__20260102T0433Z] log is present and Table table_REALTIME has 1 segments marked unavailable due to unavailable instances: is NOT present we can assume the issue is due to segment being missing in EV

[bluemeaford]

Thanks, that helps clarify things. One small follow-up to narrow this down:

Could you paste the ExternalView state map for table__1__12853__20260102T0433Z at the time the checker reports “no ONLINE/CONSUMING replica” (i.e., the instance → state entries)?
Also, what does IdealState show for that segment at the same time, and are the EV instances listed all LIVE (LiveInstances) / enabled?

If EV shows only OFFLINE/ERROR/DROPPED (or shows CONSUMING on a non-LIVE instance), that would explain the checker result; if EV shows CONSUMING on a LIVE instance, then this points more toward a controller-side cache/timing race in SegmentStatusChecker.

[shuturmurgh]

• ExternalView and IdealState both have those segments as ONLINE
• EV instances are LIVE
  I also feel it is a Controller cache issue

[shuturmurgh]

Figured out the root cause. SegmentStatusChecker would incorrectly report a segment as offline if following two conditions are true

1. It was created after EV had been fetched
2. It's creation time was checked so late in iteration that its age exceeded waitForPushTimePeriod

These conditions can happen if table has high number of segments and commits segments very frequently

The fix (pull/16496) for this issue has been merged but not released yet