fix(HeartbeatCache): Falsely timing out alive executors when heartbeat TIME_SECS does not advance

[DiogoP98]

What is the purpose of the change

Issue: #8419

After a Nimbus restart, healthy topologies with the correct number of running workers are repeatedly rescheduled. In the Nimbus logs, the following message appears for executors that are actually running:

Executor <topo-id>:<executor> not alive

This causes nimbus to detect workers down, which continuously triggers rescheduling for those topologies.

Root Cause

In HeartbeatCache, the internal liveness timestamp (nimbusTimeSecs) was only refreshed when the heartbeat's TIME_SECS value changed between consecutive calls.

For RPC-based heartbeats, TIME_SECS represents the worker's wall-clock send time. If two heartbeats are processed within the same second, TIME_SECS is identical, nimbusTimeSecs is not refreshed, and after nimbus.task.timeout.secs (default 30s) the executor is falsely considered dead — even though it is actively heartbeating.

Fix

The heartbeat update logic is split into two separate methods to fix the RPC path while preserving backwards compatibility with legacy ZK-based topologies:

• updateFromRpcHb — always refreshes nimbusTimeSecs on every heartbeat, so idle-but-alive executors are never falsely timed out.
• updateFromZkHb — retains the original behaviour, only refreshing nimbusTimeSecs when TIME_SECS advances. This preserves zombie detection for legacy topologies where TIME_SECS is stats-based and genuinely stops advancing when an executor is stuck.

How was the change tested

Compiled the project, and tested the new version of the code in Nimbus machines, forcing Nimbus restarts and leadership changes

[paxadax]

LGTM, thank you for the contribution to this bug.

[reiabreu]

Thanks for the contribution , will try to review it asap

[reiabreu]

Thanks for the fix. Will be merging it now.