fix(scheduler): default no-executors grace to executor timeout (>= heartbeat)

[akshaychitneni]

Which issue does this PR close?
Follow-up to #2029 / #2212, addressing the timeout interaction surfaced by #2240 (#2240 (comment))

Rationale for this change
The empty-cluster grace timer defaulted to 30s, but the executor heartbeat interval defaults to 60s. Since #2240 routes task-launch-failure removals through the same ExecutorLost path (aggressive removal that also drops the heartbeat), a healthy executor removed by a transient blip re-registers only on its next heartbeat (~≤60s) — which the 30s timer could beat,
spuriously failing the job.

What changes are included in this PR?

• no_executors_grace_period_seconds now defaults to executor_timeout_seconds (guaranteed longer than the heartbeat interval; the scheduler can't read the executor's heartbeat setting directly).
• CLI arg is Option, resolving to executor_timeout_seconds when unset; SchedulerConfig::Default matches.
• Removal stays immediate (fast rescheduling); only the fail decision waits, and the timer's re-check still cancels the failure if the executor re-registers. Explicit values (incl. 0 = fail-fast) honored.
• Adds config tests (default resolution + grace >= executor_timeout invariant).

Are there user-facing changes?
Yes — the default no-executors grace period changes from 30s to executor_timeout_seconds (180s by default); --no-executors-grace-period-seconds is now optional and defaults to that.

[sandugood]

Should this be 180 by default, if the executor_heartbeat_interval_seconds is 60 secs by default?

[akshaychitneni]

Yes. 180 is the default executor_timeout_seconds (the liveness window) giving 3× margin for a transiently-removed executor to re-register before its jobs fail

[milenkovicm]

Thanks @akshaychitneni this makes sense, one minor comment, and a question if we can add validation method now when we have dependencies between configuration values

[akshaychitneni]

Thanks @akshaychitneni this makes sense, one minor comment, and a question if we can add validation method now when we have dependencies between configuration values

Thanks. Added SchedulerConfig::validate(), called from start_server. It warns when no_executors_grace_period_seconds (nonzero) is below executor_timeout_seconds, and when the timeout isn't greater than the expire interval

[milenkovicm]

Thanks @akshaychitneni