Region termination retries 150 times, stalling teardown for ~30 s when a worker will not die

[aglinxinyuan]

Task Summary

Region termination retries EndWorker + gracefulStop up to 150 times at a flat 200 ms, so a region whose workers do not terminate holds teardown for ~30 s per region before the failure surfaces:

WARN [COORDINATOR] RegionExecutionManager - Failed to terminate region 1 on attempt 1 of 150. Retrying in 200 ms.

(that line is the one visible in the #7056 hang logs).

Killing a worker is deterministic: EndWorker plus gracefulStop either succeed, or something upstream is broken in a way retrying cannot fix (#6891, #6918). A 150-attempt poll therefore buys nothing except a long, quiet stall that masks the real bug and stretches CI runs.

Shorten the budget to one attempt plus three backed-off retries:

	Now	Proposed
Attempts	150	4 (1 + 3 retries)
Delay between attempts	flat 200 ms	200 -> 400 -> 800 ms
Worst-case wait per region	~30 s	~1.4 s

Give-up behavior stays as it is today: fail loudly, naming every worker that never terminated.

Before:  EndWorker fails -> 149 retries x 200 ms -> ~30 s of silence -> loud failure
After:   EndWorker fails -> 3 retries (200/400/800 ms) -> ~1.4 s -> loud failure

Also shrinks (does not close) the timer-thread race window described in #6923, since there are far fewer retry hops onto the JavaTimer thread.

Task Type

• Performance