Handle task executor gateway connection failures gracefully

[sundargates]

Context

^^^

Checklist

• ./gradlew build compiles code correctly
• Added new tests where applicable
• ./gradlew test passes all tests
• Extended README or added javadocs where applicable

[github-actions]

Test Results

127 files  ±0  127 suites  ±0   6m 25s ⏱️ +9s
537 tests ±0  528 ✔️ ±0  8 💤 ±0  1 ❌ ±0 
538 runs  ±0  529 ✔️ ±0  8 💤 ±0  1 ❌ ±0 

For more details on these failures, see this check.

Results for commit dabf29f. ± Comparison against base commit 5034b7b.

[Andyz26]

nit: any backoff/delay needed to avoid short-circuiting the scheduler

[sundargates]

I'm sending a retry message here, which delays how it re-queues the original request.

log.error("Failed to submit the request {}; Retrying in {} because of ", event.getScheduleRequestEvent(), event.getThrowable());
            getTimers().startSingleTimer(
                getSchedulingQueueKeyFor(event.getScheduleRequestEvent().getRequest().getWorkerId()),
                event.onRetry(),
                intervalBetweenRetries);