[controller] Submit roll forward calls in parallel

[misyel]

Problem Statement

When a roll forward request hits the parent, it will call roll forward to each region sequentially. If one region is slow, the next region is also stalled and has to wait for the previous region to complete before starting

Solution

Submit the roll forward request to all regions in parallel and wait for each request to complete afterwards

Code changes

• Added new code behind a config. If so list the config names and their default values in the PR description.
• Introduced new log lines.
  • Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

• Code has no race conditions or thread safety issues.
• Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).

Existing CI

Does this PR introduce any user-facing or breaking changes?

• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.

[Copilot]

Pull request overview

This PR parallelizes the parent controller’s roll-forward fan-out to child regions so a slow region doesn’t block submission to other regions.

Changes:

• Submits rollForwardToFutureVersion requests to all regions concurrently via asyncSetupExecutor.
• Waits for all region requests to complete, then aggregates failures and throws a single exception if any region failed.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sofiaz11]

We are still waiting in the future.get() for each region, which means the max timeout will be 3 * 60s in total. I wonder why don't we use something like this instead CompletableFuture.allOf(TIMEOUT, TimeUnit.MILLISECONDS)?

[misyel]

Good point - updated to make it a shared timeout

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.