KNOX-3041: Eliminate load-balancing race condition in HA dispatch

[hanicz]

KNOX-3041 - Eliminate load-balancing race condition in HA dispatch

What changes were proposed in this pull request?

Replace the peek-at-rewrite + shift-after-response backend selection with an atomic pick-and-advance at dispatch time. The old pattern let concurrent requests observe the same queue head for the entire HTTP round trip, so a slow request would pin every concurrent request onto the same backend.

• Add URLManager#getActiveURLAndAdvance (and the HaProvider passthrough), implemented as a synchronized poll+offer so concurrent callers see distinct URLs whenever more than one is configured.
• setBackendUri now picks and rotates atomically; remove shiftActiveURL / shiftCallback and the after-response shift in the sync and SSE dispatchers.
• Add updateBackendURL so a re-targeted request carries the picked backend's base path (HA instances may publish per-server paths, e.g. HiveServer2), recovering the source's base path by matching configured pool URLs.

Two pre-existing bugs remain

1. activeURL is set for the failed one after a failover for non LB-requests.
2. Failover might go to the same backend that just failed if there are concurrent requests that rotate the URLs.

How was this patch tested?

• Tested locally with custom service
• Tested locally with custom SSE service
• Unit tests

Performance testing for two backends. 500 request with 10 always running concurrently, no sleep on either of the backends

With changes in ~2 seconds:

• Backend 1 250 requests
• backend 2 250 requests

Without changes in ~2 seconds:

• Backend 1 253 requests
• backend 2 247 requests

500 request with 10 always running concurrently, with randomised (1-2s) sleep on backend 1

With changes:

• Backend 1 250 requests
• backend 2 250 requests

Without changes:

• Backend 1 276 requests
• backend 2 224 requests

Conclusion:
Tests confirm no performance degradation and the requests perfectly even out

Integration Tests

N/A

UI changes

N/A

[github-actions]

Test Results

40 tests   40 ✅  6s ⏱️
 3 suites   0 💤
 3 files     0 ❌

Results for commit a8b63ac.