fix #4788 moving retry logic into the standard layer

[shawkins]

Description

Fix #4788

This makes the retry logic common for all requests - sendAsync, consumeBytes, and websockets. There wasn't a great way of making the client executor available due to when the httpclient is created vs when the executor is obtained - further refactoring could probably make this work, but for now since the httpclient calls are non-blocking I'm just running the delayed task in the scheduling thread.

@manusa see if you want to refine the max connections test based upon this.

This does blur the line between the request and watch retries. Low level issues can be handled by the request retry, while higher level ones, such as repeated error status responses would still need to be handled by the watch retry.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change
• Chore (non-breaking change which doesn't affect codebase;
  test, version modification, documentation, etc.)

Checklist

• Code contributed by me aligns with current project license: Apache 2.0
• I Added CHANGELOG entry regarding this change
• I have implemented unit tests to cover my changes
• I have added/updated the javadocs and other documentation accordingly
• No new bugs, code smells, etc. in SonarCloud report
• I tested my code in Kubernetes
• I tested my code in OpenShift

[manusa]

@manusa see if you want to refine the max connections test based upon this.

I think the test is fine as it is. In the end we didn't add the retry logic to the test. The test "improves" the embedded HttpServer reliability (also for OkHttp MockServer). All in all, I think that having more solid Mock servers will allow us to identify and isolate implementation problems in the future.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

92.4% Coverage
0.0% Duplication