feat: add multiple attempts options to client.post API

[JoanFM]

Goal of this PR is to enable retry_mechanism in the client.

• Expose max_attempts, initial_backoff, max_backoff and backoff_multiplier in the client.post API
• Clarify usage of continue_on_error with any kind of Errors
• Test features
• Document

Changes

• Allow to pass max_attempts, initial_backoff, max_backoff and backoff_multiplier to client.post API

  • In gRPC, these are added as configuration to the grpcChannel
  • In HTTP and websocket, we do a manual retry following the formula proposed by grpc
    The initial retry attempt will occur at random(0, initialBackoff). In general, the n-th attempt will occur at random(0, min(initialBackoff*backoffMultiplier**(n-1), maxBackoff)).

• Document continue_on_error behavior.

[codecov]

Codecov Report

Merging #5176 (1231c39) into master (13edf90) will decrease coverage by 0.15%.
The diff coverage is 71.15%.

@@            Coverage Diff             @@
##           master    #5176      +/-   ##
==========================================
- Coverage   84.76%   84.60%   -0.16%     
==========================================
  Files         103      103              
  Lines        7166     7191      +25     
==========================================
+ Hits         6074     6084      +10     
- Misses       1092     1107      +15     

Flag	Coverage Δ
jina	84.60% <71.15%> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
jina/clients/base/http.py	94.59% <ø> (ø)
jina/clients/base/websocket.py	90.32% <ø> (ø)
jina/clients/mixin.py	95.00% <ø> (ø)
jina/clients/base/helper.py	81.96% <65.21%> (-2.80%)	⬇️
jina/clients/base/grpc.py	85.50% <75.86%> (-3.02%)	⬇️
...a/orchestrate/deployments/config/docker_compose.py	94.55% <0.00%> (-1.00%)	⬇️
jina/orchestrate/flow/base.py	83.33% <0.00%> (-0.70%)	⬇️
jina/hubble/hubio.py	90.10% <0.00%> (+0.17%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[JohannesMessner]

This seems like a slightly more sophisticated retry mechanism compared to the one that we currently have between gateway and executors.

There (I believe) we don't use the built-in gRPC mechanism, but we do it manually (I think because we want to be able to switch to a different replica when one of them doesn't respond).
Should we implement this exponential backoff strategy there as well?

[alexcg1]

Some clarification needed

[alexcg1]

LGTM 👍

[github-actions]

📝 Docs are deployed on https://docs-continue-on-error--jina-docs.netlify.app 🎉

changes applied