Lingo should retry on proxy failure

[nstogner]

There are many reasons a backend model server can fail to serve a request. If lingo adds retries on failure it could improve the overall reliability of the system.

[alpe]

This can easily create exponential load. Can you share some scenarios that make sense for you?

[samos123]

the scenarios I encountered so far were:

• Backend going away and proxy still routing requests to a backend that is no longer there
• Client not handing a retry and missing responses. Arguably the user should retry but sadly we have no control over the user.

The main use case would be for providers that run Lingo as a managed service or internally to internal end-users and need to minimize the amount of errors returned to their end-users

[nstogner]

It is possible we could do this an the ingress layer into the cluster as well.

The biggest source of 503s has been misconfigured termination grace periods on model backends (which can take a long time to process all of their pending requests - longer than the 30s default). This should be mostly solvable by making sure we have knobs turned correctly with max-in-flight and a gracious termination period.