Make calls against API resilient

[Brutus5000]

If we make calls against the API running into errors due to the API being down due to restart (deployment of new version). It's not a good user experience to fail on first try and show some stacktrace to the user.

Spring Retry offers mechanics to automatically retry failed calls in a configurable time span. Unresolvable errors can be globally catched.

[micheljung]

@Brutus5000 in order to implement this properly, Traefik needs to return HTTP 503. Retrying on 404 as in #2385 is not a good solution.

[Brutus5000]

This is not possible with the current setup of faf-stack. It's just not how Traefik works in a single-instance non-swarm docker environment.

Traefik listens to events on the Docker daemon. If it sees a service going up that has certain set of labels.
If a docker container is deleted, Traefik will remove all related rules for the subdomain of the service.
If a new container goes up it will accept Traefik again, but only if the service is healthy.
This wouldn't be much of a problem if the API wouldn't require 30-60 seconds to load. But that is the status quo and it's nothing that can be easily changed.

A retry on 404 is something that can be easily changed.

[micheljung]

Well that's simply not true. It's just a matter of configuration. Please see https://doc.traefik.io/traefik/getting-started/faq/

[Brutus5000]

That FAQ is fairly new (since 2.4), good to know.

So far I only deducted what is described there as well:

From Traefik's point of view, every time a request cannot be matched with a router the correct response code is a 404 Not found.

In this situation, the response code is not a 503 Service Unavailable because Traefik is not able to confirm that the lack of a matching router for a request is only temporary. Traefik's routing configuration is dynamic and aggregated from different providers, hence it's not possible to assume at any moment that a specific route should be handled or not.

However this chapter seems to be worth a shot.