Zero downtime: Connected users should be able to continue working with osparc when osparc micro-services are restarted

[sanderegg]

Use-case:

1. someone is connected to osparc, working with studies
2. the osparc-platform is re-deployed
3. the user of the osparc platform should continue to work seamlessly, maybe with a small acceptable glitch but it should definitely not complete with receiving 500 HTTP codes

references:

graylog entries related to failed e2e

the e2e of isolve-mpi failed with the webserver returning a 500 for listing projects. one can see in the logs that the webserver was restarting at that moment.

Docker reference:

• docker swarm starts a new service (the replacing webserver) and waits until it is healthy. once healthy it closes the replaced service, waits 10seconds and then kills it if it is still around.

• : restarting webserver works

• : restarting database

• : restarting other subsystems

[pcrespov]

Related to #2140

• New version is deployed.
• The deployer updates the stack
• The user gets interrupted because the service is down and sometimes gets a gateway error
  • Instead, the old service should not be shut-down until the new stack is in place

Possible cause:

The swarm is already configured to have zero downtime per service (i.e. a given service gets turned off ONLY when the new one is started). The problem might be that even if services are ready, the state between services is not ready. For example, the new webserver is updated correctly but traffik proxy has still not detected it. That would cause a wrong gateway failure on a front-end request

Ideas to solve this problem

• Incorporate more conditions on the "healtcheck" validation function (e.g. traffik has discovered all backend services)
• Might be to deploy the entire stack separately first, have a set of rules to validate (e.g. all services healthy, all services connected, traffik routings ready) and then switch.
• Notify the frontend that a new version was deployed Front-end notifies of a new version available #2128

[sanderegg]

Testing like so:

• running a lot of sleepers
  -> restarting the webserver seems to be ok
  -> restarting the database restarts some of the computational services, but seems to generate issues loosing some states.