Feature: Automatically manage number of longhorn replicas upon node deletion.

[MiroslavRepka]

Motivation

Currently, when a node contains a longhorn volume and we delete that node, a new replica of the volume is not created. The longhorn devs advise to temporarily increase the replica count, so a new one is scheduled and then decrease it back to the desired number. By doing this, we assure the volume will be accessible as long as there are any nodes assigned to that storage class. There will be a need for some additional logic, like checking if a new replica can be even created, or maybe deleting the old replica (from the deleted node).

Description

Claudie should try to manage longhorn volume replicas when deleting a node from the cluster.

Exit criteria

• The replicas are now created whenever possible on a node deletion.
• The docs will document how node deletion work in regard to the storage.

[MiroslavRepka]

Comment about how to change number of replicas via kubectl