You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now when a node disappears and appears again, clusterm updates it's state in inventory however, no configuration related action is taken on the node. This might not always be desirable as when a node reappears, it might need the services setup again.
This issue explores a few options to handle these scenarios. Feel free to pitch in with comments and feedback.
Following are some scenarios where a node may disappear and then what may happen or be desired subsequently:
Scenario 1: node looses network connectivity (i.e. control interface traffic is affected)
since node is up all the services will stay running but most likely services will readjust to assume the host to be down.
when node appears again, if nothing has changed then node will get added back to cluster
But it is not necessarily guaranteed that the service configuration has not changed.
There is a need for some sort of config checksum equivalent, that can help here perhaps.
Re-setup (after config check) of services will ensure that node is on-boarded back with correct config
Scenario 2: node is rebooted (due power-loss or admin action)
the services get stopped on the node
when the node is booted up again, services need to be re-setup.
Re-setup of services will ensure that node is on-boarded back with correct config
Scenario 3: just serf is somehow affected (due to crashes or bugs or admin action):
first this is a service failure and we may be better off debugging serf itself.
since node is up all the services will stay running and also continue to be part of cluster
when serf recovers, most likely no configuration action is required.
There is a need for some sort of config checksum equivalent, that can help here perhaps.
Re-setup of services is most likely not needed but will ensure that node is on-boarded back with correct config.
Note: There is also a time period (especially Scenario 3 above) when node is down in monitoring system but still reachable from service perspective. We will ignore this scenario for now as we may need to debug and fix the condition that caused it instead. Also the side-effect doesn't seem too bad, unless service configuration change is desired during this time.
What follows next is a high level proposal for possible ansible and clusterm enhancements that would address node disappearance and reappearance scenario:
Configuration check (ansible tasks and plays) :
We could add config check tasks per service that would verify that service is configured in desired fashion. Some example of this are:
etcd:
master node could check that etcd service is running in master mode
worker node could check that etcd service is running in proxy mode
etcd ports are up to date ... and so on
contiv_network:
master node shall be running netmaster service in addition to netplugin
worker node shall be just running netplugin
netplugin/master ports are up to date
if the checks fail, then a service re-setup shall be triggered (see next section)
Service Re-setup (clusterm discovered event handler):
a service re-setup is triggered when a configuration check fails
resetup would involve running cleanup.yml followed by regualr provisioning based on node's host-group.
The text was updated successfully, but these errors were encountered:
Right now when a node disappears and appears again, clusterm updates it's state in inventory however, no configuration related action is taken on the node. This might not always be desirable as when a node reappears, it might need the services setup again.
This issue explores a few options to handle these scenarios. Feel free to pitch in with comments and feedback.
Following are some scenarios where a node may disappear and then what may happen or be desired subsequently:
serf
is somehow affected (due to crashes or bugs or admin action):Note: There is also a time period (especially
Scenario 3
above) when node is down in monitoring system but still reachable from service perspective. We will ignore this scenario for now as we may need to debug and fix the condition that caused it instead. Also the side-effect doesn't seem too bad, unless service configuration change is desired during this time.What follows next is a high level proposal for possible ansible and clusterm enhancements that would address node disappearance and reappearance scenario:
Configuration check (ansible tasks and plays) :
Service Re-setup (clusterm discovered event handler):
The text was updated successfully, but these errors were encountered: