Have cluster running after the nodes restart

[innobead]

For now, after stopping the VMs, there are no ways to restart the VMs w/ the same configured IPs and still make the cluster running w/o issues.

Look for a solution to make cluster management during node state change.

[rugwirobaker]

cluster life cycle commands(start, stop) would be a great addition

[innobead]

This is caused by the default IPAM (host-local) behavior, which does not support the reserved IPs to reuse for the restarted instances via container ID (here it's VM ID).

✔ cat /var/lib/cni/networks/ignite-cni-bridge/10.61.0.104
ignite-527549c6aeab8e84

If you try to start an unexpected stopped VM (like the firecracker process stopped by host restart or power outage), you would encounter the below error, because the assigned IP does not get released by host-local

✔ ignite start 527549c6aeab8e8
ERRO[0000] failed to setup network for namespace "ignite-527549c6aeab8e84": failed to allocate for range 0: 10.61.0.104 has been allocated to ignite-527549c6aeab8e84, duplicate allocation is not allowed 
FATA[0000] failed to allocate for range 0: 10.61.0.104 has been allocated to ignite-527549c6aeab8e84, duplicate allocation is not allowed 

There are two solutions here:

• create another CNI plugin based on host-local but support allocated IPs to reuse when requested by the assigned VMs. Then, use this new ipam plugin instead of host-local.
• use DNS server to dynamically manage the mapping between VM domain name and IPs. All the nodes management base on domain name instead of IP.

[rugwirobaker]

which one of these do you prefer? Wouldn't a CNI plugin be too much more work?

[innobead]

I would prefer going with the first solution w/ less dependencies n overhead before working on multiple nodes support in kubefire.

I will have some progress n update these days :)

[innobead]

• host-local with the support of reserving previous allocated IPs for the existing namespace.
  innobead/plugins@ec0d154
• fix stopped VMs failed to restart
  Failed to restart a stopped VM caused by a handled signal weaveworks/ignite#660

[innobead]

Supported in v0.1.0. Closing.

• kubefire cluster start/stop
• require kubefire-cni-bridge plugin which depends on host-local-rev. (This can be installed by kubefire install)

[innobead]

About VMs unexpected stopped by the host status like power outage, this will be followed up at the upstream PR as below.
weaveworks/ignite#660

[innobead]

About VMs unexpected stopped by the host status like power outage, this will be followed up at the upstream PR as below.
weaveworks/ignite#660

Fixed and merged 💯