BUG: Cluster in unrecoverable state

[phal0r]

Hi guys,

we discovered a state in one of our clusters managed by the operator, where

• 3 agencies running
• 2 primaries are stuck in Terminating state (the finalizers are keeping the pods from being deleted)
• 1 primary is running, but couldn't sync with the error:

WARNING [de0be] {httpclient} retrying failed HTTP request for endpoint 'tcp://hugo-plus-arango-dbserver-ps3uajld.hugo-plus-arango-int.hugoplus.svc:8529' for replication applier in database 'staging'

All Coordinators are contineously restarting due to no primary available. The cluster is not available.

Arango-Version is 3.5.1, Operator Version is 0.4.4.

Do you need more information? How can I recover the cluster from this state?

Thanks in advance for your help!

[ajanikow]

Hello!

In your case you have propably ReplicationFactor set to 2 in system collections. Can you remove finalizer from one of DBServers?

Your cluster got "stuck", coordinators are not able to connect to DBServer because all pods with _system shards are down. It will be fixed in 1.1.0.

Best Regards,
Adam.

[phal0r]

@ajanikow
thanks for the insights. We will try that and report back.

Does that also mean, replicationFactor of alle _system collections should equal the number of nodes or be at least higher than 2?

[phal0r]

Removing the finalizer of one of the dbservers fixed the dead lock and the cluster recovered without data loss.

We have another problem in combination with the https://github.com/rancher/local-path-provisioner. Local volumes are bound to a node. The operator does not take this into consideration (probably it does not know of this constraint?)