EMQX cluster cannot restart after persistence

[qRainBirdp]

What happened?

My EMQX cluster has three nodes. After all three nodes crashed and restarted, it cannot be restarted using the persistent volume claims (PVCs). The pods keep restarting.

What did you expect to happen?

The cluster can be restored normally through the PVC.

How can we reproduce it (as minimally and precisely as possible)?

Adding persistence and simultaneously shutting down all nodes, then restoring the cluster.

Anything else we need to know?

No response

EMQX version

emqx-operator-controller:2.2.17
emqx:5.1

OS version

AWS EKS1.23

Log files

2024-04-23T05:53:52.007602+00:00 [error] msg: failed_to_sync_cluster_conf, mfa: emqx_conf_app:sync_cluster_conf2/1, line: 160, failed: ['emqx@emqx-core-796d49c9f4-0.emqx-headless.emqx.svc.cluster.local'], nodes: ['emqx@emqx-core-796d49c9f4-2.emqx-headless.emqx.svc.cluster.local','emqx@emqx-core-796d49c9f4-0.emqx-headless.emqx.svc.cluster.local'], not_ready: [{error,#{msg => "init_conf_load_not_done",node => 'emqx@emqx-core-796d49c9f4-2.emqx-headless.emqx.svc.cluster.local'}}] -- Tue, Apr 23 2024 1:53:56 pm | Received terminate signal, shutting down now

[zmstone]

Hi @qRainBirdp
Could you provide logs from all three nodes ?