CephFS mirroring: Importing Bootsrap peer secret sometimes fail with Status 22

[ahmedTouati]

We have two cephclusters each installed on an EKS cluster in a different AWS region, to ensure mirroring between both clusters, we had to change mons services type of secondary cluster to LoadBalancer so that the first cluster can connect to them, I created the bootstrap peer token and put the secret name in the cepFS manifest, but rook-operator is failing with Status 22. Here is an example bootstrap peer token that I use:

ZXlKbWMybGtJam9nSW1ReU9URXhOR05sTFRaaU1qVXROR00xWVMwNE56TXpMV0UyTVRSaU0yRmhOemd6WmlJc0lDSm1hV3hsYzNsemRHVnRJam9nSW1WNFlXMXdiR1V0Wm5NaUxDQWlkWE5sY2lJNklDSmpiR2xsYm5RdWJXbHljbTl5WDNKbGJXOTBaU0lzSUNKemFYUmxYMjVoYldVaU9pQWljbTl2YXkxalpYQm9JaXdnSW10bGVTSTZJQ0pCVVVSMU1HOWthRTFJVTI5TlFrRkJXbk5uWVRKNU1XbHphbFZvTkVsaVdIVnNZWEZrUVQwOUlpd2dJbTF2Ymw5b2IzTjBJam9nSWx0Mk1qcHBiblJsY201aGJDMWhZV0V0TVRJekxtVjFMWGRsYzNRdE1pNWxiR0l1WVcxaGVtOXVZWGR6TG1OdmJUb3pNekF3TEhZeE9tbHVkR1Z5Ym1Gc0xXRmhZUzB4TWpNdVpYVXRkMlZ6ZEMweUxtVnNZaTVoYldGNmIyNWhkM011WTI5dE9qWTNPRGxkTEZ0Mk1qcHBiblJsY201aGJDMWlZbUl0TVRJekxtVjFMWGRsYzNRdE1pNWxiR0l1WVcxaGVtOXVZWGR6TG1OdmJUb3pNekF3TEhZeE9tbHVkR1Z5Ym1Gc0xXSmlZaTB4TWpNdVpYVXRkMlZ6ZEMweUxtVnNZaTVoYldGNmIyNWhkM011WTI5dE9qWTNPRGxkTEZ0Mk1qcHBiblJsY201aGJDMWpZMk10TVRJekxtVjFMWGRsYzNRdE1pNWxiR0l1WVcxaGVtOXVZWGR6TG1OdmJUb3pNekF3TEhZeE9tbHVkR1Z5Ym1Gc0xXTmpZeTB4TWpNdVpYVXRkMlZ6ZEMweUxtVnNZaTVoYldGNmIyNWhkM011WTI5dE9qWTNPRGxkSW4wPQ==

How to reproduce it (minimal and precise):

1. Create mirror_remote client on the secondary cluster: ceph fs authorize example-fs client.mirror_remote /dir1 rwps
2. Create bootstrap peer token on the secondary cluster: ceph fs snapshot mirror peer_bootstrap create example-fs client.mirror_remote rook-ceph
3. Create CephFilesystemMirror crd on the first cluster
4. Create a k8s secret with the generated bootstrap peer token and call it in the cephFS manifest of the first cluster

Environment:

• Cloud provider or hardware configuration: AWS
• Rook version (use rook version inside of a Rook Pod): 1.7.5
• Storage backend version (e.g. for ceph do ceph -v): 16.2.6
• Kubernetes version (use kubectl version): 1.20

[leseb]

Can you describe the "sometimes fails"? Isn't this always reproducable? Thanks

[ahmedTouati]

Can you describe the "sometimes fails"? Isn't this always reproducable? Thanks

@leseb Yes it's not always reproducible and nothing is shown in the logs to check why it fails

[leseb]

Can you print do a kubectl -n rook-ceph get secret <peer secret> -o yaml and paste it here? Thanks.

[leseb]

Also, please make sure you don't make any typos or introduce white spaces/trailing spaces when creating the secret peer.

[leseb]

Please try again in 1.7.9, it should be released in a few. Thanks