microk8s connect-external-ceph

[lansaloni]

Hi all,
I have a Proxmox cluster with 3 nodes and Ceph storage, I installed version 1.28/stable of microk8s on 3 VMs with the command:
sudo snap install microk8s --classic --channel=1.28/stable
I would like to use Proxmox Ceph cluster as shared storage for microk8s cluster and so I enabled rook-ceph and this is the status of the cluster:

$ microk8s status
microk8s is running
high-availability: yes
  datastore master nodes: 10.15.10.121:19001 10.15.10.122:19001 10.15.10.123:19001
  datastore standby nodes: none
addons:
  enabled:
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
    rook-ceph            # (core) Distributed Ceph storage using Rook
  disabled:
    cert-manager         # (core) Cloud native certificate management
    cis-hardening        # (core) Apply CIS K8s hardening
    community            # (core) The community addons repository
    dashboard            # (core) The Kubernetes dashboard
    gpu                  # (core) Automatic enablement of Nvidia CUDA
    host-access          # (core) Allow Pods connecting to Host services smoothly
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    ingress              # (core) Ingress controller for external access
    kube-ovn             # (core) An advanced network fabric for Kubernetes
    mayastor             # (core) OpenEBS MayaStor
    metallb              # (core) Loadbalancer for your Kubernetes cluster
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    minio                # (core) MinIO object storage
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorisation
    registry             # (core) Private image registry exposed on localhost:32000
    storage              # (core) Alias to hostpath-storage add-on, deprecated

I then took the ceph.conf and ceph.client.admin.keyring files from the Proxmox cluster and used them for the connection command:
$ sudo microk8s connect-external-ceph --ceph-conf ceph.conf --keyring ceph.client.admin.keyring --rbd-pool microk8s-rbd
which however returns an error:

Attempting to connect to Ceph cluster
Successfully connected to cd8ad1c4-fde4-4a3a-b169-3f4ba4762d4b (10.15.15.121:0/3422868158)
WARNING: Pool microk8s-rbd already exists
Configuring pool microk8s-rbd for RBD
Successfully configured pool microk8s-rbd for RBD
Creating namespace rook-ceph-external
Error from server (AlreadyExists): namespaces "rook-ceph-external" already exists
Configuring Ceph CSI secrets
Traceback (most recent call last):
  File "/var/snap/microk8s/common/plugins/connect-external-ceph", line 184, in <module>
    main()
  File "/snap/microk8s/6089/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/snap/microk8s/6089/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/snap/microk8s/6089/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/microk8s/6089/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/var/snap/microk8s/common/plugins/connect-external-ceph", line 169, in main
    import_external_ceph_cluster(ceph_conf, keyring, namespace, rbd_pool)
  File "/var/snap/microk8s/common/plugins/connect-external-ceph", line 109, in import_external_ceph_cluster
    p = subprocess.run(
  File "/snap/microk8s/6089/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/snap/microk8s/6089/usr/bin/python3', PosixPath('/var/snap/microk8s/common/plugins/.rook-create-external-cluster-resources.py'), '--format=bash', '--rbd-data-pool-name=microk8s-rbd', '--ceph-conf=ceph.conf', '--keyring=ceph.client.admin.keyring']' returned non-zero exit status 1.

Can anyone help me?

[jeffreychang911]

Get a slightly different error at same step from SolQA test run, all artifacts here.
Please let me know if I should file a new bug.

2024-06-05-08:35:34 root ERROR [localhost] Command failed: juju exec -m foundations-maas:microk8s -u microk8s/0 --wait 14400s 'microk8s connect-external-ceph'
2024-06-05-08:35:34 root ERROR 1[localhost] STDOUT follows:
Looking for MicroCeph on the host
Detected existing MicroCeph installation
Attempting to connect to Ceph cluster
Successfully connected to 3e357a01-eee4-4e97-be2f-defbb5446172 (10.244.40.109:0/3997776507)
Creating pool microk8s-rbd0 in Ceph cluster
Configuring pool microk8s-rbd0 for RBD
Successfully configured pool microk8s-rbd0 for RBD
Creating namespace rook-ceph-external
Configuring Ceph CSI secrets
Successfully configured Ceph CSI secrets
Importing Ceph CSI secrets into MicroK8s
Error from server: rpc error: code = Unknown desc = exec (try: 500): database is locked
error: failed to create secret namespaces "rook-ceph-external" not found