Trying to change network binds after bootstrap does not seem to work

[gustavosr98]

Issue report

Trying to change network binds after bootstrap does not seem to work

What version of MicroCeph are you using ?

quincy/stable (0+git.4a608fc)

What happens (observed behaviour) ?

# ip a 
[..]
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:30:33:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.99/24 metric 100 brd 192.168.10.255 scope global dynamic enp5s0
       valid_lft 572785sec preferred_lft 572785sec
    inet 10.5.0.3/16 scope global enp5s0
       valid_lft forever preferred_lft forever

# microceph bootstrap

# sudo ceph config generate-minimal-conf
[global]
	fsid = d2b14f41-10a0-44e2-96a8-a1f10171e5b9
	mon_host = [v2:192.168.10.99:3300/0,v1:192.168.10.99:6789/0]

Now I followed this to change the network bind to another one
https://canonical-microceph.readthedocs-hosted.com/en/quincy-stable/how-to/configure-network-keys/

# sudo microceph cluster config set public_network 10.5.0.0/16
# sudo microceph cluster config set cluster_network 10.5.0.0/16
# sudo snap restart microceph
2024-05-31T00:06:40Z INFO Waiting for "snap.microceph.mds.service" to stop.
2024-05-31T00:06:42Z INFO Waiting for "snap.microceph.osd.service" to stop.
2024-05-31T00:06:47Z INFO Waiting for "snap.microceph.rgw.service" to stop.
Restarted.

But it still binds to the old network

# sudo ceph config generate-minimal-conf
[global]
	fsid = d2b14f41-10a0-44e2-96a8-a1f10171e5b9
	mon_host = [v2:192.168.10.99:3300/0,v1:192.168.10.99:6789/0]

What were you expecting to happen ?

I was expecting to be able to change the network configuration binding after having bootstrapped a ceph cluster to migrate from one network to another

[nobuto-m]

See #197

Changing the public network after the bootstrap doesn't quite work so it was removed from the doc but the removal hasn't been backported it seems.

[UtkarshBhatthere]

As @nobuto-m mentioned, changing the Public Network laterally is very complicated. To prevent this situation from coming in, we removed the ability to configure public network (from the CLI) as well as docs. This has been backported to quincy/edge but has not made it's way back to quincy/stable (unfortunately, taking a note here.). I would recommend using reef track since it has a better stability/velocity factor.

[UtkarshBhatthere]

(From a parallel conversation) I figure Ceph allows for configuring multiple public networks (Microceph doesn't, but we can fix that), this can be used to migrate the public networks of a Ceph cluster. Now this will require some operator diligence but MicroCeph can be more helpful here.