potentially invalid ceph.conf after moving monitors

[jschmid1]

I used ceph-salt to deploy bootstrap services on a master node. then used cephadm
to spin-up mons on node{1..3} with cephadm orch apply mon node1,node2,node3. Now, the master's ceph.conf is outdated(still pointing to the initial bootstrap mon), hence any ceph command will hang as it's relying on the ceph.conf to retrieve the mon addresses.

There is an cephadm tracker issue for a related issue already: https://tracker.ceph.com/issues/44792

[ricardoasmarques]

Should be fixed by https://tracker.ceph.com/issues/45378

[denisok]

we had same issue during our testing, there is disconnection between 2 tools - ceph-salt and ceph orch.
We also had the issue of ceph-salt not being able to connect to the ceph, and thus hanging without any errors. Looks like it runs ceph orch status and never returns (at least we weren't patient enough).

At least show error ?

Even -l debug didn't help to find out what is wrong, we just noticed that it spawns that ceph orch status process.

[smithfarm]

@denisok Did you try running ceph orch status directly from the command line (not via ceph-salt)?

[denisok]

sure, it was hanging because ceph.conf pointed to the wrong ip, as ceph orch apply mon changed that mon :)

[matthewoliver]

OK, so when we run something like:
ceph orch apply mon node2,node3

This doesn't mean run mon also on these nodes but instead only run it on these nodes? because if that's the case, that doesn't seem intuitive to me.

Changing it in /etc/ceph/ceph.conf to point at one of the new mons worked however. When I ran cephadm it didn't seem to work, should that have created a new and correct cephadm?

[denisok]

and even when we removed ceph.conf it did hang :)

ceph-salt/ceph-salt-formula/salt/_modules/ceph_orch.py

Line 12 in 7e6d33d

ret = __salt__['cmd.run_all']("ceph orch status")

looks like that is in some kind of salt state that tries to apply it over and over again ? disregarding that it fails ?

we need kind of info and timeout probably )

[ricardoasmarques]

I wonder if changing it to cephadm shell -- ceph orch status is an alternative for the timeout, see my comment on the following commit smithfarm@5e7fa83 - need to be tested

[denisok]

maybe, but some error message would be also nice

[ricardoasmarques]

cephadm PR to automatically keep /etc/ceph/ceph.conf in sync on all hosts: ceph/ceph#35576