Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

daemon/start_mon.sh: add mon without port value #1410

Merged
merged 1 commit into from Jul 3, 2019

Conversation

dsavineau
Copy link
Contributor

@dsavineau dsavineau commented Jun 27, 2019

When bootstrapping an initial cluster, we don't need to add the mon
to the monmap via 'ceph mon add' command.
But when we want to scale up the monitors then we need it.
Currently the new mon added will be out of quorum because the addr
in the ceph config and in the monmap won't match.

start_mon(): ceph --cluster ceph mon add xxx 192.168.100.5:3300
mon.xxx at v2:192.168.100.5:3300/0 already exists

Even if the ceph conf uses v2+v1 then the new mon will have only v2
address configured in the monmap creating a conflict.

0 mon.xxx@0(probing) e3 monmap addrs for rank 4 changed, i am
[v2:192.168.100.5:3300/0,v1:192.168.100.5:6789/0], monmap is
v2:192.168.100.5:3300/0, respawning
0 mon.xxx@0(probing) e3 respawn

Removing the port value on the 'ceph mon add' command allows to add
a monitor with the msgr configuration defined in the ceph config.

This change is nautilus+ only.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1719013
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1722066

Signed-off-by: Dimitri Savineau dsavinea@redhat.com

@dsavineau dsavineau added bug backport-stable-4.0 Backports for stable Nautilus branch labels Jun 27, 2019
@dsavineau dsavineau requested review from leseb and guits June 27, 2019 21:19
@dsavineau dsavineau force-pushed the ceph_mon_add_v2_v1 branch 2 times, most recently from 691d3b8 to 83c4895 Compare June 27, 2019 21:22
@dsavineau dsavineau added the DNM label Jun 27, 2019
@guits
Copy link
Collaborator

guits commented Jun 27, 2019

@leseb couldn't it be a breaking change for existing deployment where $MON_PORT would have been previously overridden ?

@leseb
Copy link
Member

leseb commented Jun 27, 2019

@guits do we allow that in ceph-ansible?

@guits
Copy link
Collaborator

guits commented Jun 28, 2019

@leseb yes, I guess using ceph_mon_docker_extra_env: -e MON_PORT=1234 would allow it.

@leseb
Copy link
Member

leseb commented Jun 28, 2019

Even though we specify a different port, I don't think this will actually work, (the mon map does not have that port). If this is supposed to work in ceph-ansible then the mon endpoint should have that vars, I don't think it does.

When bootstrapping an initial cluster, we don't need to add the mon
to the monmap via 'ceph mon add' command.
But when we want to scale up the monitors then we need it.
Currently the new mon added will be out of quorum because the addr
in the ceph config and in the monmap won't match.

start_mon(): ceph --cluster ceph mon add xxx 192.168.100.5:3300
mon.xxx at v2:192.168.100.5:3300/0 already exists

Even if the ceph conf uses v2+v1 then the new mon will have only v2
address configured in the monmap creating a conflict.

0 mon.xxx@0(probing) e3  monmap addrs for rank 4 changed, i am
  [v2:192.168.100.5:3300/0,v1:192.168.100.5:6789/0], monmap is
  v2:192.168.100.5:3300/0, respawning
0 mon.xxx@0(probing) e3 respawn

Removing the port value on the 'ceph mon add' command allows to add
a monitor with the msgr configuration defined in the ceph config.

This change is nautilus+ only.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1719013
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1722066

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
@fultonj
Copy link

fultonj commented Jul 2, 2019

@dsavineau @guits
The scratch container with this change [1] resolved our issue of scaling from 3 to 5 mons.

[1] docker.io/ceph/daemon:bz1719013-nautilus-centos-7-x86_64

@yorabl
Copy link

yorabl commented Jul 2, 2019

@fultonj is right, the scale up of the monitors was successful, but the scale up of rgw services failed

@dsavineau
Copy link
Contributor Author

@fultonj @yorabl thanks for the update !

@dsavineau dsavineau merged commit 8763b14 into ceph:master Jul 3, 2019
@dsavineau dsavineau deleted the ceph_mon_add_v2_v1 branch July 3, 2019 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-stable-4.0 Backports for stable Nautilus branch bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants