allow adding a monitor to a deployed cluster #3547

rishabh-d-dave · 2019-01-28T12:31:48Z

This (early) PR aims to add a playbook, add-mon.yml, to infrastructure-playbooks for adding a monitor to already deployed Ceph cluster.

rishabh-d-dave · 2019-01-29T09:35:35Z

Changelog:

importing ceph-facts role before ceph-common (otherwise add-mon.yml would fail at this task[1], since monitor_keyring_conf[2] would be undefined for it; ceph-facts defines monitor_keyring_conf)
removed the commit containing tasks added for debugging

rishabh-d-dave · 2019-01-29T12:16:13Z

removed the changes concerning gather_facts to true helped
made improvements suggested by yamllint

tests/functional/add-mons/container/vagrant_variables.yml

tox.ini

rishabh-d-dave · 2019-01-30T09:33:10Z

made the changes Guillaume mentioned

tox.ini

leseb

What about the containerized equivalent?

leseb · 2019-01-30T12:47:05Z

roles/ceph-mon/tasks/deploy_monitors.yml

@@ -1,4 +1,12 @@
 ---
+- name: check if monitor initial keyring already exists
+  command: ceph auth get-key mon.


Suggested change

command: ceph auth get-key mon.

command: ceph --cluster {{ cluster }} auth get-key mon.

@leseb is ceph-ansible still supporting custom cluster names? (why?)

Yes, it is, for OSP and for existing clusters too.

rishabh-d-dave · 2019-01-30T16:09:05Z

What about the containerized equivalent?

Once I get this done for non-containers, I'll focus on containerized equivalent as well.

rishabh-d-dave · 2019-02-01T07:21:01Z

running ceph-config and other roles required for it on osd to update ceph.conf
rearranged the playbook to be more succinct.

rishabh-d-dave · 2019-04-10T05:18:49Z

@guits @dsavineau
The jobs corresponding to the tox scenarios added by this PR have passed invariably for last 4-5 times. IMO, this PR is review-ready.

requested changes have been addressed

guits · 2019-04-10T08:36:56Z

jenkins test dev-centos-container-add_mons

guits · 2019-04-10T08:37:03Z

jenkins test dev-centos-non_container-add_mons

infrastructure-playbooks/add-mon.yml

guits · 2019-04-10T08:56:37Z

jenkins test dev-centos-non_container-add_mons

guits · 2019-04-10T08:56:44Z

jenkins test dev-centos-container-add_mons

guits · 2019-04-10T08:56:54Z

jenkins test dev-centos-container-all_daemons

guits · 2019-04-10T08:57:02Z

jenkins test dev-centos-non_container-all_daemons

guits · 2019-04-10T11:35:46Z

roles/ceph-mon/tasks/deploy_monitors.yml

+    - initial_mon_key.skipped is defined
+    - ceph_current_status.fsid is undefined
+
+- name: debugtask


guits · 2019-04-10T11:35:53Z

roles/ceph-mon/tasks/deploy_monitors.yml


-  # add code to read the key or/and try to find it on other nodes
+- name: debugtask


guits · 2019-04-10T11:36:00Z

roles/ceph-mon/tasks/deploy_monitors.yml


-  # add code to read the key or/and try to find it on other nodes
+- name: debugtask
+  debug:


When adding a new monitor, we must reuse the existing initial monitor keyring. Otherwise, the new monitor will issue its 'mkfs' with a new monitor keyring and it will result with a mismatch between them. The new monitor will be unable to join the quorum in the end. Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com> Co-authored-by: Rishabh Dave <ridave@redhat.com>

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Add a playbook that deploys a new monitor on a new node, adds that node to the Ceph cluster and the monitor to the quorum and updates the ceph configuration file on OSD nodes. Signed-off-by: Rishabh Dave <ridave@redhat.com>

guits · 2019-04-10T11:46:45Z

jenkins test dev-centos-non_container-all_daemons

guits · 2019-04-10T11:46:53Z

jenkins test dev-centos-container-all_daemons

guits · 2019-04-10T11:47:08Z

jenkins test dev-centos-non_container-add_mons

guits · 2019-04-10T11:47:22Z

jenkins test dev-centos-container-add_mons

rishabh-d-dave changed the title ~~allow adding a monitor to a deployed cluster~~ [skip ci] allow adding a monitor to a deployed cluster Jan 28, 2019

rishabh-d-dave requested review from guits and leseb January 28, 2019 12:32

rishabh-d-dave added DNM Do NOT merge feature labels Jan 28, 2019

rishabh-d-dave force-pushed the add-mon branch 4 times, most recently from 19f8f3e to 1be7ba8 Compare January 29, 2019 09:27

rishabh-d-dave force-pushed the add-mon branch 2 times, most recently from d084d23 to a73091e Compare January 29, 2019 12:15

guits requested changes Jan 29, 2019

View reviewed changes

rishabh-d-dave force-pushed the add-mon branch 2 times, most recently from 3a438de to 3a604d2 Compare January 30, 2019 09:29

rishabh-d-dave force-pushed the add-mon branch 2 times, most recently from cfe2186 to bdf1bd3 Compare January 30, 2019 10:11

guits requested changes Jan 30, 2019

View reviewed changes

tox.ini Outdated Show resolved Hide resolved

guits requested changes Jan 30, 2019

View reviewed changes

tox.ini Outdated Show resolved Hide resolved

rishabh-d-dave force-pushed the add-mon branch 2 times, most recently from 296e7f8 to 5533f77 Compare January 30, 2019 11:21

leseb previously requested changes Jan 30, 2019

View reviewed changes

leseb added this to the v4.0 milestone Jan 30, 2019

rishabh-d-dave force-pushed the add-mon branch 3 times, most recently from cf5c187 to 560f42c Compare February 1, 2019 07:20

guits changed the title ~~[skip ci]allow adding a monitor to a deployed cluster~~ allow adding a monitor to a deployed cluster Apr 10, 2019

guits reviewed Apr 10, 2019

View reviewed changes

infrastructure-playbooks/add-mon.yml Show resolved Hide resolved

guits changed the title ~~allow adding a monitor to a deployed cluster~~ [skip ci] allow adding a monitor to a deployed cluster Apr 10, 2019

guits added the backport-stable-4.0 label Apr 10, 2019

rishabh-d-dave force-pushed the add-mon branch 2 times, most recently from a45a2b3 to 8f2f9fb Compare April 10, 2019 08:50

guits changed the title ~~[skip ci] allow adding a monitor to a deployed cluster~~ allow adding a monitor to a deployed cluster Apr 10, 2019

guits requested changes Apr 10, 2019

View reviewed changes

guits changed the title ~~allow adding a monitor to a deployed cluster~~ [skip ci] allow adding a monitor to a deployed cluster Apr 10, 2019

guits and others added 3 commits April 10, 2019 17:13

check if mon daemon is installed before restarting it

b80abe2

Signed-off-by: Rishabh Dave <ridave@redhat.com>

allow adding a monitor to a deployed cluster

508930d

Add a playbook that deploys a new monitor on a new node, adds that node to the Ceph cluster and the monitor to the quorum and updates the ceph configuration file on OSD nodes. Signed-off-by: Rishabh Dave <ridave@redhat.com>

rishabh-d-dave force-pushed the add-mon branch from 8f2f9fb to 508930d Compare April 10, 2019 11:44

guits changed the title ~~[skip ci] allow adding a monitor to a deployed cluster~~ allow adding a monitor to a deployed cluster Apr 10, 2019

guits approved these changes Apr 10, 2019

View reviewed changes

guits merged commit d5967af into ceph:master Apr 15, 2019

mergify bot mentioned this pull request Apr 15, 2019

Automatic backport of pull request #3547 #3859

Merged

rishabh-d-dave deleted the add-mon branch April 15, 2019 08:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

allow adding a monitor to a deployed cluster #3547

allow adding a monitor to a deployed cluster #3547

rishabh-d-dave commented Jan 28, 2019 •

edited

rishabh-d-dave commented Jan 29, 2019

rishabh-d-dave commented Jan 29, 2019

rishabh-d-dave commented Jan 30, 2019

leseb left a comment

leseb Jan 30, 2019

alfredodeza Jan 30, 2019

leseb Jan 30, 2019

rishabh-d-dave commented Jan 30, 2019

rishabh-d-dave commented Feb 1, 2019

rishabh-d-dave commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits Apr 10, 2019

guits Apr 10, 2019

guits Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

	command: ceph auth get-key mon.
	command: ceph --cluster {{ cluster }} auth get-key mon.


		# add code to read the key or/and try to find it on other nodes
		- name: debugtask

allow adding a monitor to a deployed cluster #3547

allow adding a monitor to a deployed cluster #3547

Conversation

rishabh-d-dave commented Jan 28, 2019 • edited

rishabh-d-dave commented Jan 29, 2019

rishabh-d-dave commented Jan 29, 2019

rishabh-d-dave commented Jan 30, 2019

leseb left a comment

Choose a reason for hiding this comment

leseb Jan 30, 2019

Choose a reason for hiding this comment

alfredodeza Jan 30, 2019

Choose a reason for hiding this comment

leseb Jan 30, 2019

Choose a reason for hiding this comment

rishabh-d-dave commented Jan 30, 2019

rishabh-d-dave commented Feb 1, 2019

rishabh-d-dave commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits Apr 10, 2019

Choose a reason for hiding this comment

guits Apr 10, 2019

Choose a reason for hiding this comment

guits Apr 10, 2019

Choose a reason for hiding this comment

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

guits commented Apr 10, 2019

rishabh-d-dave commented Jan 28, 2019 •

edited