tools: add a tool to restore crush map after a faulty one is injected #5052

tchaikov · 2015-06-23T09:26:39Z

add "--dump " command to osdmaptool, so we are able to extract osdmap epoch using xmlstarlet
add a command to ceph-monstore-tool to rewrite monstore
add a script to help fix the crush map

see http://tracker.ceph.com/issues/11815

ghost · 2015-06-23T09:50:27Z

loic-bot says FAIL: http://jenkins.ceph.dachary.org/job/ceph/6032/LABELS=centos-7&&x86_64/console

tchaikov · 2015-06-23T09:53:09Z

thanks @dachary, i just fixed the script's path, and also added it to the dist tarball.

loic-bot · 2015-06-23T10:10:50Z

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/6033/

FAILURE http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/6033/

tchaikov · 2015-06-23T13:51:37Z

@jecluis could you take a quick look at this change?

packed all the changes in a proposal, and put it into a new paxos commit. but i didn't update the infected osdmap epoches as you suggested, since the paxos commit will be applied on all monitors, so the bad epoches on leader will be fixed once the proposal is accepted.
added a new osdmap commit, so the OSDMonitor will update its current osdmap at seeing it.

thanks =)

loic-bot · 2015-06-23T20:28:54Z

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/6048/

FAILURE http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/6048/

loic-bot · 2015-06-24T04:59:42Z

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/6057/

FAILURE http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/6057/

loic-bot · 2015-06-24T15:39:25Z

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/6073/

FAILURE http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/6073/

loic-bot · 2015-06-24T16:55:07Z

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/6075/

tchaikov · 2015-06-24T17:00:10Z

the failure was due to:

2015-06-24 16:42:15.427958 2b22a8bf2bc0 -1 mon.a@-1(probing).osd e3 update_from_paxos full map CRC mismatch, resetting to canonical

loic-bot · 2015-06-25T06:01:24Z

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/6090/

tchaikov · 2015-07-07T15:52:55Z

per the discussion tonight, we should also offer a command, with which user can extract both the full map and the incrementals, which can be injected to the OSD's object store. we can do it as part of #5127.

loic-bot · 2015-07-13T13:58:21Z

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/6389/

loic-bot · 2015-07-14T22:33:50Z

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/6426/

tchaikov · 2015-07-15T07:30:19Z

@jecluis i managed to workaround the "local" command.

$ help local
...
Exit Status:
Returns success unless an invalid option is supplied, a variable
assignment error occurs, or the shell is not executing a function.

so "local" is practically always happy at whatever the following $() subshell returns, and what we need is to put the assignment at a different line.

following is what i have if the monitor is running:

$ ./tools/ceph-monstore-update-crush.sh dev/mon.a/
IO error: lock dev/mon.a/store.db/LOCK: Resource temporarily unavailable

error accessing mon store at dev/mon.a/

loic-bot · 2015-07-15T10:41:47Z

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/6444/

tchaikov · 2015-07-15T11:53:54Z

FAIL: test/libradosstriper/rados-striper.sh

the failed test is not relevant .

ghost · 2015-07-15T12:30:35Z

@tchaikov could you please rebase and repush ? It's the first time this rados-striper.sh error happens and I'd like to make sure it's a transient error.

tchaikov · 2015-07-15T14:09:34Z

@dachary rebased and repushed.

loic-bot · 2015-07-15T19:26:56Z

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/6455/

"rewrite" command will - add a new osdmap version to update current osdmap held by OSDMonitor - add a new paxos version, as a proposal it will * rewrite all osdmap epochs from specified epoch to the last_committed one with the specified crush map. * add the new osdmap which is added just now so the leader monitor can trigger a recovery process to apply the transaction to all monitors in quorum, and hence bring them back to normal after being injected with a faulty crushmap. Fixes: #11815 Signed-off-by: Kefu Chai <kchai@redhat.com>

* --dump will accept a formatter argument. Signed-off-by: Kefu Chai <kchai@redhat.com>

* its '--dump-json' option is replaced by '--dump json' Signed-off-by: Kefu Chai <kchai@redhat.com>

Fixes: #11815 Signed-off-by: Kefu Chai <kchai@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

tchaikov · 2015-07-17T11:19:32Z

rebased against master to resolve the merge conflicts of PendingReleaseNotes .

@jecluis ping?

jecluis · 2015-07-17T13:02:16Z

Let's wait for the bot to come back with a clean build and I'll merge it then ;)

tchaikov · 2015-07-17T15:05:52Z

cool, thanks in advance! =)

loic-bot · 2015-07-17T15:48:09Z

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/6511/

tools: add a tool to restore crush map after a faulty one is injected Reviewed-by: Joao Eduardo Luis <joao@suse.de>

tchaikov added bug-fix tools labels Jun 23, 2015

tchaikov force-pushed the wip-11815-restore-crushmap branch from 0229aab to b879531 Compare June 23, 2015 09:47

tchaikov changed the title ~~tools: add a tool to restore crush map after a faulty one is injected~~ [DNM] tools: add a tool to restore crush map after a faulty one is injected Jun 23, 2015

tchaikov force-pushed the wip-11815-restore-crushmap branch 2 times, most recently from 79d1874 to 3c6fc77 Compare June 23, 2015 13:04

tchaikov force-pushed the wip-11815-restore-crushmap branch from 3c6fc77 to bd8f3a1 Compare June 23, 2015 17:08

tchaikov force-pushed the wip-11815-restore-crushmap branch 3 times, most recently from 9fc5f93 to eda9844 Compare June 24, 2015 04:05

tchaikov force-pushed the wip-11815-restore-crushmap branch 2 times, most recently from 0e78e54 to b7cddf8 Compare June 24, 2015 14:22

tchaikov force-pushed the wip-11815-restore-crushmap branch from b7cddf8 to 099b18f Compare June 24, 2015 15:54

tchaikov changed the title ~~[DNM] tools: add a tool to restore crush map after a faulty one is injected~~ tools: add a tool to restore crush map after a faulty one is injected Jun 24, 2015

tchaikov changed the title ~~tools: add a tool to restore crush map after a faulty one is injected~~ [DNM] tools: add a tool to restore crush map after a faulty one is injected Jun 24, 2015

tchaikov force-pushed the wip-11815-restore-crushmap branch from 099b18f to 657ba88 Compare June 25, 2015 04:38

tchaikov force-pushed the wip-11815-restore-crushmap branch 3 times, most recently from a961462 to 2e362d3 Compare June 25, 2015 08:01

tchaikov changed the title ~~[DNM] tools: add a tool to restore crush map after a faulty one is injected~~ tools: add a tool to restore crush map after a faulty one is injected Jun 25, 2015

tchaikov mentioned this pull request Jul 7, 2015

tools: add "set-osdmap" command to ceph-objectstore-tool #5127

Merged

tchaikov force-pushed the wip-11815-restore-crushmap branch from 5eb38c8 to a87d2ac Compare July 15, 2015 07:19

tchaikov force-pushed the wip-11815-restore-crushmap branch from a87d2ac to 391f551 Compare July 15, 2015 14:08

tchaikov and others added 10 commits July 17, 2015 19:14

tool/osdmaptool: replace --dump-json with --dump

ad6e6a1

* --dump will accept a formatter argument. Signed-off-by: Kefu Chai <kchai@redhat.com>

PendingReleaseNotes: add the notes about osdmaptool

1bb0cf4

* its '--dump-json' option is replaced by '--dump json' Signed-off-by: Kefu Chai <kchai@redhat.com>

tools: add ceph-monstore-update-crush.sh

39e25b9

Fixes: #11815 Signed-off-by: Kefu Chai <kchai@redhat.com>

test: add a test to exercise ceph-monstore-update-crush.sh

9d8b6d8

Fixes: #11815 Signed-off-by: Kefu Chai <kchai@redhat.com>

package ceph-monstore-update-crush.sh

50a33de

Fixes: #11815 Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSDMap: remove unused dump_json()

fbd4d12

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSDMap: fix a typo in the comment

2349eb9

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools: ceph_monstore_tool: describe behavior of rewrite command

a881f93

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

tools: ceph-monstore-update-crush: fail early if mon's running

8eb9a34

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

tchaikov force-pushed the wip-11815-restore-crushmap branch from 391f551 to 8eb9a34 Compare July 17, 2015 11:16

jecluis added a commit that referenced this pull request Jul 17, 2015

Merge pull request #5052 from ceph/wip-11815-restore-crushmap

a89cae4

tools: add a tool to restore crush map after a faulty one is injected Reviewed-by: Joao Eduardo Luis <joao@suse.de>

jecluis merged commit a89cae4 into master Jul 17, 2015

jecluis deleted the wip-11815-restore-crushmap branch July 17, 2015 15:50

tools: add a tool to restore crush map after a faulty one is injected #5052

tools: add a tool to restore crush map after a faulty one is injected #5052

tchaikov commented Jun 23, 2015

ghost commented Jun 23, 2015

tchaikov commented Jun 23, 2015

loic-bot commented Jun 23, 2015

tchaikov commented Jun 23, 2015

loic-bot commented Jun 23, 2015

loic-bot commented Jun 24, 2015

loic-bot commented Jun 24, 2015

loic-bot commented Jun 24, 2015

tchaikov commented Jun 24, 2015

loic-bot commented Jun 25, 2015

tchaikov commented Jul 7, 2015

loic-bot commented Jul 13, 2015

loic-bot commented Jul 14, 2015

tchaikov commented Jul 15, 2015

loic-bot commented Jul 15, 2015

tchaikov commented Jul 15, 2015

ghost commented Jul 15, 2015

tchaikov commented Jul 15, 2015

loic-bot commented Jul 15, 2015

tchaikov commented Jul 17, 2015

jecluis commented Jul 17, 2015

tchaikov commented Jul 17, 2015

loic-bot commented Jul 17, 2015

tools: add a tool to restore crush map after a faulty one is injected #5052

tools: add a tool to restore crush map after a faulty one is injected #5052

Conversation

tchaikov commented Jun 23, 2015

ghost commented Jun 23, 2015

tchaikov commented Jun 23, 2015

loic-bot commented Jun 23, 2015

tchaikov commented Jun 23, 2015

loic-bot commented Jun 23, 2015

loic-bot commented Jun 24, 2015

loic-bot commented Jun 24, 2015

loic-bot commented Jun 24, 2015

tchaikov commented Jun 24, 2015

loic-bot commented Jun 25, 2015

tchaikov commented Jul 7, 2015

loic-bot commented Jul 13, 2015

loic-bot commented Jul 14, 2015

tchaikov commented Jul 15, 2015

loic-bot commented Jul 15, 2015

tchaikov commented Jul 15, 2015

ghost commented Jul 15, 2015

tchaikov commented Jul 15, 2015

loic-bot commented Jul 15, 2015

tchaikov commented Jul 17, 2015

jecluis commented Jul 17, 2015

tchaikov commented Jul 17, 2015

loic-bot commented Jul 17, 2015