New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tools: Backport ceph-objectstore-tool changes to firefly #3866

Merged
97 commits merged into from Mar 16, 2015

Conversation

Projects
None yet
8 participants
@dzafman
Member

dzafman commented Mar 4, 2015

http://tracker.ceph.com/issues/12327

This passes unit test and the ceph-qa-suite test tasks/ceph_objectstore_tool.py.

Currently scheduled rados:thrash suite

Requires ceph/ceph-qa-suite#361 to test.

dalgaaf and others added some commits May 7, 2014

ceph_filestore_dump.cc: cleanup includes
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 8620609)
ceph_filestore_dump: Remove unused bufferlist databl
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 398b418)
ceph_filestore_dump: Export omap in batches for large omap case
New function get_omap_batch()
Create a TYPE_OMAP section for each batch

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 501dd3c)
ceph_filestore_dump: Use cerr now that we aren't closing stderr
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 087c0f9)
common,ceph_filestore_dump: Add ability for utilities to suppress lib…
…rary dout output

Suppress dout output with CODE_ENVIRONMENT_UTILITY_NODOUT
ceph_filestore_dump turns on dout output if --debug specified
When used it can still be enable with --log-to-stderr --err-to-stderr

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit f7f9b25)
common: Add missing ghobject_t::decode() for json
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit c05f895)

Changes:
	Adjusted for older shard_t
ceph_filestore_dump: Add --skip-journal-replay and --skip-mount-omap
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 2e9dcb2)
os: Add optional flags to generic ObjectStore creation (SKIP_JOURNAL_…
…REPLAY

and SKIP_MOUNT_OMAP)

Only FileStore cares about these flags, so passed on during create()

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3d9fde9)

Conflicts:
	src/os/FileStore.cc
ceph_filestore_dump: Improve debug output by showing actual offset
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 44b261d)
ceph_filstore_dump: Save if stdout is a tty and add routine to clean …
…binary strings

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3a574cc)
ceph_filestore_dump: Add --type list to output objects in a pg in json
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 844dabb)
ceph_filestore_dump: Add remove object command and usage for new comm…
…ands

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 605caec)
ceph_filestore_dump: Add list-omap object command
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 48890c7)
ceph_filestore_dump: Add get-bytes object command
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 869dd92)
ceph_filestore_dump: Add set-bytes object command
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3c24d1f)
ceph_filestore_dump: Add list-attrs object command
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 00c6b06)
ceph_filestore_dump: Add get-attr object command
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 55d43c0)
ceph_filestore_dump: Fail import/export with a tty for stdin/stdout
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 7520e50)
ceph_filestore_dump: Add set-attr object command
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 95554e0)
ceph_filestore_dump: Add rm-attr object command
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 465d777)
ceph_filestore_dump: Add get-omap object command
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit b50c43c)
ceph_filestore_dump: Add set-omap object command
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 50cd57e)
ceph_filestore_dump: Add rm-omap object command
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 0fc6bd2)
ceph_filestore_dump: Add get-omaphdr object command
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 30c0f31)
ceph_filestore_dump: Add set-omaphdr object command
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit b4d95cc)
ceph_filestore_dump: Add utility function get_fd_data()
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit d4a9daf)
Renames and removal towards a unified ceph_objectstore_tool
Rename ceph_filestore_dump.cc and ceph_filestore_dump.py
Remove ceph_filestore_tool.cc

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 7786419)
Complete replacement of ceph_filestore_tool and ceph_filestore_dump
with unified ceph_objectstore_tool

Move list-lost-objects and fix-lost-objects features from
  ceph_filestore_tool to ceph_objectstore_tool as list-lost, fix-lost
Change --type to --op for info, log, export...operations
Add --type for the ObjectStore type (defaults to filestore)
Change --filestore-path to --data-path
Update installation, Makefile.am, and .gitignore
Fix and rename test case to match
  Add some additional invalid option checks

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 83fbc91)
ceph_objectstore_tool, test: Add list-pgs operations and unit test case
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit f01e334)
ceph_objectstore_tool, test: Implement import-rados feature and unit …
…test code

Fixes: #8276

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 23ec93a)

dzafman and others added some commits Oct 9, 2014

osd: Get pgid ancestor from last_map when building past intervals
Fixed OSD::build_past_intervals_parallel() and PG::generate_past_intervals()

Fixes: #10430

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0c5b66d)

Conflicts:
	src/osd/OSD.cc
ceph_objectstore_tool: Verify that object pool and pgid pool match
Also, earlier check for invalid --pgid with import op

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit da3be80)
ceph_objectstore_tool: Handle import of pg which OSD has split
Fail import if import data doesn't include OSDMap and can't find it locally
See if local map can be read for import's epoch
Jump to current epoch like a split would if local map not present

Fixes: #9781

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit afda6e4)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
ceph_objectstore_tool: On import following a split skip objects no lo…
…nger in pg

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 68b27e2)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
osd: build_push_op() handle short reads so recipient doesn't crash
Fixes: #8121

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b9a0484)
osd: Pass oldest_map when generating past intervals
From load_pgs() the superblock hasn't been published yet
so we need to retrieve the value of oldest_map depending on the context.

Fixes: #10427

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 7fb721c)

Conflicts:
	src/osd/OSD.cc
ceph_objectstore_tool: For import get object_info_t available for eac…
…h object

Add object_info_t to object_begin so we have at object create time
This will be useful for importing from multiple erasure coded exports

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 05d916e)
ceph_objectstore_tool: fix check_output on python2.6
* backported the subprocess.check_output from python2.7

Fixes: #10756

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 15350a0)

Conflicts:
	src/test/ceph_objectstore_tool.py
@loic-bot

This comment has been minimized.

loic-bot commented Mar 4, 2015

SUCCESS: the output of run-make-check.sh on centos-7 for 9938595 is http://paste2.org/XBZYML9x

:octocat: Sent from GH.

@dzafman

This comment has been minimized.

Member

dzafman commented Mar 5, 2015

rados:thrash results
139 PASSED
26 fail wget failures - infrastructure
1 dead - timeout trying to copy an export file - infrastructure
1 fail - unexplained ceph-objectstore-tool SIGSEGV doing list-pgs (no core or log found)
1 running hung because /dev/sdg failed on burnupi50

@ghost ghost assigned ghost and unassigned athanatos Mar 10, 2015

@ghost ghost added core and removed needs-qa labels Mar 10, 2015

@ghost

This comment has been minimized.

ghost commented Mar 10, 2015

@dzafman it's in the firefly-backport integration branch now

@ghost ghost assigned dzafman and unassigned ghost Mar 11, 2015

@ghost ghost added this to the firefly milestone Mar 11, 2015

@ghost

This comment has been minimized.

ghost commented Mar 11, 2015

@dzafman a rados suite came back green ( http://tracker.ceph.com/issues/11090#teuthology-run-b0d9b862d7f1a9a98cf4b41ff7cb0eb428740633-march-2015 ). Good to merge in my opinion. Since it's a very large patch series, could you confirm that it's ready to merge in your opinion ?

@dzafman

This comment has been minimized.

Member

dzafman commented Mar 11, 2015

@dachary @athanatos The only issue I found in my testing using the ceph/ceph-qa-suite#361 is a race with pool deletion causing a ceph-objectstore-tool --op list-pgs to SIGSEGV. I'll file a bug report. This is ready to merge subject to code review. I think Sam should look at it, but another set of eyes would be nice.

A suggested way to code review would be as follows. Examine all changes NOT in src/tools/ceph_objectstore_tool.cc. Then check-out the differences in the tool with git diff hammer wip-cot-firefly -- src/tools/ceph_objectstore_tool.cc. All the changes should be to newer features/changes like map CRC checking, removal changes, META_COLL, NO_SHARD etc...

ghost pushed a commit that referenced this pull request Mar 16, 2015

Merge pull request #3866 from ceph/wip-cot-firefly
Backport ceph-objectstore-tool changes to firefly

Reviewed-by: Loic Dachary <ldachary@redhat.com>

@ghost ghost merged commit e79e5f8 into firefly Mar 16, 2015

@dzafman dzafman deleted the wip-cot-firefly branch Mar 18, 2015

@ghost ghost changed the title from Backport ceph-objectstore-tool changes to firefly to tools: Backport ceph-objectstore-tool changes to firefly Jul 14, 2015

smithfarm added a commit to SUSE/ceph that referenced this pull request Sep 7, 2015

ceph_objectstore_tool.cc: change ghobject_t::NO_SHARD to shard_id_t::…
…NO_SHARD

This commit extends the changes from dedd8e2 to
src/tools/ceph_objectstore_tool.cc, which was backported to firefly
(after dedd8e2) by ceph#3866

Signed-off-by: Nathan Cutler <ncutler@suse.com>

smithfarm added a commit to SUSE/ceph that referenced this pull request Oct 8, 2015

ceph_objectstore_tool.cc: change ghobject_t::NO_SHARD to shard_id_t::…
…NO_SHARD

This commit extends the changes from dedd8e2 to
src/tools/ceph_objectstore_tool.cc, which was backported to firefly
(after dedd8e2) by ceph#3866

Signed-off-by: Nathan Cutler <ncutler@suse.com>

This issue was closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment