Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tools: Backport ceph-objectstore-tool changes to firefly #3866

Merged
97 commits merged into from Mar 16, 2015
Merged

tools: Backport ceph-objectstore-tool changes to firefly #3866

97 commits merged into from Mar 16, 2015

Conversation

dzafman
Copy link
Contributor

@dzafman dzafman commented Mar 4, 2015

http://tracker.ceph.com/issues/12327

This passes unit test and the ceph-qa-suite test tasks/ceph_objectstore_tool.py.

Currently scheduled rados:thrash suite

Requires ceph/ceph-qa-suite#361 to test.

dalgaaf and others added 30 commits March 3, 2015 09:51
@dalgaaf @dzafman
ceph_filestore_dump.cc: cleanup includes
4a742fe 
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 8620609)
@dzafman
ceph_filestore_dump: Remove unused bufferlist databl
0b757af 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 398b418)
@dzafman
ceph_filestore_dump: Export omap in batches for large omap case
3f4cabd 
New function get_omap_batch()
Create a TYPE_OMAP section for each batch

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 501dd3c)
@dzafman
common,ceph_filestore_dump: Add ability for utilities to suppress lib…
f224429 
…rary dout output

Suppress dout output with CODE_ENVIRONMENT_UTILITY_NODOUT
ceph_filestore_dump turns on dout output if --debug specified
When used it can still be enable with --log-to-stderr --err-to-stderr

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit f7f9b25)
@dzafman
ceph_filestore_dump: Use cerr now that we aren't closing stderr
1164b2e 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 087c0f9)
@dzafman
ceph_filestore_dump: Improve debug output by showing actual offset
3381aeb 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 44b261d)
@dzafman
os: Add optional flags to generic ObjectStore creation (SKIP_JOURNAL_…
c636998 
…REPLAY

and SKIP_MOUNT_OMAP)

Only FileStore cares about these flags, so passed on during create()

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3d9fde9)

Conflicts:
	src/os/FileStore.cc
@dzafman
ceph_filestore_dump: Add --skip-journal-replay and --skip-mount-omap
dadecb1 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 2e9dcb2)
@dzafman
common: Add missing ghobject_t::decode() for json
d4aedeb 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit c05f895)

Changes:
	Adjusted for older shard_t
@dzafman
ceph_filstore_dump: Save if stdout is a tty and add routine to clean …
9816f87 
…binary strings

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3a574cc)
@dzafman
ceph_filestore_dump: Fail import/export with a tty for stdin/stdout
62dc823 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 7520e50)
@dzafman
ceph_filestore_dump: Add utility function get_fd_data()
a469464 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit d4a9daf)
@dzafman
ceph_filestore_dump: Add remove object command and usage for new comm…
7cda8b7 
…ands

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 605caec)
@dzafman
ceph_filestore_dump: Add --type list to output objects in a pg in json
aff2c99 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 844dabb)
@dzafman
ceph_filestore_dump: Add list-attrs object command
303e4ce 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 00c6b06)
@dzafman
ceph_filestore_dump: Add list-omap object command
fcabb81 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 48890c7)
@dzafman
ceph_filestore_dump: Add get-bytes object command
2f1926d 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 869dd92)
@dzafman
ceph_filestore_dump: Add set-bytes object command
55c21b8 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3c24d1f)
@dzafman
ceph_filestore_dump: Add get-attr object command
6724da8 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 55d43c0)
@dzafman
ceph_filestore_dump: Add set-attr object command
7bd2dd3 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 95554e0)
@dzafman
ceph_filestore_dump: Add rm-attr object command
f391feb 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 465d777)
@dzafman
ceph_filestore_dump: Add get-omap object command
7547f3d 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit b50c43c)
@dzafman
ceph_filestore_dump: Add set-omap object command
20165d1 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 50cd57e)
@dzafman
ceph_filestore_dump: Add rm-omap object command
271e3ca 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 0fc6bd2)
@dzafman
ceph_filestore_dump: Add get-omaphdr object command
c7d261d 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 30c0f31)
@dzafman
ceph_filestore_dump: Add set-omaphdr object command
9ee2c27 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit b4d95cc)
@dzafman
Renames and removal towards a unified ceph_objectstore_tool
a42273e 
Rename ceph_filestore_dump.cc and ceph_filestore_dump.py
Remove ceph_filestore_tool.cc

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 7786419)
@dzafman
Complete replacement of ceph_filestore_tool and ceph_filestore_dump
edc9f71 
with unified ceph_objectstore_tool

Move list-lost-objects and fix-lost-objects features from
  ceph_filestore_tool to ceph_objectstore_tool as list-lost, fix-lost
Change --type to --op for info, log, export...operations
Add --type for the ObjectStore type (defaults to filestore)
Change --filestore-path to --data-path
Update installation, Makefile.am, and .gitignore
Fix and rename test case to match
  Add some additional invalid option checks

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 83fbc91)
@dzafman
ceph_objectstore_tool, test: Add list-pgs operations and unit test case
5e3f89e 
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit f01e334)
@dzafman
ceph_objectstore_tool: Minor improvements
6352907 
Make all non-error non-debug output to stdout
Fix a message

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit a2bd2aa)
dzafman and others added 8 commits March 3, 2015 11:21
@dzafman
ceph_objectstore_tool: Verify that object pool and pgid pool match
c3de607 
Also, earlier check for invalid --pgid with import op

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit da3be80)
@dzafman
ceph_objectstore_tool: On import following a split skip objects no lo…
aedd324 
…nger in pg

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 68b27e2)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
@dzafman
ceph_objectstore_tool: Handle import of pg which OSD has split
f70590b 
Fail import if import data doesn't include OSDMap and can't find it locally
See if local map can be read for import's epoch
Jump to current epoch like a split would if local map not present

Fixes: #9781

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit afda6e4)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
@dzafman
ceph_objectstore_tool: For import get object_info_t available for eac…
d5d34dd 
…h object

Add object_info_t to object_begin so we have at object create time
This will be useful for importing from multiple erasure coded exports

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 05d916e)
@tchaikov @dzafman
ceph_objectstore_tool: fix check_output on python2.6
7ce0cb8 
* backported the subprocess.check_output from python2.7

Fixes: #10756

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 15350a0)

Conflicts:
	src/test/ceph_objectstore_tool.py
@dzafman
osd: build_push_op() handle short reads so recipient doesn't crash
9235d78 
Fixes: #8121

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b9a0484)
@dzafman
osd: Pass oldest_map when generating past intervals
c09b6d9 
From load_pgs() the superblock hasn't been published yet
so we need to retrieve the value of oldest_map depending on the context.

Fixes: #10427

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 7fb721c)

Conflicts:
	src/osd/OSD.cc
@dzafman
osd: Get pgid ancestor from last_map when building past intervals
91b2aca 
Fixed OSD::build_past_intervals_parallel() and PG::generate_past_intervals()

Fixes: #10430

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0c5b66d)

Conflicts:
	src/osd/OSD.cc
@loic-bot
Copy link

loic-bot commented Mar 4, 2015

SUCCESS: the output of run-make-check.sh on centos-7 for 9938595 is http://paste2.org/XBZYML9x

:octocat: Sent from GH.

@dzafman
Copy link
Contributor Author

dzafman commented Mar 5, 2015

rados:thrash results
139 PASSED
26 fail wget failures - infrastructure
1 dead - timeout trying to copy an export file - infrastructure
1 fail - unexplained ceph-objectstore-tool SIGSEGV doing list-pgs (no core or log found)
1 running hung because /dev/sdg failed on burnupi50

@ghost ghost assigned ghost and unassigned athanatos Mar 10, 2015
@ghost ghost added core and removed needs-qa labels Mar 10, 2015
@ghost
Copy link

ghost commented Mar 10, 2015

@dzafman it's in the firefly-backport integration branch now

@ghost ghost assigned dzafman and unassigned ghost Mar 11, 2015
@ghost ghost added this to the firefly milestone Mar 11, 2015
@ghost
Copy link

ghost commented Mar 11, 2015

@dzafman a rados suite came back green ( http://tracker.ceph.com/issues/11090#teuthology-run-b0d9b862d7f1a9a98cf4b41ff7cb0eb428740633-march-2015 ). Good to merge in my opinion. Since it's a very large patch series, could you confirm that it's ready to merge in your opinion ?

@ghost ghost mentioned this pull request Mar 11, 2015
Merged
@dzafman
Copy link
Contributor Author

dzafman commented Mar 11, 2015

@dachary @athanatos The only issue I found in my testing using the ceph/ceph-qa-suite#361 is a race with pool deletion causing a ceph-objectstore-tool --op list-pgs to SIGSEGV. I'll file a bug report. This is ready to merge subject to code review. I think Sam should look at it, but another set of eyes would be nice.

A suggested way to code review would be as follows. Examine all changes NOT in src/tools/ceph_objectstore_tool.cc. Then check-out the differences in the tool with git diff hammer wip-cot-firefly -- src/tools/ceph_objectstore_tool.cc. All the changes should be to newer features/changes like map CRC checking, removal changes, META_COLL, NO_SHARD etc...

ghost pushed a commit that referenced this pull request Mar 16, 2015
@ldachary
Merge pull request #3866 from ceph/wip-cot-firefly
e79e5f8 
Backport ceph-objectstore-tool changes to firefly

Reviewed-by: Loic Dachary <ldachary@redhat.com>
@ghost ghost merged commit e79e5f8 into firefly Mar 16, 2015
@dzafman dzafman deleted the wip-cot-firefly branch March 18, 2015 18:37
@ghost ghost changed the title Backport ceph-objectstore-tool changes to firefly tools: Backport ceph-objectstore-tool changes to firefly Jul 14, 2015
smithfarm added a commit to SUSE/ceph that referenced this pull request Sep 7, 2015
@smithfarm
ceph_objectstore_tool.cc: change ghobject_t::NO_SHARD to shard_id_t::…
53a89c2 
…NO_SHARD

This commit extends the changes from dedd8e2 to
src/tools/ceph_objectstore_tool.cc, which was backported to firefly
(after dedd8e2) by ceph#3866

Signed-off-by: Nathan Cutler <ncutler@suse.com>
smithfarm added a commit to SUSE/ceph that referenced this pull request Oct 8, 2015
@smithfarm
ceph_objectstore_tool.cc: change ghobject_t::NO_SHARD to shard_id_t::…
b53bd60 
…NO_SHARD

This commit extends the changes from dedd8e2 to
src/tools/ceph_objectstore_tool.cc, which was backported to firefly
(after dedd8e2) by ceph#3866

Signed-off-by: Nathan Cutler <ncutler@suse.com>
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@dzafman dzafman

Labels
core feature
Projects
None yet
Milestone
firefly
8 participants
@dzafman @loic-bot @athanatos @dalgaaf @wonzhq @liewegas @ldachary @tchaikov