Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tools: Backport ceph-objectstore-tool changes to firefly #3866

Merged
97 commits merged into from Mar 16, 2015
Merged

tools: Backport ceph-objectstore-tool changes to firefly #3866

97 commits merged into from Mar 16, 2015

Commits on Mar 3, 2015

  1. ceph_filestore_dump.cc: cleanup includes 

    Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 8620609)
    @dalgaaf @dzafman
    dalgaaf authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    4a742fe View commit details
    Browse the repository at this point in the history

  2. ceph_filestore_dump: Remove unused bufferlist databl 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 398b418)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    0b757af View commit details
    Browse the repository at this point in the history

  3. ceph_filestore_dump: Export omap in batches for large omap case 

    New function get_omap_batch()
Create a TYPE_OMAP section for each batch

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 501dd3c)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    3f4cabd View commit details
    Browse the repository at this point in the history

  4. common,ceph_filestore_dump: Add ability for utilities to suppress lib… 

    …rary dout output

Suppress dout output with CODE_ENVIRONMENT_UTILITY_NODOUT
ceph_filestore_dump turns on dout output if --debug specified
When used it can still be enable with --log-to-stderr --err-to-stderr

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit f7f9b25)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    f224429 View commit details
    Browse the repository at this point in the history

  5. ceph_filestore_dump: Use cerr now that we aren't closing stderr 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 087c0f9)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    1164b2e View commit details
    Browse the repository at this point in the history

  6. ceph_filestore_dump: Improve debug output by showing actual offset 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 44b261d)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    3381aeb View commit details
    Browse the repository at this point in the history

  7. os: Add optional flags to generic ObjectStore creation (SKIP_JOURNAL_… 

    …REPLAY

and SKIP_MOUNT_OMAP)

Only FileStore cares about these flags, so passed on during create()

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3d9fde9)

Conflicts:
	src/os/FileStore.cc
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    c636998 View commit details
    Browse the repository at this point in the history

  8. ceph_filestore_dump: Add --skip-journal-replay and --skip-mount-omap 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 2e9dcb2)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    dadecb1 View commit details
    Browse the repository at this point in the history

  9. common: Add missing ghobject_t::decode() for json 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit c05f895)

Changes:
	Adjusted for older shard_t
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    d4aedeb View commit details
    Browse the repository at this point in the history

  10. ceph_filstore_dump: Save if stdout is a tty and add routine to clean … 

    …binary strings

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3a574cc)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    9816f87 View commit details
    Browse the repository at this point in the history

  11. ceph_filestore_dump: Fail import/export with a tty for stdin/stdout 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 7520e50)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    62dc823 View commit details
    Browse the repository at this point in the history

  12. ceph_filestore_dump: Add utility function get_fd_data() 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit d4a9daf)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    a469464 View commit details
    Browse the repository at this point in the history

  13. ceph_filestore_dump: Add remove object command and usage for new comm… 

    …ands

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 605caec)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    7cda8b7 View commit details
    Browse the repository at this point in the history

  14. ceph_filestore_dump: Add --type list to output objects in a pg in json 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 844dabb)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    aff2c99 View commit details
    Browse the repository at this point in the history

  15. ceph_filestore_dump: Add list-attrs object command 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 00c6b06)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    303e4ce View commit details
    Browse the repository at this point in the history

  16. ceph_filestore_dump: Add list-omap object command 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 48890c7)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    fcabb81 View commit details
    Browse the repository at this point in the history

  17. ceph_filestore_dump: Add get-bytes object command 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 869dd92)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    2f1926d View commit details
    Browse the repository at this point in the history

  18. ceph_filestore_dump: Add set-bytes object command 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3c24d1f)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    55c21b8 View commit details
    Browse the repository at this point in the history

  19. ceph_filestore_dump: Add get-attr object command 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 55d43c0)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    6724da8 View commit details
    Browse the repository at this point in the history

  20. ceph_filestore_dump: Add set-attr object command 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 95554e0)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    7bd2dd3 View commit details
    Browse the repository at this point in the history

  21. ceph_filestore_dump: Add rm-attr object command 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 465d777)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    f391feb View commit details
    Browse the repository at this point in the history

  22. ceph_filestore_dump: Add get-omap object command 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit b50c43c)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    7547f3d View commit details
    Browse the repository at this point in the history

  23. ceph_filestore_dump: Add set-omap object command 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 50cd57e)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    20165d1 View commit details
    Browse the repository at this point in the history

  24. ceph_filestore_dump: Add rm-omap object command 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 0fc6bd2)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    271e3ca View commit details
    Browse the repository at this point in the history

  25. ceph_filestore_dump: Add get-omaphdr object command 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 30c0f31)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    c7d261d View commit details
    Browse the repository at this point in the history

  26. ceph_filestore_dump: Add set-omaphdr object command 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit b4d95cc)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    9ee2c27 View commit details
    Browse the repository at this point in the history

  27. Renames and removal towards a unified ceph_objectstore_tool 

    Rename ceph_filestore_dump.cc and ceph_filestore_dump.py
Remove ceph_filestore_tool.cc

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 7786419)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    a42273e View commit details
    Browse the repository at this point in the history

  28. Complete replacement of ceph_filestore_tool and ceph_filestore_dump 

    with unified ceph_objectstore_tool

Move list-lost-objects and fix-lost-objects features from
  ceph_filestore_tool to ceph_objectstore_tool as list-lost, fix-lost
Change --type to --op for info, log, export...operations
Add --type for the ObjectStore type (defaults to filestore)
Change --filestore-path to --data-path
Update installation, Makefile.am, and .gitignore
Fix and rename test case to match
  Add some additional invalid option checks

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 83fbc91)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    edc9f71 View commit details
    Browse the repository at this point in the history

  29. ceph_objectstore_tool, test: Add list-pgs operations and unit test case 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit f01e334)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    5e3f89e View commit details
    Browse the repository at this point in the history

  30. ceph_objectstore_tool: Minor improvements 

    Make all non-error non-debug output to stdout
Fix a message

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit a2bd2aa)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    6352907 View commit details
    Browse the repository at this point in the history

  31. ceph_objectstore_tool: Add past_intervals to export/import code 

    Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 9325ec1)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    6b50b38 View commit details
    Browse the repository at this point in the history

  32. ceph_objectstore_tool: Add operation "rm-past-intervals" for testing … 

    …purposes

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3694068)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    ecd25cf View commit details
    Browse the repository at this point in the history

  33. test: ceph_objectstore_tool unit test improvements 

    Add namespaces to testing
Increase filesize so export will have multiple chunks
Put json for each object into the db dict

Signed-off-by: David Zafman <david.zafman@inktank.com>
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    df33461 View commit details
    Browse the repository at this point in the history

  34. ceph_objectstore_tool, test: Implement import-rados feature and unit … 

    …test code

Fixes: #8276

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 23ec93a)
    @dzafman
    David Zafman authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    4f83005 View commit details
    Browse the repository at this point in the history

  35. ceph_objectstore_tool: Bug fixes and test improvements 

    ceph_objectgstore_tool:
Fix bugs in the way collection_list_partial() was being called
which caused objects to be seen over and over again.

Unit test:
Fix get_objs() to walk pg tree for pg with sub-directories
Create more objects to test object listing code
Limit number of larger objects
Limit number of objects which get attributes and omaps

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a03f719)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    b863970 View commit details
    Browse the repository at this point in the history

  36. Test: fixing a compile warning in ceph_objectstore_tool.cc 

    For the compiler's sake:
tools/ceph_objectstore_tool.cc:2547:15: warning: ‘r’ may be used
uninitialized in this function [-Wmaybe-uninitialized]

Signed-off-by: Zhiqiang Wang <wonzhq@hotmail.com>
(cherry picked from commit c3e1466)
    @wonzhq @dzafman
    wonzhq authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    ac396f5 View commit details
    Browse the repository at this point in the history

  37. ceph_objectstore_tool.cc: prefer ++operator for non-primitive iterators 

    Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit a5468ab)
    @dalgaaf @dzafman
    dalgaaf authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    ca9df28 View commit details
    Browse the repository at this point in the history

  38. ceph_objectstore_tool: When exporting to stdout, don't cout messages 

    Fixes: #10128
Caused by a2bd2aa

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0d5262a)
(cherry picked from commit 6cb9a24)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    3075919 View commit details
    Browse the repository at this point in the history

  39. vstart.sh: warn less 

    Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a69b845)
    @liewegas @dzafman
    liewegas authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    4b51645 View commit details
    Browse the repository at this point in the history

  40. ceph_objectstore_tool: Rename generated binary to ceph-objectstore-tool 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 4f72ba5)

Conflicts:
	debian/ceph-test.install
	src/.gitignore
	src/test/ceph-helpers.sh (doesn't exist in firefly)
	src/test/ceph_objectstore_tool.py
	src/tools/ceph_objectstore_tool.cc
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    7470c7c View commit details
    Browse the repository at this point in the history

  41. ceph_objectstore_tool: MemStore needs a CephContext 

    Pass g_ceph_context to ObjectStore::create() needed by MemStore

Fixes: #9661

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0b155d0)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    5f788ea View commit details
    Browse the repository at this point in the history

  42. ceph_objectstore_tool: Accept CEPH_ARGS environment arguments 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 10fe7cf)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    e0dab88 View commit details
    Browse the repository at this point in the history

  43. ceph_objectstore_tool: Fixes to make import work again 

    The is_pg() call is now true even for pgs pending removal, fix broken
    finish_remove_pgs() by removing is_pg() check.
Need to add create_collection() to the initial transaction on import

Fixes: #10090

Signed-off-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 5ce0919)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    5b4c2ee View commit details
    Browse the repository at this point in the history

  44. tests: ceph_objectstore_tool.py use env python 

    Using #/usr/bin/env python instead of a hard coded path is more flexible
and can also be used to run from virtualenv.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 74506d2)
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    8548d7c View commit details
    Browse the repository at this point in the history

  45. tests: ceph_objectstore_tool.py can use a WARN cluster 

    The tests do not need HEALTH_OK exclusively, a HEALTH_WARN cluster can
also run them successfully.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 783378c)
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    28ed34a View commit details
    Browse the repository at this point in the history

  46. tests: ceph_objectstore_tool.py run mon and osd on specific port 

    By default vstart.sh runs MDS but they are not needed for the tests,
only run mon and osd instead. Instead of using the default vstart.sh
port which may conflict with a already running vstart.sh, set the
CEPH_PORT=7400 which is not used by any other test run with make check.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit f04d4e7)
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    427fe91 View commit details
    Browse the repository at this point in the history

  47. tests: ceph_objectstore_tool.py run faster by default 

    By default use only a small number of objects to speed up the tests. If
the argument "big" is given, use a large number of objects as it may
help find some problems.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 235257c)
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    454ec85 View commit details
    Browse the repository at this point in the history

  48. tests: ceph_objectstore_tool.py use a dedicated directory 

    Set CEPH_DIR to a directory that is specific to ceph_objectstore_tool so
that it can run in parallel with other vstart.sh clusters.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit e303d1a)

Conflicts:
	src/test/ceph_objectstore_tool.py
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    5185558 View commit details
    Browse the repository at this point in the history

  49. tests: ceph_objectstore_tool.py replace stop.sh with init-ceph 

    The stop.sh will stop all ceph-* processes. Use the init-ceph script
instead to selectively kill the daemons run by the vstart.sh cluster
used for ceph_objectstore_tool.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit e8f34bd)
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    04e480b View commit details
    Browse the repository at this point in the history

  50. tests: ceph_objectstore_tool.py main returns 

    Instead of calling sys.exit() the main function returns the desired exit
code.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 0f3d7b1)
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    b78d802 View commit details
    Browse the repository at this point in the history

  51. tests: ceph_objectstore_tool.py encapsulate init-ceph stop 

    Call init-ceph in kill_daemons and add a call to kill_daemon when main
returns on error so that it never leaves daemons hanging.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit fc435ff)
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    8dc2635 View commit details
    Browse the repository at this point in the history

  52. tests: ceph_objectstore_tool.py skip if /dev/tty fails 

    Some environments do not have a /dev/tty. When opening /dev/tty fails,
skip the test instead of returning an error.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 4c94f17)
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    951c951 View commit details
    Browse the repository at this point in the history

  53. tests: ceph_objectstore_tool.py fix off by 1 ATTR_OBJS handling 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit eaf1d1e)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    99ffd13 View commit details
    Browse the repository at this point in the history

  54. tests: ceph_objectstore_tool.py check for malformed JSON for erasure … 

    …code objs

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 40717aa)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    bc921c7 View commit details
    Browse the repository at this point in the history

  55. tests: ceph_objectstore_tool.py fix list-attr for erasure code 

    Adding testing of xattr for erasure coded shards
Fix error message when finding an unexpected xattr key

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit cbecab4)

Conflicts:
	src/test/ceph_objectstore_tool.py
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    70329be View commit details
    Browse the repository at this point in the history

  56. ceph_objectstore_tool: Make --file option consistent by treating "-" … 

    …as stdout/stdin

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 8c87f32)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    2a58bde View commit details
    Browse the repository at this point in the history

  57. tests: ceph_objectstore_tool.py test all variants of export/import 

    Handle change of error message text

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 9e53c35)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    3f862da View commit details
    Browse the repository at this point in the history

  58. ceph_objectstore_tool: Clear ...INCOMPAT_SHARDS from feature if expor… 

    …ting replicated pg

Don't require importing OSD to have shards feature for replicated pg

http://tracker.ceph.com/issues/10077 Fixes: #10077

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 86baf2d)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    f997b12 View commit details
    Browse the repository at this point in the history

  59. ceph_objectstore_tool: Add utility routine get_osdmap() 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b3021b0)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    4e30d8c View commit details
    Browse the repository at this point in the history

  60. ceph_objectstore_tool: Add feature called set-allow-sharded-objects 

    Uses --op set-allow-sharded-objects option
This operation will be rejected if on the target OSD's osdmap there is
    at least one OSD which does not support ERASURE CODES.
Prompt the user that they could import if sharded state allowed
Prompt the user to use new feature if sharded state found inconsistent

Fixes: #10077

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f3dab44)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    5bbe41c View commit details
    Browse the repository at this point in the history

  61. objectstore_tool: update usage strings 

    Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit a90233c)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    eab3226 View commit details
    Browse the repository at this point in the history

  62. objectstore_tool: refactor list-lost and fix-lost 

    Abstract out the PG exploration loops and encapsulate the list-lost and
fix-lost semantic in a callable object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit d9e747b)
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    eb48aba View commit details
    Browse the repository at this point in the history

  63. objectstore_tool: lookup objects by name 

    If the object is not a parsable JSON string, assume an object name and
look it up in all the PGs. If multiple objects have the same name, only
apply the command to one of them. It is primarily useful in a test
environment where the names of the tests objects are known and only a
small number of objects exists. It replaces the following:

    path='--data-path dev/osd0 --journal-path dev/osd0.journal'
    for pgid in $(./ceph_objectstore_tool $path --op list-pgs) ; do
      object=$(./ceph_objectstore_tool $path --pgid $pgid --op list |
               grep '"oid":"NAME"')
      test -n "$object" && break
    done
    ./ceph_objectstore_tool $path --pgid $pgid "$object" remove

with:

    ./ceph_objectstore_tool $path NAME remove

http://tracker.ceph.com/issues/10192 Fixes: #10192

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 7c1165f)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    2f874fd View commit details
    Browse the repository at this point in the history

  64. objectstore_tool: filter --op list and explore all PGs 

    The positional object name is used to filter the output of --op list and
only show the objects with a matching name. If both the object name and
the pgid are omitted, all objects from all PGs are displayed.

The output format is changed from

    {"oid":"GROUP","key":"","snapid":-2,
     "hash":2659194943,"max":0,"pool":0,"namespace":""}

to

    [["0.7_head",{"oid":"GROUP","key":"","snapid":-2,
                  "hash":2659194943,"max":0,"pool":0,
                  "namespace":""}]]

where the first member is the pgid where the object is found.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit c69aace)
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    fce87c9 View commit details
    Browse the repository at this point in the history

  65. objectstore_tool: parse new object description format 

    The object format changed from

    {json object}

to

    [pgid,{json object}]

The parser is updated accordingly. If the --pgid is present, check that
it equals the pgid from the object description.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit df9d5c5)
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    f30e053 View commit details
    Browse the repository at this point in the history

  66. objectstore_tool: test --op list variants 

    Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit ddba267)
    @ldachary @dzafman
    ldachary authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    d802ab6 View commit details
    Browse the repository at this point in the history

  67. ceph_objectstore_tool: In error case umount objectstore 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit bc6ea9c)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    22c6bf4 View commit details
    Browse the repository at this point in the history

  68. ceph_objectstore_tool: For terminal output of list one object per line 

    Instead of a parsable array make it easier to cut and paste listed objects

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5a66db9)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    06579b9 View commit details
    Browse the repository at this point in the history

  69. ceph_objectstore_tool: BUG: --op list wasn't including snapshots 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b617ee2)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    4a0a42f View commit details
    Browse the repository at this point in the history

  70. ceph_objectstore_tool: Strip _* (always _head) from pgid in list entr… 

    …y output

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5eacd3c)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    a5544c1 View commit details
    Browse the repository at this point in the history

  71. ceph_objectstore_tool: Add --format and --pretty-format support 

    --pretty-format defaults true
Add --format so xml output can be requested
--op list defaults to single line of json per object
To override this more human readable output use --pretty-format=false
Add testing of --op list special handling

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit cca85a5)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    f2d2110 View commit details
    Browse the repository at this point in the history

  72. ceph_objectstore_tool: Remove extraneous endl on error throw messages 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 32c832f)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    1703867 View commit details
    Browse the repository at this point in the history

  73. ceph_objectstore_tool: Fix errors messages in newer code 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit eae7c02)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    4f95409 View commit details
    Browse the repository at this point in the history

  74. ceph_objectstore_tool: Improve object spec parsing error messages 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit de6384f)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    60e9a81 View commit details
    Browse the repository at this point in the history

  75. ceph_objectstore_tool: Accept json object with --pgid instead of array 

    It isn't anticipated that anyone would use this but keeps backward compatible

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 59b423e)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    53aa04f View commit details
    Browse the repository at this point in the history

  76. ceph_objectstore_tool: Prevent tool from transitioning struct_v on rm… 

    …-past-intervals

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f056bdf)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    2a22bfe View commit details
    Browse the repository at this point in the history

  77. ceph_objectstore_tool.cc: prefer ++operator for non-primitive iterators 

    Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 252fc03)
    @dalgaaf @dzafman
    dalgaaf authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    2f97c51 View commit details
    Browse the repository at this point in the history

  78. ceph_objectstore_tool.cc: reduce scope of variable 

    Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 78542f9)
    @dalgaaf @dzafman
    dalgaaf authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    5f82f82 View commit details
    Browse the repository at this point in the history

  79. ceph_objectstore_tool: Describe super_ver values 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0aeba0f)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    ee183b8 View commit details
    Browse the repository at this point in the history

  80. ceph-objectstore-tool: Remove --pretty-format and use new --format op… 

    …tions

Call new_formatter() with --format specified argument

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 22b7174)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    8b7f4cb View commit details
    Browse the repository at this point in the history

  81. ceph_objectstore_tool: read_fd() doesn't handle ^D from tty stdin, do… 

    …n't allow

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5cb6925)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    d962c79 View commit details
    Browse the repository at this point in the history

  82. ceph_objectstore_tool: Fix import-rados skipping of snapshots 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit fe93602)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    bbed3a7 View commit details
    Browse the repository at this point in the history

  83. ceph_objectstore_tool: Get g_ceph_context available to import-rados 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit ddc4613)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    2d69076 View commit details
    Browse the repository at this point in the history

  84. ceph_objectstore_tool: import-rados shouldn't import internal namespa… 

    …ce objects

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f727d2e)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    c3fcbe6 View commit details
    Browse the repository at this point in the history

  85. ceph_objectstore_tool: Allow the metadata_section to be anywhere in t… 

    …he export

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 62dd912)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    360f68e View commit details
    Browse the repository at this point in the history

  86. ceph_objectstore_tool: Check cluster_fsid before allowing an import 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 196c811)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    f65db43 View commit details
    Browse the repository at this point in the history

  87. ceph_objectstore_tool: Check that pool exists before allowing import 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f24f646)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    22b7c2f View commit details
    Browse the repository at this point in the history

  88. ceph_objectstore_tool: Add special exit code for test infrastructure 

    Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b1f12f0)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    aa472fa View commit details
    Browse the repository at this point in the history

  89. ceph_objectstore_tool: Check for splits and fail import if there were… 

    … splits

Add osdmap into metadata_section
On export put metadata_section before file data

Fixes: #9780

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 19fdeea)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    5ec38e5 View commit details
    Browse the repository at this point in the history

  90. ceph_objectstore_tool: Verify that object pool and pgid pool match 

    Also, earlier check for invalid --pgid with import op

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit da3be80)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    c3de607 View commit details
    Browse the repository at this point in the history

  91. ceph_objectstore_tool: On import following a split skip objects no lo… 

    …nger in pg

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 68b27e2)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    aedd324 View commit details
    Browse the repository at this point in the history

  92. ceph_objectstore_tool: Handle import of pg which OSD has split 

    Fail import if import data doesn't include OSDMap and can't find it locally
See if local map can be read for import's epoch
Jump to current epoch like a split would if local map not present

Fixes: #9781

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit afda6e4)

Conflicts:
	src/tools/ceph_objectstore_tool.cc
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    f70590b View commit details
    Browse the repository at this point in the history

  93. ceph_objectstore_tool: For import get object_info_t available for eac… 

    …h object

Add object_info_t to object_begin so we have at object create time
This will be useful for importing from multiple erasure coded exports

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 05d916e)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    d5d34dd View commit details
    Browse the repository at this point in the history

  94. ceph_objectstore_tool: fix check_output on python2.6 

    * backported the subprocess.check_output from python2.7

Fixes: #10756

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 15350a0)

Conflicts:
	src/test/ceph_objectstore_tool.py
    @tchaikov @dzafman
    tchaikov authored and dzafman committed Mar 3, 2015
    Copy the full SHA
    7ce0cb8 View commit details
    Browse the repository at this point in the history

  95. osd: build_push_op() handle short reads so recipient doesn't crash 

    Fixes: #8121

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b9a0484)
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    9235d78 View commit details
    Browse the repository at this point in the history

  96. osd: Pass oldest_map when generating past intervals 

    From load_pgs() the superblock hasn't been published yet
so we need to retrieve the value of oldest_map depending on the context.

Fixes: #10427

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 7fb721c)

Conflicts:
	src/osd/OSD.cc
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    c09b6d9 View commit details
    Browse the repository at this point in the history

  97. osd: Get pgid ancestor from last_map when building past intervals 

    Fixed OSD::build_past_intervals_parallel() and PG::generate_past_intervals()

Fixes: #10430

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0c5b66d)

Conflicts:
	src/osd/OSD.cc
    @dzafman
    dzafman committed Mar 3, 2015
    Copy the full SHA
    91b2aca View commit details
    Browse the repository at this point in the history