rbd: introduce v2 format for rbd export/import. #10487

yangdongsheng · 2016-07-29T04:26:29Z

Fixes: http://tracker.ceph.com/issues/13186

yangdongsheng · 2016-07-29T04:29:57Z

@dillaman
This PR added this feature http://tracker.ceph.com/issues/13186 , please help to take a look. thanx.

yangdongsheng · 2016-07-29T08:47:40Z

@dillaman I found there is still some bug in this PR, so it's not ready for code-review.

yangdongsheng · 2016-08-01T07:27:14Z

@dillaman it passed my local manual testing now, please help to take a look. thanx

dillaman · 2016-08-05T18:34:34Z

src/tools/rbd/action/MergeDiff.cc

@@ -196,7 +196,7 @@ static int do_merge_diff(const char *first, const char *second,
    }
  }

-  //We just handle the case like 'banner, [ftag], [ttag], stag, [wztag]*,etag',
+  //We just handle the case like 'banner, [ftag], [ttag], [stag], [wztag]*, [etag]',


I think this was an attempt to say that the "s" and "e" tags are not optional and the ones in the brackets are optional.

right. 👍

dillaman · 2016-08-05T18:53:51Z

@yangdongsheng I'd recommend combining Export/ExportDiff and Import/ImportDiff into the same file now that they are going to be interdependent.

dillaman · 2016-08-05T18:58:21Z

@yangdongsheng It might be cleaner to just 100% re-use the existing export/import diff format for the export/import with snaps. Just create a new "v2" header to distinguish the new behavior that it might contain >1 snapshot segment. Then on import(diff), if it's a v2, it should keep looping when it gets to the end snapshot marker, then the next field should be a from snapshot marker, data, etc, end snapshot, etc.

dillaman · 2016-08-05T19:00:48Z

src/tools/rbd/action/Import.cc

+	goto done;
+      }
+      // seek back to the beginning.
+      lseek(fd, 0, SEEK_SET);


Couldn't you just set "import_snaps" to false here and eliminate the need for new_format?

Sorry, I am not sure what's your suggestion here, but I can explain the logic here more clear.

(1).import_snaps == true means --with-snaps is specified.
(2).(from_stdin == false )means importing something from file.

Then there are 3 possibility we will into this if statement.
(1). (import_snaps == true) and (from_stdin == true), in this case, when we found it's old format, because it's from stdin, we can't seek back, so return -EINVAL. That means, you have to make sure the you are accepting a new format importing stdin when you specify --with-snaps manually.

(2). (import_snaps == true) and (from_stdin == false), in this case, when we found it's a old format. we will lseek to the beginning, because it's from a fd. and the all data in file should be imported into image.

(3). (import_snaps == false) and (from_stdin == false), in this case, --with-snaps are not specified, and we are importing data from a file. then we try to check the banner at first, if we found it's old format. lseek to 0 to import it in old manner.

In this way, we can keep the backforwad compatability for the old format of export/exportdiff

yangdongsheng · 2016-09-09T05:24:51Z

Hi @dillaman sorry for the late update. Please help to take a look again. thanx

yangdongsheng · 2016-09-18T06:10:41Z

@dillaman do you have a time to take a look at this PR?

dillaman · 2016-09-15T15:40:48Z

qa/workunits/rbd/import_export.sh

+rbd info testimg_import
+rbd info testimg_import@snap
+rm ${TMPDIR}/img
+rbd snap rm testimg_import@snap


Compare the contents between snaps and image HEAD

dillaman · 2016-09-15T15:41:53Z

src/librbd/ExclusiveLock.cc

@@ -711,8 +711,11 @@ void ExclusiveLock<I>::handle_shutdown(int r) {

  m_image_ctx.aio_work_queue->clear_require_lock_on_read();
  m_image_ctx.aio_work_queue->unblock_writes();
-  m_image_ctx.image_watcher->flush(util::create_context_callback<
-    ExclusiveLock<I>, &ExclusiveLock<I>::complete_shutdown>(this));
+  if (m_image_ctx.image_watcher == nullptr)


Why do you need to make any changes here?

I got a segmentation fault without this change in "rbd export --with-snaps"

dillaman · 2016-09-15T15:44:04Z

src/tools/rbd/action/Export.cc

@@ -154,6 +213,8 @@ void get_arguments(po::options_description *positional,
  at::add_path_options(positional, options,
                       "export file (or '-' for stdout)");
  at::add_no_progress_option(options);
+  options->add_options()
+    ("with-snaps", po::bool_switch(), "export snapshots at the same time");


Minor: for some reason, "with-snaps" doesn't feel clear to me. Perhaps "--export-snapshots" or "--include-snapshots"

dillaman · 2016-09-15T15:46:30Z

src/tools/rbd/Utils.cc

@@ -857,5 +857,265 @@ std::string timestr(time_t t) {
  return buf;
 }

+int export_diff_fd(librbd::Image& image, const char *fromsnapname,


Minor: I would recommend combining Export.cc and ExportDiff.cc into the same file and then keeping the export_diff and export_diff_fd within the new combined Export.cc file. Same logic for Import.cc and ImportDiff.cc

dillaman · 2016-09-15T15:51:13Z

src/tools/rbd/Utils.h

 #include <string>
 #include <boost/program_options.hpp>

 namespace rbd {
 namespace utils {

+static const std::string RBD_IMAGE_BANNER ("rbd image v1\n");
+static const std::string RBD_IMAGE_SNAPS_BANNER ("rbd image snaps v1\n");


Minor: I can foresee wanted to eventually export other image attributes in addition to snapshots (e.g. features, metadata, etc). Therefore, perhaps it should be a v2 banner and then the export format should have a set of flag at the start that indicate what optional attributes have been included in the export (e.g. snapshots, features, etc).

dillaman · 2016-09-15T15:52:44Z

src/tools/rbd/action/Export.cc

+  if (with_snaps) {
+    // header
+    bufferlist bl;
+    bl.append(utils::RBD_IMAGE_BANNER);


Minor: new RBD_IMAGE_BANNER_V2

dillaman · 2016-09-15T15:58:36Z

src/tools/rbd/action/Export.cc

  }

-  if (r < 0) {
+  if (with_snaps) {


I think you would first want to export the image starting at the first available snapshot via export_diff (i.e. only record the data that actually exists in the image), then iterating through all the available snapshots to generate diffs for each successive snapshot until you get to the specified head location. This would create a nice sparse export.

sounds good.

But then we can't import a image without snaps in this case.

rbd export --with-snaps test test.export rbd import test.import test.export

I mean, sometimes, we can get a exported image with snapshots, but we only want to import a full image from it.

So I name it as "with snaps" rather than "snaps mode". But yes, --include-snapshots sounds better. :)

dillaman · 2016-09-15T16:02:32Z

src/tools/rbd/action/Import.cc

-    if (from_stdin && (image_pos + (size_t)blklen) > size) {
-      size *= 2;
-      r = image.resize(size);
+    lseek(fd, size, SEEK_CUR);


Seems like exported image has the full (non-sparse) image export which isn't used.

Because this will be used when we import a image without --with-snaps. Sometimes, maybe we only want to import a full image even the snaps were exported already.

dillaman · 2016-09-18T22:29:01Z

@yangdongsheng Sorry -- the comments were sitting in "pending" state for days since I forgot to publish them.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

…xport.sh Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

… importing test cases. Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

…port Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

…platform Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

…ff and import. Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

tchaikov · 2017-03-30T16:32:16Z

@yangdongsheng and @dillaman

the workunit change breaks the jewel-x-singleton test, see http://pulpito.ceph.com/kchai-2017-03-30_14:32:23-rados-wip-13999-kefu---basic-mira/966061/

2017-03-30T14:41:21.411 INFO:tasks.workunit.client.0.mira071.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_13580/img_v2
2017-03-30T14:41:21.434 INFO:tasks.workunit.client.0.mira071.stderr:rbd: unrecognised option '--export-format'

dillaman · 2017-03-30T16:33:26Z

@tchaikov Please open a tracker ticket -- this PR is already merged

tchaikov · 2017-03-30T16:44:51Z

@dillaman sure, will do tomorrow. just wanted to give you guys a quick heads-up in case you run into this.

dillaman · 2017-03-30T16:55:00Z

@tchaikov It's a teuthology issue where the test case specifies that the "jewel" branch of the workunit should be used, but instead it downloads the "head" version (d463f52) of the workunit. Can't help but notice that particular commit changes the behavior of tasks/workunit.py so potentially this issue was just introduced?

xiexingguo added feature rbd labels Jul 29, 2016

yangdongsheng changed the title ~~rbd: add --with-snaps to allow user to import or export snapshots at the same time with image~~ [RFC] rbd: add --with-snaps to allow user to import or export snapshots at the same time with image Jul 29, 2016

yangdongsheng force-pushed the rbd_export branch from 967dfa5 to f46cb1e Compare August 1, 2016 07:22

yangdongsheng changed the title ~~[RFC] rbd: add --with-snaps to allow user to import or export snapshots at the same time with image~~ rbd: add --with-snaps to allow user to import or export snapshots at the same time with image Aug 1, 2016

yangdongsheng force-pushed the rbd_export branch 3 times, most recently from e16ab22 to 7125613 Compare August 3, 2016 01:26

dillaman reviewed Aug 5, 2016
View reviewed changes

yangdongsheng force-pushed the rbd_export branch from 7125613 to 9dedd83 Compare August 11, 2016 08:44

yangdongsheng force-pushed the rbd_export branch 2 times, most recently from bbc99e7 to 9a37724 Compare September 9, 2016 05:09

dillaman reviewed Sep 18, 2016

View reviewed changes

yangdongsheng added 23 commits February 19, 2017 20:41

rbd: finish the pc in importing even if from_stdin.

4658cf1

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: use macro for rbd diff tags

12f50c4

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: squash ExportDiff.cc into Export.cc

262c8d3

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: squash ImportDiff.cc into Import.cc.

f6192de

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: move do_export_diff out from namespace of export_diff.

21be324

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: move do_import_diff out from namespace of import_diff

de671d7

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: refactor do_export_diff to accept fd

bc80b14

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: refactor do_import_diff to accept fd.

58c8e8c

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: introduce --import-format to import command

5e03460

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: introduce --export-format to export command

26b858e

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

qa/workunit/rbd: add test case for --import/export-format in import_e…

55edd81

…xport.sh Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import/export image_order.

8679521

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import/export features of image

912ff4d

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import/export image stripe_unit and stripe_count.

d7479a0

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

qa/workunit/rbd: import_export.sh: add image priorities exporting and…

4ee764f

… importing test cases. Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: use the unified --export-format option for rbd export and rbd im…

f16899a

…port Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: refactor export/import to v1 and v2 methods.

08d0c2a

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import_diff/export_diff: encode length for each tag.

435104a

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import: decode file data in le64 for compatability on different …

c2356be

…platform Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: combine the namespaces of export_diff and export_full, import_di…

8d2a613

…ff and import. Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: add docs for new format of rbd export.

aa913ce

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import: use read() instead of lseek() when from_stdin is true

fedf772

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import/export/mergediff: use STD[IN|OUT]_FILENO instead of 0 or 1

5b14582

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

yangdongsheng force-pushed the rbd_export branch from f50ff33 to 5b14582 Compare February 19, 2017 12:42

dillaman approved these changes Feb 19, 2017

View reviewed changes

dillaman merged commit b2b1899 into ceph:master Feb 19, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rbd: introduce v2 format for rbd export/import. #10487

rbd: introduce v2 format for rbd export/import. #10487

yangdongsheng commented Jul 29, 2016

yangdongsheng commented Jul 29, 2016

yangdongsheng commented Jul 29, 2016

yangdongsheng commented Aug 1, 2016

dillaman Aug 5, 2016

yangdongsheng Aug 8, 2016

dillaman commented Aug 5, 2016

dillaman commented Aug 5, 2016

dillaman Aug 5, 2016

yangdongsheng Aug 8, 2016

yangdongsheng commented Sep 9, 2016

yangdongsheng commented Sep 18, 2016

dillaman Sep 15, 2016

yangdongsheng Oct 10, 2016

dillaman Sep 15, 2016

yangdongsheng Oct 10, 2016

dillaman Sep 15, 2016

yangdongsheng Oct 10, 2016

dillaman Sep 15, 2016

yangdongsheng Oct 10, 2016

dillaman Sep 15, 2016

yangdongsheng Oct 10, 2016

dillaman Sep 15, 2016

yangdongsheng Oct 10, 2016

dillaman Sep 15, 2016

yangdongsheng Oct 10, 2016

yangdongsheng Oct 10, 2016

yangdongsheng Oct 10, 2016 •

edited

dillaman Sep 15, 2016

yangdongsheng Oct 10, 2016

dillaman commented Sep 18, 2016

tchaikov commented Mar 30, 2017 •

edited

dillaman commented Mar 30, 2017

tchaikov commented Mar 30, 2017

dillaman commented Mar 30, 2017 •

edited

rbd: introduce v2 format for rbd export/import. #10487

rbd: introduce v2 format for rbd export/import. #10487

Conversation

yangdongsheng commented Jul 29, 2016

yangdongsheng commented Jul 29, 2016

yangdongsheng commented Jul 29, 2016

yangdongsheng commented Aug 1, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dillaman commented Aug 5, 2016

dillaman commented Aug 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yangdongsheng commented Sep 9, 2016

yangdongsheng commented Sep 18, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yangdongsheng Oct 10, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dillaman commented Sep 18, 2016

tchaikov commented Mar 30, 2017 • edited

dillaman commented Mar 30, 2017

tchaikov commented Mar 30, 2017

dillaman commented Mar 30, 2017 • edited

yangdongsheng Oct 10, 2016 •

edited

tchaikov commented Mar 30, 2017 •

edited

dillaman commented Mar 30, 2017 •

edited