Permalink
Browse files

OpenZFS 7252 - compressed zfs send / receive

OpenZFS 7252 - compressed zfs send / receive
OpenZFS 7628 - create long versions of ZFS send / receive options

Authored by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed by: David Quigley <dpquigl@davequigley.com>
Reviewed by: Thomas Caputi <tcaputi@datto.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Reviewed by: David Quigley <dpquigl@davequigley.com>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Ported-by: bunder2015 <omfgbunder@gmail.com>
Ported-by: Don Brady <don.brady@intel.com>
Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>

Porting Notes:
- Most of 7252 was already picked up during ABD work.  This
  commit represents the gap from the final commit to openzfs.
- Fixed split_large_blocks check in do_dump()
- An alternate version of the write_compressible() function was
  implemented for Linux which does not depend on fio.  The behavior
  of fio differs significantly based on the exact version.
- mkholes was replaced with truncate for Linux.

OpenZFS-issue: https://www.illumos.org/issues/7252
OpenZFS-commit: openzfs/openzfs@5602294
Closes #6067
  • Loading branch information...
dankimmel authored and behlendorf committed Apr 11, 2017
1 parent 7a25f08 commit a7004725d0ad52fa005c29f68ce55273f11d95ac
Showing with 1,668 additions and 104 deletions.
  1. +17 −1 cmd/zfs/zfs_main.c
  2. +2 −2 cmd/zstreamdump/zstreamdump.c
  3. +2 −2 lib/libzfs_core/libzfs_core.c
  4. +10 −10 man/man8/zfs.8
  5. +70 −41 module/zfs/arc.c
  6. +2 −3 module/zfs/dbuf.c
  7. +4 −0 module/zfs/dmu.c
  8. +7 −7 module/zfs/dmu_send.c
  9. +1 −1 module/zfs/lz4.c
  10. +10 −4 tests/runfiles/linux.run
  11. +2 −1 tests/zfs-tests/include/commands.cfg
  12. +68 −0 tests/zfs-tests/include/libtest.shlib
  13. +25 −0 tests/zfs-tests/include/math.shlib
  14. +84 −8 tests/zfs-tests/include/properties.shlib
  15. +16 −1 tests/zfs-tests/tests/functional/rsend/Makefile.am
  16. +3 −3 tests/zfs-tests/tests/functional/rsend/rsend.cfg
  17. +125 −7 tests/zfs-tests/tests/functional/rsend/rsend.kshlib
  18. +2 −1 tests/zfs-tests/tests/functional/rsend/rsend_019_pos.ksh
  19. +2 −1 tests/zfs-tests/tests/functional/rsend/rsend_020_pos.ksh
  20. +2 −1 tests/zfs-tests/tests/functional/rsend/rsend_021_pos.ksh
  21. +2 −1 tests/zfs-tests/tests/functional/rsend/rsend_022_pos.ksh
  22. +2 −1 tests/zfs-tests/tests/functional/rsend/rsend_024_pos.ksh
  23. +77 −0 tests/zfs-tests/tests/functional/rsend/send-cD.ksh
  24. +109 −0 tests/zfs-tests/tests/functional/rsend/send-c_embedded_blocks.ksh
  25. +100 −0 tests/zfs-tests/tests/functional/rsend/send-c_incremental.ksh
  26. +73 −0 tests/zfs-tests/tests/functional/rsend/send-c_lz4_disabled.ksh
  27. +54 −0 tests/zfs-tests/tests/functional/rsend/send-c_mixed_compression.ksh
  28. +67 −0 tests/zfs-tests/tests/functional/rsend/send-c_props.ksh
  29. +55 −0 tests/zfs-tests/tests/functional/rsend/send-c_recv_dedup.ksh
  30. +68 −0 tests/zfs-tests/tests/functional/rsend/send-c_recv_lz4_disabled.ksh
  31. +49 −0 tests/zfs-tests/tests/functional/rsend/send-c_resume.ksh
  32. +91 −0 tests/zfs-tests/tests/functional/rsend/send-c_stream_size_estimate.ksh
  33. +55 −0 tests/zfs-tests/tests/functional/rsend/send-c_verify_contents.ksh
  34. +66 −0 tests/zfs-tests/tests/functional/rsend/send-c_verify_ratio.ksh
  35. +80 −0 tests/zfs-tests/tests/functional/rsend/send-c_volume.ksh
  36. +59 −0 tests/zfs-tests/tests/functional/rsend/send-c_zstreamdump.ksh
  37. +199 −0 tests/zfs-tests/tests/functional/rsend/send-cpL_varied_recsize.ksh
  38. +2 −2 tests/zfs-tests/tests/perf/fio/mkfiles.fio
  39. +2 −2 tests/zfs-tests/tests/perf/fio/random_readwrite.fio
  40. +2 −2 tests/zfs-tests/tests/perf/fio/random_writes.fio
  41. +2 −2 tests/zfs-tests/tests/perf/fio/sequential_writes.fio
View
@@ -32,6 +32,7 @@
#include <assert.h>
#include <ctype.h>
#include <errno.h>
#include <getopt.h>
#include <libgen.h>
#include <libintl.h>
#include <libuutil.h>
@@ -3771,8 +3772,23 @@ zfs_do_send(int argc, char **argv)
nvlist_t *dbgnv = NULL;
boolean_t extraverbose = B_FALSE;
struct option long_options[] = {
{"replicate", no_argument, NULL, 'R'},
{"props", no_argument, NULL, 'p'},
{"parsable", no_argument, NULL, 'P'},
{"dedup", no_argument, NULL, 'D'},
{"verbose", no_argument, NULL, 'v'},
{"dryrun", no_argument, NULL, 'n'},
{"large-block", no_argument, NULL, 'L'},
{"embed", no_argument, NULL, 'e'},
{"resume", required_argument, NULL, 't'},
{"compressed", no_argument, NULL, 'c'},
{0, 0, 0, 0}
};
/* check options */
while ((c = getopt(argc, argv, ":i:I:RDpvnPLet:c")) != -1) {
while ((c = getopt_long(argc, argv, ":i:I:RDpvnPLet:c", long_options,
NULL)) != -1) {
switch (c) {
case 'i':
if (fromname)
@@ -430,8 +430,8 @@ main(int argc, char *argv[])
drro->drr_bonuslen);
}
if (drro->drr_bonuslen > 0) {
(void) ssread(buf, P2ROUNDUP(drro->drr_bonuslen,
8), &zc);
(void) ssread(buf,
P2ROUNDUP(drro->drr_bonuslen, 8), &zc);
if (dump) {
print_block(buf,
P2ROUNDUP(drro->drr_bonuslen, 8));
@@ -493,10 +493,10 @@ lzc_send_resume(const char *snapname, const char *from, int fd,
fnvlist_add_string(args, "fromsnap", from);
if (flags & LZC_SEND_FLAG_LARGE_BLOCK)
fnvlist_add_boolean(args, "largeblockok");
if (flags & LZC_SEND_FLAG_COMPRESS)
fnvlist_add_boolean(args, "compressok");
if (flags & LZC_SEND_FLAG_EMBED_DATA)
fnvlist_add_boolean(args, "embedok");
if (flags & LZC_SEND_FLAG_COMPRESS)
fnvlist_add_boolean(args, "compressok");
if (resumeobj != 0 || resumeoff != 0) {
fnvlist_add_uint64(args, "resume_object", resumeobj);
fnvlist_add_uint64(args, "resume_offset", resumeoff);
View
@@ -2755,7 +2755,7 @@ Generate a stream package that sends all intermediary snapshots from the first s
.sp
.ne 2
.na
\fB\fB-R\fR\fR
\fB\fB-R\fR, \fB--replicate\fR\fR
.ad
.sp .6
.RS 4n
@@ -2767,7 +2767,7 @@ If the \fB-i\fR or \fB-I\fR flags are used in conjunction with the \fB-R\fR flag
.sp
.ne 2
.na
\fB\fB-D\fR\fR
\fB\fB-D\fR, \fB--dedup\fR\fR
.ad
.sp .6
.RS 4n
@@ -2777,7 +2777,7 @@ Generate a deduplicated stream. Blocks which would have been sent multiple times
.sp
.ne 2
.na
\fB\fB-L\fR\fR
\fB\fB-L\fR, \fB--large-block\fR\fR
.ad
.sp .6
.RS 4n
@@ -2792,7 +2792,7 @@ flags and the \fBlarge_blocks\fR feature.
.sp
.ne 2
.na
\fB\fB-e\fR\fR
\fB\fB-e\fR, \fB--embed\fR\fR
.ad
.sp .6
.RS 4n
@@ -2825,7 +2825,7 @@ decompressed before sending so it can be split into smaller block sizes.
.sp
.ne 2
.na
\fB\fB-p\fR\fR
\fB\fB-p\fR, \fB--props\fR\fR
.ad
.sp .6
.RS 4n
@@ -2835,7 +2835,7 @@ Include the dataset's properties in the stream. This flag is implicit when -R i
.sp
.ne 2
.na
\fB\fB-n\fR\fR
\fB\fB-n\fR, \fB--dryrun\fR\fR
.ad
.sp .6
.RS 4n
@@ -2849,7 +2849,7 @@ to standard output and the verbose output goes to standard error).
.sp
.ne 2
.na
\fB\fB-P\fR\fR
\fB\fB-P\fR, \fB--parsable\fR\fR
.ad
.sp .6
.RS 4n
@@ -2859,7 +2859,7 @@ Print machine-parsable verbose information about the stream package generated.
.sp
.ne 2
.na
\fB\fB-v\fR\fR
\fB\fB-v\fR, \fB--verbose\fR\fR
.ad
.sp .6
.RS 4n
@@ -2887,7 +2887,7 @@ name will be "--head--".
.sp
.ne 2
.na
\fB\fB-L\fR\fR
\fB\fB-L\fR, \fB--large-block\fR\fR
.ad
.sp .6
.RS 4n
@@ -2902,7 +2902,7 @@ flags and the \fBlarge_blocks\fR feature.
.sp
.ne 2
.na
\fB\fB-e\fR\fR
\fB\fB-e\fR, \fB--embed\fR\fR
.ad
.sp .6
.RS 4n
Oops, something went wrong.

0 comments on commit a700472

Please sign in to comment.