Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataset is being sent as a compressed stream by zfs send regardless of the compression option used #77

Closed
Scrin opened this issue May 30, 2021 · 4 comments
Labels
Milestone

Comments

@Scrin
Copy link

Scrin commented May 30, 2021

Not entirely sure whether this is a "less-ideal feature" or an actual bug, but I noticed that send_pipe() (in ZfsDataset.py) appends the --compressed option to zfs send (if it's supported), which causes two notable things:

  • As per the manual "Streams sent with -c will not have their data recompressed on the receiver side using -o -compress=value. The data will stay compressed as it was from the sender." which prevents using different compression method on the receiver side, which is relatively common case; use fast lz4 in production and gzip for the backups for trading performance for saved space.
  • Since the stream is now already compressed by zfs send, it makes the "wrapping" compressor (defined by the zfs_autobackup's --compress flag) very inefficient since it now has to compress already compressed data rather than the raw, uncompressed source data.

Perhaps the inclusion of the --compressed option in zfs send should be removed, or at least made configurable?

@psy0rz psy0rz added the bug label May 30, 2021
@psy0rz psy0rz added this to the 3.1 milestone May 30, 2021
@psy0rz
Copy link
Owner

psy0rz commented May 30, 2021

Yes indeed this is a problem now that we have --compress. Also i didnt realize sometimes the administrator might want to recompress on the other side.

We could either:

  • Disable it automaticly when the user specifies --compress.
  • Disable it by default and have an extra option called --zfs-compressed that enables zfs send --compressed.

I'm for the second option since i want to keep automatic setting guessing/magic to the minimum and we should let the administrator decide which function is desired.

Perhaps a warning when both --zfs-compressed and --compress are specified.

Anyone following this, let me know your opinion about this. Justice will be swift :)

@Scrin
Copy link
Author

Scrin commented May 30, 2021

The second option sounds better to me. It would also allow a scenario where compression to be applied only on the receiving side, which can be beneficial in cases where the network link is fast and the source host cpu is slow.

@psy0rz psy0rz closed this as completed in 37f91e1 May 31, 2021
@psy0rz
Copy link
Owner

psy0rz commented May 31, 2021

fixed in the rc3

@psy0rz
Copy link
Owner

psy0rz commented May 31, 2021

also updated documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants