Skip to content

v0.17.0
Compare
Choose a tag to compare
@xenoscopic xenoscopic released this 21 Feb 16:23
· 36 commits to release-branch-v0.17.x since this release
v0.17.0
2855c81
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Overview

Mutagen v0.17 is an important release for Mutagen. This release is primarily focused on synchronization performance and is the first release to include some features under a paid subscription called Mutagen Pro. The release notes here will primarily focus on the technical changes, but we've created a blog post explaining the reasoning behind the subscription and licensing changes in v0.17.

Licensing changes

For more detailed information on this topic, please see the aforementioned blog post, but the short summary is that Mutagen's official release binaries will now include code licensed under the Server Side Public License. It is still possible to build release binaries that contain only MIT-licensed and non-copyleft-licensed code, and these binaries are still interoperable with the official Mutagen release builds.

Mutagen Pro functionality

The Mutagen Pro subscription, in addition to providing full access to the Mutagen Extension for Docker Desktop, will also begin including features in Mutagen and Mutagen Compose. The first two features to ship under this subscription are xxHash and Zstandard support.

After signing up for a Mutagen Pro subscription on mutagen.io, you can enter your license key into Mutagen by typing mutagen license activate. For more information on the new Mutagen Pro license commands, see mutagen help license.

xxHash

The xxHash family of hashing algorithms provides extremely fast hashing and improves Mutagen's filesystem scanning speed, especially on systems without SHA-1 hardware acceleration. Mutagen has selected XXH128 (a 128-bit hash in the xxHash family) for its initial xxHash support. In tests, use of XXH128 for cold filesystem scans (i.e. those without cached digests) saw a 10-90% reduction in scan time (depending on the hardware) when compared with SHA-1. With XXH128's smaller digest size, filesystem snapshots and caches stored in Mutagen's data directory will also be approximately 20% smaller than with SHA-1. A similar reduction in data volume will be seen when Mutagen transfers filesystem snapshot data and file digests over the wire (meaning faster synchronization).

Some users may note that xxHash hash functions are not cryptographic hashes and wonder if that's cause for concern given that Mutagen currently uses SHA-1 by default. Mutagen's use of SHA-1 is not for cryptographic purposes, but simply for differential-transfer and content-addressing purposes. This is part of the reason that SHA-1's cryptographic brokenness is not a huge issue for Mutagen. Mutagen did choose SHA-1 prior to its break, but it was primarily chosen for speed due its broad support for assembly-based implementations. XXH128 still provides sufficiently high entropy to meet Mutagen's needs. Even xxHash's lower-entropy variants are sufficient for file synchronization, as demonstrated by tools like rsync.

To specify use of XXH128, you can use the new -H/--hash flags in mutagen sync create:

mutagen sync create --hash=xxh128 ...

You can also specify use of XXH128 via YAML in ~/.mutagen.yml or Mutagen session definitions in projects or Mutagen Compose:

sync:
  sessionName | defaults:
    ...
    hash: xxh128
    ...

Zstandard

The Zstandard compression algorithm provides both improved compression ratios and throughput compared with Mutagen's default DEFLATE-based compression. General benchmarks can be found here (in this case, zlib is referring to DEFLATE), though note that the performance ratio will depend heavily on the implementation, your hardware, and the data being compressed. Benchmarks for the Zstandard implementation being used by Mutagen can be found here (with Mutagen using a default level of 6 for DEFLATE and 3 for Zstandard). This implementation is particularly performant on x86_64 hardware and Apple Silicon chips.

To specify use of Zstandard, you can use the new --compression flag in mutagen sync create:

mutagen sync create --compression=zstandard ...

There are also endpoint-specific variants of this flag named --compression-alpha and --compression-beta, which can be useful for syncing between endpoints with different Zstandard performance characteristics. In certain cases, the default DEFLATE-based compression may be faster than Zstandard, especially on hardware like Raspberry Pis.

You can also specify use of Zstandard via YAML in ~/.mutagen.yml or Mutagen session definitions in projects or Mutagen Compose:

sync:
  sessionName | defaults:
    ...
    compression:
      algorithm: zstandard
    ...

Because the compression algorithm can be specified on an endpoint-specific basis, it can also appear in endpoint-specific YAML configuration, for example:

sync:
  sessionName:
    ...
    configurationBeta:
      compression:
        algorithm: zstandard

Staging optimization

Significant effort has been made to optimize Mutagen's file staging pipeline. This includes use of XXH128 for computing path digests (including for non-Mutagen Pro users), optimization of the rsync data block size, a reduction in system call overhead, and an improved architecture for compression and storage buffering. These efforts will be most visible with local synchronization that is system-call-limited (rather than bandwidth-limited), for example when synchronizing with Docker containers running in Docker Desktop.

Improved DEFLATE implementation

Mutagen has switched from Go's standard library DEFLATE implementation to an optimized implementation. Comparative benchmarks are available here, but the tl;dr version is that the default DEFLATE-based compression used in synchronization should see a roughly 90% increase in throughput compared with the standard library implementation.

SHA-256

For users who do want a cryptographic hash, Mutagen v0.17 is also adding support for SHA-256 for all users, though note that this will incur some minor additional performance and storage overhead. You can select SHA-256 hashing by using the -H/--hash flag mentioned above (or the corresponding YAML) and specifying sha256.

Ability to disable compression

Although not common, users may wish to disable compression entirely. You can do so by using the --compression flags mentioned above (or the corresponding YAML) and specifying none.

Changes

A full accounting of changes since v0.16.x can be found here. Notable changes include:

  • Added support for XXH128 hashing [Mutagen Pro]
  • Added support for SHA-256 hashing
  • Added support for Zstandard compression [Mutagen Pro]
  • Added ability to disable compression
  • Added -H/--hash flag to mutagen sync create
    • Options include sha1, sha256, and xxh128
  • Added --compression, --compression-alpha, and --compression-beta flags to mutagen sync create
    • Options include deflate, zstandard, and none
  • Add license commands to support Mutagen Pro licensing
  • Renamed --sync-mode to --mode flag in mutagen sync create
    • The old flag is still supported and simply aliased to --mode
  • Modified -c/--configuration-file flags in create commands to support multiple specifications
    • Configuration files made by later specifications are merged on top of earlier files
  • Optimized file staging pipeline
  • Switched to optimized DEFLATE implementation
  • Updated dependencies
  • Updated to Go 1.20.1
    • Go 1.19.x is still supported
Assets 41
Loading