Skip to content

Overview

This is large and exciting release with a number of new features, including network forwarding and session orchestration! The goal of this release is to enable developers to create remote containerized development environments that work with their local tools. For more information, check out the recently renovated project site at https://mutagen.io.

Please bear in mind that network forwarding and session orchestration are still a little experimental and subject to change, so please report any feedback or issues you encounter on the issue tracker or Spectrum chat.

Changes

This release includes the following changes from the v0.9.x series:

  • Added support for port forwarding between endpoints, including TCP and Unix domain socket forwarding (see below)
  • Added support for project-based session orchestration (see below)
  • Switched to YAML for configuration (see below)
    • Legacy TOML synchronization configurations are still supported, but deprecated
  • Reorganized Mutagen command structure to account for expanded functionality
    • Synchronization session management commands are still available as hidden commands directly under the root command in order to avoid breaking existing scripts and tools
  • Sessions can now be created in a paused state using the -p/--paused flag
  • Sessions can now have user-friendly names assigned using the -n/--name flag
  • Session specifications must now be either a session name or session identifier (fuzzy matching is no longer supported)
  • Added the ability to customize the Mutagen data directory location using the MUTAGEN_DATA_DIRECTORY environment variable, allowing for isolated daemon instances (or just simple relocation of the directory)
  • Modified session management commands to automatically start the daemon if not running
    • This behavior can be disabled by setting the environment variable MUTAGEN_DISABLE_AUTOSTART=1
  • Added logging infrastructure
  • Switched the default scan mode to accelerated

Forwarding

With v0.10.0, Mutagen adds support for network forwarding sessions. These sessions function very similarly to Mutagen's synchronization sessions, with a few necessary differences.

Forwarding sessions are created using the mutagen forward create command:

mutagen forward create <source> <destination>

Unlike with synchronization, where endpoints are referred to as "alpha" and "beta", forwarding endpoints are referred to as "source" and "destination" since network forwarding is, by its nature, unidirectional.

Forwarding endpoint URLs are similar to synchronization endpoint URLs, except that they encode a network address instead of a filesystem path.

Type Format
Local <network-endpoint>
Docker docker://[<user>@]<container>:<network-endpoint>
SSH [<user>@]<host>[:<port>]:<network-endpoint>

Each <network-endpoint> has the form <protocol>:<address>. The <protocol> and <address> components allow the same values as the network and address arguments (respectively) of Go's net.Dial and net.Listen functions, except that <protocol> is restricted to tcp, tcp4, tcp6, and unix.

As with synchronization, each endpoint URL can be either local or remote, and they can be combined in arbitrary ways. Basic network forwarding might look like:

mutagen forward create tcp:localhost:8080 docker://appcontainer:tcp:localhost:8080

whereas reverse forwarding would look like:

mutagen forward create docker://appcontainer:tcp:localhost:8080 tcp:localhost:8080

You can even do remote-to-remote forwarding, e..g.:

mutagen forward create user@example.org:tcp:[::1]:8080 : docker://appcontainer:tcp:localhost:8080

With Unix domain socket forwarding, you can expose remote resources:

mutagen forward create unix:~/Projects/docker.sock user@host.example.org:/var/run/docker.sock

You can even mix-and-match protocols:

mutagen forward create unix:~/mysocket.sock user@example.org:tcp:localhost:9090

Forwarding sessions can then be managed in a very similar fashion to synchronization sessions with the following commands:

  • mutagen forward list
  • mutagen forward monitor
  • mutagen forward pause
  • mutagen forward resume
  • mutagen forward terminate

Mutagen's network forwarding is still new and experimental, so please provide feedback about your experience via the Spectrum chat or issue tracker.

YAML-based configuration

To help support Mutagen's growing functionality, configuration has been switched to use YAML instead. The legacy TOML-based configuration (such as ~/.mutagen.toml and files passed to mutagen sync create with the -c/--configuration-file flag) will still be supported, but support for new configuration options, including network forwarding options, will not be added to these files.

The Mutagen YAML format looks like:

sync:
  defaults:
    mode: ...
    maxEntryCount: ...
    maxStagingFileSize: ...
    probeMode: ...
    scanMode: ...
    stageMode: ...
    symlink:
      mode: ...
    watch:
      mode: ...
      pollingInterval: ...
    ignore:
      paths:
        - ...
        - ...
      vcs: ...
    permissions:
      defaultFileMode: ...
      defaultDirectoryMode: ...
      defaultOwner: ...
      defaultGroup: ...

forward:
  defaults:
    socket:
      overwriteMode: ...
      owner: ...
      group: ...
      permissionMode: ...

All fields optional and the syntax for field values is otherwise unchanged (except where YAML requires certain values to be quoted).

Mutagen now looks for its global configuration file in ~/.mutagen.yml (falling back to ~/.mutagen.toml, if it exists, for synchronization sessions) and the -c/--configuration-file flags have been updated to support this YAML-based format.

Orchestration

Building on its new YAML-based configuration, Mutagen is adding experimental support for synchronization and forwarding session orchestration via the mutagen project command. This lets you create project configurations defining the synchronization and forwarding sessions for your project and have Mutagen automatically manage those sessions. This is particularly powerful when combined with a tool like Docker Compose to set up sessions. This functionality is still in its infancy, so any and all feedback is welcome, though either the Spectrum chat or issue tracker.

In order to use this functionality, you need to create a mutagen.yml file that contains your session configuration(s). The format for this file is based off of the YAML layout defined above. In addition to a defaults key, you can specify named sessions with pre-defined endpoint URLs and configuration options. An example of this file can be found in the Mutagen repository, along with a corresponding Docker Compose configuration and underlying Dockerfile.

Using the orchestration infrastructure, your workflow might look something like:

# Start Docker Compose services
docker-compose up --build --force-recreate --detach
# Start Mutagen project sessions
mutagen project start

You can then work interactively in containers by starting a shell with a command like:

docker-compose exec <service-name> /bin/sh

Finally, to shut down the orchestration infrastructure, you can do:

# Terminate Mutagen project sessions
mutagen project terminate
# Terminate Docker Compose services
docker-compose down --rmi=all

Once you have that workflow set up, you can even move the entire development infrastructure off of your laptop using the DOCKER_HOST environment variable when invoking docker-compose up, mutagen project start, and docker-compose down (Mutagen will lock in DOCKER_HOST and other Docker environment variables and doesn't need them re-specified for each command).

The mutagen project command also provides other subcommands for managing projects, including:

  • mutagen project list (to list project sessions and their status)
  • mutagen project flush (to flush project synchronization sessions)
  • mutagen project pause (to pause project sessions)
  • mutagen project resume (to resume paused sessions or sessions that require authentication)
  • mutagen project terminate (to terminate project sessions)
Assets 30
Pre-release

@havoc-io havoc-io released this Jul 25, 2019

NOTE: This is an experimental, beta-quality, pre-release version of Mutagen. It may have unknown issues. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license). For a better experience, you may wish to use the latest stable release.

NOTE: Sessions created with this release are not guaranteed to be compatible with the final release version. Please wait for the final tagged release (v0.10.0 - which will be released shortly) if you want to be sure that you can avoid recreating sessions. Documentation is also still pending for the new features in this release.

Changes

This release includes the following changes from v0.10.0-beta1:

  • Clarified usage for mutagen project commands

For a more extensive summary of the changes from the v0.9.x series, see the v0.10.0-beta1 release notes.

Assets 30
Pre-release

@havoc-io havoc-io released this Jul 24, 2019

NOTE: This is an experimental, beta-quality, pre-release version of Mutagen. It may have unknown issues. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license). For a better experience, you may wish to use the latest stable release.

NOTE: Sessions created with this release are not guaranteed to be compatible with the final release version. Please wait for the final tagged release (v0.10.0 - which will be released shortly) if you want to be sure that you can avoid recreating sessions. Documentation is also still pending for the new features in this release.

Changes

This release includes the following changes from the v0.9.x series:

  • Added support for port forwarding between endpoints, including TCP and Unix domain socket forwarding (see below)
  • Added support for project-based session orchestration (see below)
  • Switched to YAML for configuration (see below)
    • Legacy TOML synchronization configurations are still supported, but deprecated
  • Reorganized Mutagen command structure to account for expanded functionality
    • Synchronization session management commands are still available as hidden commands directly under the root command in order to avoid breaking existing scripts and tools
  • Sessions can now be created in a paused state using the -p/--paused flag
  • Sessions can now have user-friendly names assigned using the -n/--name flag
  • Session specifications must now be either a session name or session identifier (fuzzy matching is no longer supported)
  • Added the ability to customize the Mutagen data directory location using the MUTAGEN_DATA_DIRECTORY environment variable, allowing for isolated daemon instances (or just simple relocation of the directory)
  • Modified session management commands to automatically start the daemon if not running
    • This behavior can be disabled by setting the environment variable MUTAGEN_DISABLE_AUTOSTART=1
  • Added logging infrastructure
  • Switched the default scan mode to accelerated

Forwarding

With v0.10.0, Mutagen adds support for network forwarding sessions. These sessions function very similarly to Mutagen's synchronization sessions, with a few necessary differences.

Forwarding sessions are created using the mutagen forward create command:

mutagen forward create <source> <destination>

Unlike with synchronization, where endpoints are referred to as "alpha" and "beta", forwarding endpoints are referred to as "source" and "destination" since network forwarding is, by its nature, unidirectional.

Forwarding endpoint URLs are similar to synchronization endpoint URLs, except that they encode a network address instead of a filesystem path.

Type Format
Local <network-endpoint>
Docker docker://[<user>@]<container>:<network-endpoint>
SSH [<user>@]<host>[:<port>]:<network-endpoint>

Each <network-endpoint> has the form <protocol>:<address>. The <protocol> and <address> components allow the same values as the network and address arguments (respectively) of Go's net.Dial and net.Listen functions, except that <protocol> is restricted to tcp, tcp4, tcp6, and unix.

As with synchronization, each endpoint URL can be either local or remote, and they can be combined in arbitrary ways. Basic network forwarding might look like:

mutagen forward create tcp:localhost:8080 docker://appcontainer:tcp:localhost:8080

whereas reverse forwarding would look like:

mutagen forward create docker://appcontainer:tcp:localhost:8080 tcp:localhost:8080

You can even do remote-to-remote forwarding, e..g.:

mutagen forward create user@example.org:tcp:[::1]:8080 : docker://appcontainer:tcp:localhost:8080

With Unix domain socket forwarding, you can expose remote resources:

mutagen forward create unix:~/Projects/docker.sock user@host.example.org:/var/run/docker.sock

You can even mix-and-match protocols:

mutagen forward create unix:~/mysocket.sock user@example.org:tcp:localhost:9090

Forwarding sessions can then be managed in a very similar fashion to synchronization sessions with the following commands:

  • mutagen forward list
  • mutagen forward monitor
  • mutagen forward pause
  • mutagen forward resume
  • mutagen forward terminate

Mutagen's network forwarding is still new and experimental, so please provide feedback about your experience via the Spectrum chat or issue tracker.

YAML-based configuration

To help support Mutagen's growing functionality, configuration has been switched to use YAML instead. The legacy TOML-based configuration (such as ~/.mutagen.toml and files passed to mutagen sync create with the -c/--configuration-file flag) will still be supported, but support for new configuration options, including network forwarding options, will not be added to these files.

The Mutagen YAML format looks like:

sync:
  defaults:
    mode: ...
    maxEntryCount: ...
    maxStagingFileSize: ...
    probeMode: ...
    scanMode: ...
    stageMode: ...
    symlink:
      mode: ...
    watch:
      mode: ...
      pollingInterval: ...
    ignore:
      paths:
        - ...
        - ...
      vcs: ...
    permissions:
      defaultFileMode: ...
      defaultDirectoryMode: ...
      defaultOwner: ...
      defaultGroup: ...

forward:
  defaults:
    socket:
      overwriteMode: ...
      owner: ...
      group: ...
      permissionMode: ...

All fields optional and the syntax for field values is otherwise unchanged (except where YAML requires certain values to be quoted).

Mutagen now looks for its global configuration file in ~/.mutagen.yml (falling back to ~/.mutagen.toml, if it exists, for synchronization sessions) and the -c/--configuration-file flags have been updated to support this YAML-based format.

Orchestration

Building on its new YAML-based configuration, Mutagen is adding experimental support for synchronization and forwarding session orchestration via the mutagen project command. This lets you create project configurations defining the synchronization and forwarding sessions for your project and have Mutagen automatically manage those sessions. This is particularly powerful when combined with a tool like Docker Compose to set up sessions. This functionality is still in its infancy, so any and all feedback is welcome, though either the Spectrum chat or issue tracker.

In order to use this functionality, you need to create a mutagen.yml file that contains your session configuration(s). The format for this file is based off of the YAML layout defined above. In addition to a defaults key, you can specify named sessions with pre-defined endpoint URLs and configuration options. An example of this file can be found in the Mutagen repository, along with a corresponding Docker Compose configuration and underlying Dockerfile.

Using the orchestration infrastructure, your workflow might look something like:

# Start Docker Compose services
docker-compose up --build --force-recreate --detach
# Start Mutagen project sessions
mutagen project start

You can then work interactively in containers by starting a shell with a command like:

docker-compose exec <service-name> /bin/sh

Finally, to shut down the orchestration infrastructure, you can do:

# Terminate Mutagen project sessions
mutagen project terminate
# Terminate Docker Compose services
docker-compose down --rmi=all

Once you have that workflow set up, you can even move the entire development infrastructure off of your laptop using the DOCKER_HOST environment variable when invoking docker-compose up, mutagen project start, and docker-compose down (Mutagen will lock in DOCKER_HOST and other Docker environment variables and doesn't need them re-specified for each command).

The mutagen project command also provides other subcommands for managing projects, including:

  • mutagen project list (to list project sessions and their status)
  • mutagen project flush (to flush project synchronization sessions)
  • mutagen project pause (to pause project sessions)
  • mutagen project resume (to resume paused sessions or sessions that require authentication)
  • mutagen project terminate (to terminate project sessions)
Assets 30

@havoc-io havoc-io released this Jul 17, 2019

NOTE: This is a beta-quality release. It may have unknown issues. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license).

Changes

This release includes the following changes from v0.9.1:

  • Fixed a feedback loop in Linux filesystem watching that cause high CPU usage
  • Updated build to use Go 1.12.7
Assets 30

@havoc-io havoc-io released this Jun 12, 2019

NOTE: This is a beta-quality release. It may have unknown issues. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license).

Changes

This release includes the following changes from v0.9.0:

  • Updated build to use Go 1.12.6 (which includes a fix for CVE-2019-11888)
Assets 30

@havoc-io havoc-io released this May 29, 2019

NOTE: This is a beta-quality release. It may have unknown issues. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license).

Changes

This release includes the following changes from the v0.8.x series:

  • Added a "scan mode" option that allows for accelerated filesystem scans that reduce synchronization latency by utilizing filesystem watching information. This is controlled by the --scan-mode flag in the create command and the scanMode setting in the [sync] section of ~/.mutagen.toml. The options for this flag are full (the default) and accelerated. Please be aware that accelerated scans are experimental, though testing is appreciated! Using mutagen flush always forces a full scan.
  • Support for session labels has been added. The syntax and semantics of these labels are identical to those of Kubernetes (in fact the underlying implementation is shared). Labels can be attached to a session at creation time using the create commands -l/--label flag, e.g. --label=mutagen=awesome. Multiple labels can be created by specifying this flag repeatedly. Labels can be used to identify sessions in the list, monitor, flush, pause, resume, and terminate commands using the --label-selector flag, e.g. `--label-selector='mutagen==awesome'.
  • Added a "probe mode" option and optimized fast paths that allow endpoints to avoid probing filesystem behavior with temporary files. This is controlled by the --probe-mode flag in the create command and the probeMode setting in the [sync] section of ~/.mutagen.toml. The options for this flag are probe (the default) and assume. With the probe setting, Mutagen will attempt to determine filesystem behavior using filesystem queries, falling back to probe files if these queries aren't available or the filesystem format is unknown. Using assume will cause endpoints to assume session behavior based on the platform, which is less accurate but significantly faster.
  • Added a "stage mode" option that controls where Mutagen stages files before transforming a synchronization root, allowing for more efficient synchronization when working across volumes. This is controlled by the --stage-mode flag in the create command and the stageMode setting in the [sync] section of ~/.mutagen.toml. The options for this flag are mutagen (the default) and neighboring. With the mutagen setting, files will be staged in the Mutagen data directory (~/.mutagen). With the neighboring setting, files will be staged in a temporary directory that neighbors the synchronization root.
  • Added a --no-global-configuration flag to the create command to exclude the global configuration (~/.mutagen.toml) from the session configuration
  • Added the ability to specify a custom TOML configuration file (with the same format as ~/.mutagen.toml) to the create command using the -c/--configuration-file flag. This file will be loaded and merged on top of the ~/.mutagen.toml file (unless that file has been disabled with the --no-global-configuration flag).
  • Filesystem watching has been completely refactored and should be significantly more reliable (though still limited on platforms without recursive watching support)
  • Filesystem probing using temporary files has been modified to work on filesystems that don't support POSIX-compliant behavior
  • Added a check to all commands that they are communicating with a compatible version of the daemon.
    Compatibility is currently restricted to daemons with the same version. The Mutagen daemon termination API is frozen, so mutagen daemon stop will always work to terminate an older daemon instance.
  • Added myriad internal optimizations and fixes
  • Performed significant code reorganization and refactoring
  • Updated code and module files for Go 1.12
  • Increased Go version dependency to 1.12
  • Updated all dependencies

Platforms

The following platform support has been added:

  • Windows ARM
Assets 30
Pre-release

@havoc-io havoc-io released this May 25, 2019

NOTE: This is an experimental, beta-quality, pre-release version of Mutagen. This release in particular is very large and contains changes to significant portions of Mutagen's implementation. It may have unknown issues. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license). For a better experience, you may wish to use the latest stable release.

NOTE: Sessions created with previous non-beta Mutagen releases are forward-compatible, but sessions created with beta releases are not guaranteed to be compatible with their corresponding final release version. Please wait for the final tagged release (v0.9.0 - which will be released shortly) if you want to be sure that you can avoid recreating sessions. Documentation is also still pending for the new features in this release.

Changes

This release includes the following changes from v0.9.0-beta2:

  • Fixed and improved printout of labels in session listing and monitoring

Changes since the v0.8.x release series can be found in the release notes for v0.9.0-beta2.

Assets 30
Pre-release

@havoc-io havoc-io released this May 23, 2019

NOTE: This is an experimental, beta-quality, pre-release version of Mutagen. This release in particular is very large and contains changes to significant portions of Mutagen's implementation. It may have unknown issues. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license). For a better experience, you may wish to use the latest stable release.

NOTE: Sessions created with previous non-beta Mutagen releases are forward-compatible, but sessions created with beta releases are not guaranteed to be compatible with their corresponding final release version. Please wait for the final tagged release (v0.9.0 - which will be released shortly) if you want to be sure that you can avoid recreating sessions. Documentation is also still pending for the new features in this release.

Changes

This release includes the following changes from the v0.8.x series:

  • Added a "scan mode" option that allows for accelerated filesystem scans that reduce synchronization latency by utilizing filesystem watching information. This is controlled by the --scan-mode flag in the create command and the scanMode setting in the [sync] section of ~/.mutagen.toml. The options for this flag are full (the default) and accelerated. Please be aware that accelerated scans are experimental, though testing is appreciated! Using mutagen flush always forces a full scan.
  • Support for session labels has been added. The syntax and semantics of these labels are identical to those of Kubernetes (in fact the underlying implementation is shared). Labels can be attached to a session at creation time using the create commands -l/--label flag, e.g. --label=mutagen=awesome. Multiple labels can be created by specifying this flag repeatedly. Labels can be used to identify sessions in the list, monitor, flush, pause, resume, and terminate commands using the --label-selector flag, e.g. `--label-selector='mutagen==awesome'.
  • Added a "probe mode" option and optimized fast paths that allow endpoints to avoid probing filesystem behavior with temporary files. This is controlled by the --probe-mode flag in the create command and the probeMode setting in the [sync] section of ~/.mutagen.toml. The options for this flag are probe (the default) and assume. With the probe setting, Mutagen will attempt to determine filesystem behavior using filesystem queries, falling back to probe files if these queries aren't available or the filesystem format is unknown. Using assume will cause endpoints to assume session behavior based on the platform, which is less accurate but significantly faster.
  • Added a "stage mode" option that controls where Mutagen stages files before transforming a synchronization root, allowing for more efficient synchronization when working across volumes. This is controlled by the --stage-mode flag in the create command and the stageMode setting in the [sync] section of ~/.mutagen.toml. The options for this flag are mutagen (the default) and neighboring. With the mutagen setting, files will be staged in the Mutagen data directory (~/.mutagen). With the neighboring setting, files will be staged in a temporary directory that neighbors the synchronization root.
  • Added a --no-global-configuration flag to the create command to exclude the global configuration (~/.mutagen.toml) from the session configuration
  • Added the ability to specify a custom TOML configuration file (with the same format as ~/.mutagen.toml) to the create command using the -c/--configuration-file flag. This file will be loaded and merged on top of the ~/.mutagen.toml file (unless that file has been disabled with the --no-global-configuration flag).
  • Filesystem watching has been completely refactored and should be significantly more reliable (though still limited on platforms without recursive watching support)
  • Filesystem probing using temporary files has been modified to work on filesystems that don't support POSIX-compliant behavior
  • Added a check to all commands that they are communicating with a compatible version of the daemon.
    Compatibility is currently restricted to daemons with the same version. The Mutagen daemon termination API is frozen, so mutagen daemon stop will always work to terminate an older daemon instance.
  • Added myriad internal optimizations and fixes
  • Performed significant code reorganization and refactoring
  • Updated code and module files for Go 1.12
  • Increased Go version dependency to 1.12
  • Updated all dependencies

Platforms

The following platform support has been added:

  • Windows ARM
Assets 30

@havoc-io havoc-io released this May 22, 2019

NOTE: This is a beta-quality release. It may have unknown issues. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license).

This release includes the following changes from v0.8.2:

  • Updated build to use Go 1.12.5
Assets 29
Pre-release

@havoc-io havoc-io released this Apr 29, 2019

NOTE: This is an experimental, beta-quality, pre-release version of Mutagen. It may have unknown issues. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license). For a better experience, you may wish to use the latest stable release.

NOTE: Sessions created with this release are not guaranteed to be compatible with the final release version. Please wait for the final tagged release (v0.9.0 - which will be released shortly) if you want to be sure that you can avoid recreating sessions. Documentation is also still pending for the new features in this release.

This release includes the following changes from the v0.8.x series:

  • Added the ability to specify the filesystem behavior probing mechanism (including the ability to avoid probe files)
  • Optimized filesystem scans by reducing allocations
  • Updated code and module files for Go 1.12
  • Increased Go version dependency to 1.12
  • Updated all dependencies
Assets 30
You can’t perform that action at this time.