Support for environment variables in mutagen.yml file

[derschatta]

I am not sure if that might work already as I wasn't able to properly test it yet (see #127) but it would be awesome if I could use environment variables in the mutagen.yml files, i.e. for the alpha and beta config.

We use a .env file where a user can specify his own paths and those get used in bash files and in docker-compose files.

[xenoscopic]

That makes sense to me. I'll queue it up on the roadmap.

[derschatta]

Great, thanks @havoc-io

Awesome work by the way :D

[Toilal]

2 things I wish to land with this feature:

• It should support default values when environment variable is not set, with the same syntax as is bash

beta: '${MUTAGEN_BETA:-root@192.168.1.100:/home/vagrant/projects/my-project}'

• Maybe some default environment variable could be set by mutagen. (ie: the project directory basename and fullpath, to be able to bring alpha and beta configuration in the default configuration)

sync:
  defaults:
    alpha: .
    beta: '${MUTAGEN_BETA:-root@192.168.1.100:/home/vagrant/projects/${MUTAGEN_SYNC_ROOT_BASENAME}'

In this example, MUTAGEN_SYNC_ROOT_BASENAME is provided by MUTAGEN and automatically match the synchronisation root directory name.

[xenoscopic]

Thanks @Toilal for the additional input. If Mutagen does implement this, having a set of built-in environment variables could be interesting. This would also align well with the text/template-based approach mentioned below.

In any case, I've been looking at implementing this for v0.11.x, but I'd like to get that shipped ASAP, so I think I'm going to move this to the roadmap for v0.12.x and flag it as a backport candidate for a later v0.11.x release.

I've been looking at what other related projects and tools do here and I'm not sure there's a clear consensus on how this should work or whether it should even be supported, so I'm hoping for some additional opinions/feedback.

Docker Compose is the simplest case, and it supports a shell-inspired syntax and .env files in certain cases. It sounds like this is what we all had in mind. There are some behavior details that I'm still not sure about though (see below).

Helm seems to have rejected environment variables, at least in the incarnation proposed here. Their concerns were security (since the environment variables would need to be bundled and sent to the remote server) and reproducibility. Mutagen's project files don't really have the same security concerns since they're processed locally, but the reproducibility aspect of things is something that concerns me. One of the big benefits of doing containerized development is that you can create reproducible environments that work across a team. Having environment variables that can parameterize a project file do undercut that pretty drastically.

Kubernetes also seems to have rejected this idea in favor of recommending external tools like envsubst and others. I actually agree with this philosophy, but I'm pragmatic and willing to hear other thoughts.

I'm also wondering if we want to restrict these substitutions to only YAML value elements, because that's kind of difficult to implement. It's not difficult to perform a regular expression or text/template-based¹ processing on the unparsed YAML, but that would allow substitutions anywhere, including YAML keys and structures. If we want to restrict this substitution to only YAML values, it becomes more complex because we'd probably have to restrict the support to string values (since we can't store these expressions in any other type once parsed) and then figure out how to recurse over the configuration structures once loaded and perform the substitution (which means either large and hard-to-maintain custom code or complex reflection-based code). If we want to go one step further and support substitution for any type (while still restricting only to YAML value elements), then we basically have to teach Mutagen how to lex YAML. Such support for substitution in numeric values would be necessary in the case of size-limiting configuration values. So, in the end, we can take a simple and stupid approach and just perform substitution on the raw, unparsed YAML and it will be significantly simpler, but it will probably encourage even less reproducibility because people may abuse it to restructure or re-order YAML.

So basically 2/3 of the most relevant open-source projects (the newer ones) intentionally don't support this and the one that does support this would be tedious (though certainly not impossible) to emulate with Go. For the time being, I'll take the Kubernetes route and recommend the aforementioned tools as a workaround, but I'd like to hear from other users about (a) how they would use this, (b) what they'd like the syntax to be, and (c) how they'd like this to behave.

---

¹ Using a text/template-based approach would come at the cost of an unfamiliar syntax, except perhaps for Go and Angular developers.

[IgorVitol]

Hey! I also trying to make beta endpoint dynamic somehow. Recently we were created sync sessions from sh scripts, but now I want to create mutagen.yml with all the configurations.

@havoc-io I have few ideas:

1. Add argument to CLI - "mutagen project start" to override xml configuration. So, it will be possible to create custom script and pass dynamic values externally.
   For example (helm-style):

mutagen project start \
--set-config sync.code.alpha="myAlpha" \
--set-config sync.code.beta="myBeta"

2. Or/And make fields support SH/CMD inline (e.g. using $() ). For example:

docker://$(docker ps -f "label=myLabel=php" -f "name=fpm" -q | awk '{print $1}')/app

[markomitranic]

Currently, i use envsubst that parses a mutagen.yml.dist file and re-creates mutagen.yml on each run