Interpolate environment variables

[aanand]

Uses string.Template. No docs yet, but here's what works:

web:
  # unbracketed name
  image: "$IMAGE"

  # bracketed name
  command: "${COMMAND}"

  # array element
  ports:
    - "${HOST_PORT}:8000"

  # dictionary item value
  labels:
    mylabel: "${LABEL_VALUE}"

  # unset value - this will expand to "host-"
  hostname: "host-${UNSET_VALUE}"

  # escaped interpolation - this will expand to "${ESCAPED}"
  command: "$${ESCAPED}"

Closes #1377.

[elgalu]

Will accept default values? e.g. ${HOST_PORT || 8080} falls back to 8080 if HOST_PORT is undefined.

[dnephin]

Using safe_substitute() would be closer to how os.path.expandvars() worked previously (when keys are missing, leave the original value).

Otherwise this except should also include KeyError, which is raised when the value isn't in the environment (which I think should be a test case).

[dnephin]

Oh, I just realized that is what defaultdict is handling. So the result of a missing value would be empty string instead of leaving the original string. That should work.

[aanand]

I've updated the test file so that the "unset variable" case gets its own test, just so it's clear we do handle that case.

[dnephin]

I like it.

What do you think about moving all the interpolation stuff into a new module (since right now it's spread across config.py and utils.py) ?

I'm trying to figure out if we could extend this to support default values (not right now, but maybe at some later time). From what I can see, we'd have to duplicate a bunch of Template.substitute() in a subclass to handle that case.

[aanand]

Yeah, makes sense to move it into a new module.

We might be able to support default values by extending Template, although the amount of regular expression magic going on is already worrying.

@elgalu No, it doesn't support default values. If we choose to support them in the future, the syntax should ideally be the same as the POSIX syntax, i.e. ${HOST_PORT-8080} or ${HOST_PORT:-8080} (or both).

[aanand]

Had a go at some docs - ping @moxiegirl

[aanand]

I've split out interpolation logic into its own module, which necessitated some preliminary refactoring of compose.config.

[moxiegirl]

@aanand a few tweaks:

Variable substitution

Your configuration options can contain environment variables. Compose uses the variable values from the shell environment in which docker-compose is run. For
example, suppose the shell contains POSTGRES_VERSION=9.3 and you supply this configuration:

db:
  image: "postgres:${POSTGRES_VERSION}"

When you run docker-compose up with this configuration, Compose looks for the
POSTGRES_VERSION environment variable in the shell and substitutes its value in. For this example, Compose resolves the image to postgres:9.3 before running the configuration.

If an environment variable is not set, Compose substitutes with an empty
string. In the example above, if POSTGRES_VERSION is not set, the value for
the image option is postgres:.

Both $VARIABLE and ${VARIABLE} syntax are supported. Extended shell-style
features, such as ${VARIABLE-default} and ${VARIABLE/foo/bar}, are not
supported.

If you need to put a literal dollar sign in a configuration value, use a double
dollar sign ($$).

[aanand]

Updated.

[aanand]

One important question: when a variable is unset, should we substitute an empty string (as is currently implemented) or raise an error?

1. Substituting an empty string is POSIX-compliant. The correct way to make a variable "required" in POSIX is ${FOO?} or ${FOO?error message} (which isn't supported by String.template, but which we could implement later if we were willing to write our own parser).

2. However, raising an error is arguably better for backwards-compatibility. Users may already have $FOO-like syntax in their docker-compose.yml - for example, a command string might pluck variables out of the container's environment:

   command: sh -c "python app.py $VAR1 $VAR2"

   If we merge this feature as is, this is going to come out as sh -c "python app.py " in the highly likely event that they don't have those environment variables set in their shell. They'll need to update it to escape the dollar signs, but they probably won't know that straight away - they'll just get weird errors thanks to the arguments not being passed in to their script.

   If we instead raise an error, they'll at least know straight away.

3. Third option: show a warning if the variable is unset, but still pass in an empty string. This will make for better compliance and hopefully give transitioning users the hint they need.

[moxiegirl]

Personally, I like the flexibility of a warning (option 3). So, say I run a configuration across a set of hosts that I swap out or that aren't identical, I could have a value configured such that it is empty on one type of host but not another. Which could be useful...

[aanand]

Pushed some refactoring to make #1808 merge more easily.

Also, we now show a warning when a variable is not set:

$ cd tests/fixtures/environment-interpolation
$ docker-compose up
The IMAGE variable is not set. Substituting a blank string.
The LABEL_VALUE variable is not set. Substituting a blank string.
The HOST_PORT variable is not set. Substituting a blank string.
The UNSET_VALUE variable is not set. Substituting a blank string.
Name   Command   State   Ports
------------------------------

It could still be improved in that, if you reference the same variable multiple times, it'll show a warning for each one.

[mnowster]

Yay! This is ace 🎈

LGTM

[thaJeztah]

@rosskevin this has not been released yet (it's part of the coming 1.5 release), so if you want to test this feature, you'll have to build it yourself; you can find some instructions here https://github.com/docker/compose/blob/master/CONTRIBUTING.md

If you're question is "how to run compose" (in general); docs are here https://docs.docker.com/compose/

[marcellodesales]

Can't wait for this... I'm looking for passing environment variables for DEV and PROD instances.

[thaJeztah]

@marcellodesales a release candidate for 1.5 is available for testing, see https://github.com/docker/compose/releases/tag/1.5.0rc1. It should be used with care of course, release candidates can still contain bugs

[marcellodesales]

@thaJeztah Sounds good... I will be playing with it this week and I may report anything weird...

[ashb]

I think this caused a regression in behaviour. My docker-compose.yml looks like this:

postgres:
  container_name: mailserverdocker_postgres
  build: postgres
  environment:
    POSTGRES_PASSWORD:
    PGDATA: /var/lib/postgresql/data/pgdata
  volumes:
    # This is not on the host but on the docker-machine. Cos VMHGFS permissions
    # are fail. FULL OF FAIL AND PERMISSIONS ERRORS
    - /docker-volumes/run/pg-data:/var/lib/postgresql/data

Previously this would use the POSTGRES_PASSWORD env var from the docker-compose up - now it gives this error:

Service 'postgres' configuration key 'environment' 'POSTGRES_PASSWORD' contains None, which is an invalid type, it should be a stringnumber or a boolean

I can fix it by changing to POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} -- I'm just not sure if this regression was intentional or not? It wasn't clear from the Changelog that this would break.

Should I open another bug for this issue? (Maybe just a documentation change and a note in the changelog about this being a (small) breaking change)

[mnowster]

Hi @ashb , can you open a new issue with this info in it for me please? I can't triage and assign the issue properly if it's in a PR comment. Thank you 👍

[gchan]

+1 to fetching default variables for environment variables from a .env file.

[gchan]

^ I actually take that back. Defining a docker-compose.override.yml solved my problem :)

[dherben]

What I'd like to do is use the contents of an env-file, or environment block in variable subsitution. I.e., I have a VERSION file which contains BUILD_VERSION=1.2.3-SNAPSHOT, and I'd like to be able to use the following in my docker-compose.yml:

myworker:
  env-file: VERSION
  image: myimagename:${BUILD_VERSION}

There currently does not seem to be a way to do this and still keep using a normal docker-compose up. I can export the values from the file to the shell in a script that wraps docker-compose up, but I'd like to keep things as standard as possible for our developers. The VERSION file is already part of our build infrastructure.

Any idea how to tackle this would be much appreciated.

[aanand]

@dherben There's some discussion on related topics in the comment thread for #745, but for now you'll either have to wrap docker-compose up with something or have your developers run another command before it, e.g.

$ export `cat VERSION`
$ docker-compose up

[IAmJulianAcosta]

Hi @aanand! I want to know if you implemented default values?

[marcellodesales]

Can we also use the default values when env vars are not defined?

  version: $VERSION:0.1.0

Not defining $VERSION should lead to 0.1.0, correct?

[aanand]

Default values have not been implemented. There's a WIP implementation in #3108.

[anuragpatil94]

Since DockerCompose uses .env as default environmental file.

How do I force Docker-Compose to use a custom named environmental file? For example .env.dev.

I believe I cannot use env_file TAG since, I have DOCKER SETTINGS like ports, container name in my .env.dev file.