New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interpolate environment variables in docker-compose.yml #1377

Closed
aanand opened this Issue Apr 30, 2015 · 108 comments

Comments

Projects
None yet
@aanand
Contributor

aanand commented Apr 30, 2015

(I'm creating a fresh issue for this as the old one has accumulated rather a lot of baggage.)

It should be possible to pass in environment variables to the value of any* configuration entry in docker-compose.yml. A lot of people want to do it, it's good for portability and I'm satisfied it's not going to cause chaos.

I have some reckons.

Required variables and optional defaults

It's useful to be able to specify that a variable that must be present in the environment, i.e. that Compose will refuse to run if it isn't. However, this will be a pain when you've got lots of them, so it should either be something you explicitly enable, or it should be possible to specify a default value.

The MVP implementation does not need to have either feature, but there should be a clear path to implementing both in a backwards-compatible way.

Syntax

There's a strong case for implementing an established standard, as long as it's not heavyweight - our requirements for functionality are minimal.

  • POSIX parameter expansion is OK. It has a few too many features, but we could implement a subset of them:
  • ${VARIABLE} - outputs empty string if VARIABLE is unset
  • ${VARIABLE-default} - outputs default if VARIABLE is unset
  • ${VARIABLE?} - errors out if VARIABLE is unset

#845 implemented a Bash-style ${VARIABLE:default} syntax, which is similar to POSIX parameter expansion but slightly different.

  • Python's format syntax would be trivial to implement, but is tied to the Python language, and has rather too many features.

Implementation

Python's os.path.expandvars function implements the most basic case of POSIX parameter expansion:

>>> from os.path import expandvars
>>> expandvars('${HOME}')
'/Users/aanand'

However, it's got at least 2 problems:

  1. An unset variable doesn't expand to an empty string - instead, it results in no expansion:
>>> expandvars('${UNSET}')
'${UNSET}'
  1. Malformed syntax doesn't error out - instead, it also results in no expansion:
>>> expandvars('${HOME')
'${HOME'

So far, #845 is the closest we've got, but I'm fundamentally wary of an implementation that relies on regular expressions. Templating is a non-trivial job, and people are going to put all kinds of broken stuff in, so we need something that's robust, strict and errors out with helpful messages. Two important requirements:

  • If someone puts in something malformed, Compose will not run.
  • It is possible to escape any of the special characters used in the template syntax.

There may well be good Python implementations of Bash-like variable interpolation out there already - if not, creating something standalone would be far preferable to bloating the Compose codebase.

*Actually, are there any configuration keys for which we shouldn't allow interpolation?

@kojiromike

This comment has been minimized.

Show comment
Hide comment
@kojiromike

kojiromike Apr 30, 2015

Contributor

How far do you want to go with these established UNIX standards? (FWIW, it's not a defacto standard, it's an actual standard.)

As someone who occasionally accidentally tries to use POSIX parameter expansions in Dockerfiles, if they were at all supported in docker-compose.yml it would make me a happy camper.

Contributor

kojiromike commented Apr 30, 2015

How far do you want to go with these established UNIX standards? (FWIW, it's not a defacto standard, it's an actual standard.)

As someone who occasionally accidentally tries to use POSIX parameter expansions in Dockerfiles, if they were at all supported in docker-compose.yml it would make me a happy camper.

@aanand

This comment has been minimized.

Show comment
Hide comment
@aanand

aanand May 1, 2015

Contributor

@kojiromike Hmm, so POSIX parameter expansion is actually what I was going for, but reading over the docs it looks like I misremembered the syntax/semantics.

Edit: I've updated my thoughts on syntax in the description.

Contributor

aanand commented May 1, 2015

@kojiromike Hmm, so POSIX parameter expansion is actually what I was going for, but reading over the docs it looks like I misremembered the syntax/semantics.

Edit: I've updated my thoughts on syntax in the description.

@Vad1mo

This comment has been minimized.

Show comment
Hide comment
@Vad1mo

Vad1mo May 1, 2015

I have been following the old thread and we urgently wanted to have this feature. Finally the pain was too big and we created a yaml preprocessor bahs script to substitute variables in POSIX style. it worked fine but eventually we stopped using it, because it had one issue. You have to run the preprocessor first and set all the parameters before you get the final solution. Now we are using the docker yaml extends feature. Because it allows us to checkin the actual configuration and just execute it on the target. We know better what is going to happen.

Even though I was a supporter of docker-compose passing variables, I am now not so sure.

As an ideal solution I would rather see docker extends done right. In a sense that would be a solution that fits both. So what is broken in docker extends? It is basically the fact that you have to write all entries in the inherited file. It is not a merge where you enter only what you want to override.

Look at out actual example and how verbose it is. There are only two lines that matter.

#Common 
elasticsearch:
  image: zinvoice/elasticsearch
  hostname: elasticsearch
  restart: always
  dns: 172.17.42.1
  ports:
    - "9200:9200"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro
    - /data/elasticsearch:/opt/elasticsearch/data/elasticsearch

logstash:
  image: zinvoice/logstash
  hostname: logstash
  dns: 172.17.42.1
  restart: always
  ports:
    - "5000:5000"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

kibana:
  image: zinvoice/kibana
  hostname: kibana
  dns: 172.17.42.1
  restart: always
  ports:
    - "5601:5601"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

logspout:
  image: zinvoice/logspout
  hostname: logspout
  command: logstash://logstash.docker:5000
  restart: always
  dns: 172.17.42.1
  ports:
    - "8003:8000"
  volumes:
    - /var/run/docker.sock:/tmp/docker.sock

doorman:
  image: zinvoice/doorman
  hostname: doorman
  restart:  always
  dns: 172.17.42.1
  ports:
    - "8085:8085"
# inherited
elasticsearch:
  extends:
    file: ../common.yml
    service: elasticsearch

logstash:
  extends:
    file: ../common.yml
    service: logstash

kibana:
  extends:
    file: ../common.yml
    service: kibana

logspout:
  extends:
    file: ../common.yml
    service: logspout

doorman:
  environment:
    - DOORMAN_GITHUB_APPID=xxxxxxxx
    - DOORMAN_GITHUB_APPSECRET=xxxxxx
  links:
    - nginxtrusted
  extends:
    file: ../common.yml
    service: doorman

So my recommendation fix docker extends an make it less verbose. You don't even have to write that much code as YAML provides all the functionality you need. If you stick with standard YAML the file could be analyzed or created by other tools and UIs.

Take a look at YAML "node anchors" and YAML "file merge" it might be the perfect solution.

FYI: this discussion continuous now on #1380

Vad1mo commented May 1, 2015

I have been following the old thread and we urgently wanted to have this feature. Finally the pain was too big and we created a yaml preprocessor bahs script to substitute variables in POSIX style. it worked fine but eventually we stopped using it, because it had one issue. You have to run the preprocessor first and set all the parameters before you get the final solution. Now we are using the docker yaml extends feature. Because it allows us to checkin the actual configuration and just execute it on the target. We know better what is going to happen.

Even though I was a supporter of docker-compose passing variables, I am now not so sure.

As an ideal solution I would rather see docker extends done right. In a sense that would be a solution that fits both. So what is broken in docker extends? It is basically the fact that you have to write all entries in the inherited file. It is not a merge where you enter only what you want to override.

Look at out actual example and how verbose it is. There are only two lines that matter.

#Common 
elasticsearch:
  image: zinvoice/elasticsearch
  hostname: elasticsearch
  restart: always
  dns: 172.17.42.1
  ports:
    - "9200:9200"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro
    - /data/elasticsearch:/opt/elasticsearch/data/elasticsearch

logstash:
  image: zinvoice/logstash
  hostname: logstash
  dns: 172.17.42.1
  restart: always
  ports:
    - "5000:5000"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

kibana:
  image: zinvoice/kibana
  hostname: kibana
  dns: 172.17.42.1
  restart: always
  ports:
    - "5601:5601"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

logspout:
  image: zinvoice/logspout
  hostname: logspout
  command: logstash://logstash.docker:5000
  restart: always
  dns: 172.17.42.1
  ports:
    - "8003:8000"
  volumes:
    - /var/run/docker.sock:/tmp/docker.sock

doorman:
  image: zinvoice/doorman
  hostname: doorman
  restart:  always
  dns: 172.17.42.1
  ports:
    - "8085:8085"
# inherited
elasticsearch:
  extends:
    file: ../common.yml
    service: elasticsearch

logstash:
  extends:
    file: ../common.yml
    service: logstash

kibana:
  extends:
    file: ../common.yml
    service: kibana

logspout:
  extends:
    file: ../common.yml
    service: logspout

doorman:
  environment:
    - DOORMAN_GITHUB_APPID=xxxxxxxx
    - DOORMAN_GITHUB_APPSECRET=xxxxxx
  links:
    - nginxtrusted
  extends:
    file: ../common.yml
    service: doorman

So my recommendation fix docker extends an make it less verbose. You don't even have to write that much code as YAML provides all the functionality you need. If you stick with standard YAML the file could be analyzed or created by other tools and UIs.

Take a look at YAML "node anchors" and YAML "file merge" it might be the perfect solution.

FYI: this discussion continuous now on #1380

@aanand

This comment has been minimized.

Show comment
Hide comment
@aanand

aanand May 1, 2015

Contributor

@Vad1mo I agree that extends falls short in your case. There are a lot of things we can do to improve that experience - could you open a separate issue for it, so we don't get sidetracked here?

Contributor

aanand commented May 1, 2015

@Vad1mo I agree that extends falls short in your case. There are a lot of things we can do to improve that experience - could you open a separate issue for it, so we don't get sidetracked here?

@Vad1mo

This comment has been minimized.

Show comment
Hide comment
@Vad1mo

Vad1mo May 1, 2015

Of course! I just wanted to highlight that this could be an easy and elegant alternative.
If compose extends gets you half the way too variable passing, then an improved compose-extends will make variable passing obsolete. Having less concepts to understand make easier for the user.

Vad1mo commented May 1, 2015

Of course! I just wanted to highlight that this could be an easy and elegant alternative.
If compose extends gets you half the way too variable passing, then an improved compose-extends will make variable passing obsolete. Having less concepts to understand make easier for the user.

@mattes

This comment has been minimized.

Show comment
Hide comment
@mattes

mattes May 2, 2015

My use case is to allow $PWD in volumes, so every developer in the team can clone a repo to wherever and paths still get mounted correctly.

elasticsearch:
  image: zinvoice/elasticsearch
  volumes:
    - $PWD:/app

mattes commented May 2, 2015

My use case is to allow $PWD in volumes, so every developer in the team can clone a repo to wherever and paths still get mounted correctly.

elasticsearch:
  image: zinvoice/elasticsearch
  volumes:
    - $PWD:/app
@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin May 2, 2015

Contributor

@mattes I believe that is already supported, I think .:/app is supported as well

Contributor

dnephin commented May 2, 2015

@mattes I believe that is already supported, I think .:/app is supported as well

@kojiromike

This comment has been minimized.

Show comment
Hide comment
@kojiromike

kojiromike May 2, 2015

Contributor

@aanand As a PoC I did a dirty hackup of POSIX PE in Python. For the Saturdays.

Contributor

kojiromike commented May 2, 2015

@aanand As a PoC I did a dirty hackup of POSIX PE in Python. For the Saturdays.

@aanand

This comment has been minimized.

Show comment
Hide comment
@aanand

aanand May 5, 2015

Contributor

@kojiromike Looks great. Let me know if you plan to continue on it.

Contributor

aanand commented May 5, 2015

@kojiromike Looks great. Let me know if you plan to continue on it.

@kojiromike

This comment has been minimized.

Show comment
Hide comment
@kojiromike

kojiromike May 5, 2015

Contributor

@aanand I intend to, but it definitely has a few bugs right now (and I think it may have been a bad idea to use shlex). Bug reports and PRs are welcome, of course.

Contributor

kojiromike commented May 5, 2015

@aanand I intend to, but it definitely has a few bugs right now (and I think it may have been a bad idea to use shlex). Bug reports and PRs are welcome, of course.

@nafg

This comment has been minimized.

Show comment
Hide comment
@nafg

nafg May 5, 2015

@dnephin how about $HOME / ~?

nafg commented May 5, 2015

@dnephin how about $HOME / ~?

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin May 9, 2015

Contributor

@nafg Both of those are supported for the host path of a volume

Contributor

dnephin commented May 9, 2015

@nafg Both of those are supported for the host path of a volume

@nafg

This comment has been minimized.

Show comment
Hide comment
@nafg

nafg May 10, 2015

@dnephin interesting, b/c somehow I ended up with a directory named '$HOME'...

nafg commented May 10, 2015

@dnephin interesting, b/c somehow I ended up with a directory named '$HOME'...

@andrerom

This comment has been minimized.

Show comment
Hide comment
@andrerom

andrerom May 11, 2015

Contributor

@aanand Like the "${VARIABLE:default}" proposal, with global_extends (or "import") this would become rather powerful.

Q: Would this allow to specify port number that is exposed to host? like - "${WEB_PORT:80}:80"?
Use case is to be able to easily spin up several instances of an app on same machine/cluster, typically listening to different ports or assigned to different local domain names.

Contributor

andrerom commented May 11, 2015

@aanand Like the "${VARIABLE:default}" proposal, with global_extends (or "import") this would become rather powerful.

Q: Would this allow to specify port number that is exposed to host? like - "${WEB_PORT:80}:80"?
Use case is to be able to easily spin up several instances of an app on same machine/cluster, typically listening to different ports or assigned to different local domain names.

@aanand

This comment has been minimized.

Show comment
Hide comment
@aanand

aanand May 11, 2015

Contributor

Yes, you'd be able to do that.

Contributor

aanand commented May 11, 2015

Yes, you'd be able to do that.

@alexanderilyin

This comment has been minimized.

Show comment
Hide comment
@alexanderilyin

alexanderilyin May 19, 2015

I'd like to use vars in volumes together with docker-compose scale my_app=3. I have this docker-compose.yml

server:
  image: alexanderilyin/docker-teamcity-server
  ports:
   - "8111:8111"
  volumes:
    - .TeamCity:/root/.BuildServer
  links:
   - mysql
mysql:
  image: alexanderilyin/docker-mysql
  volumes:
    - .MySQL:/var/lib/mysql
  environment:
    MYSQL_DATABASE: teamcity
    MYSQL_USER: teamcity
    MYSQL_PASSWORD: teamcity
    MYSQL_ALLOW_EMPTY_PASSWORD: yes
agent:
  image: alexanderilyin/docker-teamcity-agent
  links:
   - server

And I want to be able use scale for agents and use dynamical volumes for them to keep data between launches, for example:

agent:
  image: alexanderilyin/docker-teamcity-agent
  volumes:
    - .agent_{$AGENT_INSTANCE_ID}:/opt/buildAgent
  links:
   - server

alexanderilyin commented May 19, 2015

I'd like to use vars in volumes together with docker-compose scale my_app=3. I have this docker-compose.yml

server:
  image: alexanderilyin/docker-teamcity-server
  ports:
   - "8111:8111"
  volumes:
    - .TeamCity:/root/.BuildServer
  links:
   - mysql
mysql:
  image: alexanderilyin/docker-mysql
  volumes:
    - .MySQL:/var/lib/mysql
  environment:
    MYSQL_DATABASE: teamcity
    MYSQL_USER: teamcity
    MYSQL_PASSWORD: teamcity
    MYSQL_ALLOW_EMPTY_PASSWORD: yes
agent:
  image: alexanderilyin/docker-teamcity-agent
  links:
   - server

And I want to be able use scale for agents and use dynamical volumes for them to keep data between launches, for example:

agent:
  image: alexanderilyin/docker-teamcity-agent
  volumes:
    - .agent_{$AGENT_INSTANCE_ID}:/opt/buildAgent
  links:
   - server
@Maxim-Filimonov

This comment has been minimized.

Show comment
Hide comment
@Maxim-Filimonov

Maxim-Filimonov May 20, 2015

I hope it would be possible to interpolate variables as part of image name too
We are using https://github.com/openshift/source-to-image to build local container on CI for every branch and then run tests on it using docker-compose.
Running tests with dynamic image is quite complicated with docker-compose and requires manual template rendering.. 👎

Maxim-Filimonov commented May 20, 2015

I hope it would be possible to interpolate variables as part of image name too
We are using https://github.com/openshift/source-to-image to build local container on CI for every branch and then run tests on it using docker-compose.
Running tests with dynamic image is quite complicated with docker-compose and requires manual template rendering.. 👎

@andrerom

This comment has been minimized.

Show comment
Hide comment
@andrerom

andrerom May 20, 2015

Contributor

But you can set COMPOSE_PROJECT_NAME to control prefix per run to be able to do that already right? If so, no need to have complex logic and unreadable yml files around names.

Contributor

andrerom commented May 20, 2015

But you can set COMPOSE_PROJECT_NAME to control prefix per run to be able to do that already right? If so, no need to have complex logic and unreadable yml files around names.

@Maxim-Filimonov

This comment has been minimized.

Show comment
Hide comment
@Maxim-Filimonov

Maxim-Filimonov May 21, 2015

@andrerom don't follow. According to docs that controls the following Sets the project name, which is prepended to the name of every container started by Compose while we are trying to set an image property instead:

web:
  image: <I_AM_DYNAMIC>

Maxim-Filimonov commented May 21, 2015

@andrerom don't follow. According to docs that controls the following Sets the project name, which is prepended to the name of every container started by Compose while we are trying to set an image property instead:

web:
  image: <I_AM_DYNAMIC>
@andrerom

This comment has been minimized.

Show comment
Hide comment
@andrerom

andrerom May 21, 2015

Contributor

ah, my mistake.

Thought you meant

<I_AM_DYNAMIC>:
  image: nginx

Dynamic image (and build) reference would indeed make a lot of sense. For instance switching between debug and non debug base containers for your programming language for instance would be a good use case for this.

Contributor

andrerom commented May 21, 2015

ah, my mistake.

Thought you meant

<I_AM_DYNAMIC>:
  image: nginx

Dynamic image (and build) reference would indeed make a lot of sense. For instance switching between debug and non debug base containers for your programming language for instance would be a good use case for this.

@andrerom

This comment has been minimized.

Show comment
Hide comment
@andrerom

andrerom May 21, 2015

Contributor

Additional use case (which might be what @Maxim-Filimonov has in mind): Being able to override which tag to use of an image, so you can use :latest by default, but change to easily test something else without changing yml file (needed for CI use cases basically).

Contributor

andrerom commented May 21, 2015

Additional use case (which might be what @Maxim-Filimonov has in mind): Being able to override which tag to use of an image, so you can use :latest by default, but change to easily test something else without changing yml file (needed for CI use cases basically).

@Maxim-Filimonov

This comment has been minimized.

Show comment
Hide comment
@Maxim-Filimonov

Maxim-Filimonov May 21, 2015

@andrerom that is exactly our use case 👍

Maxim-Filimonov commented May 21, 2015

@andrerom that is exactly our use case 👍

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost May 24, 2015

Will this also work for things like??

web:
  environment:
    - FOO=${whoami}

ghost commented May 24, 2015

Will this also work for things like??

web:
  environment:
    - FOO=${whoami}
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah May 24, 2015

Member

@k0377 I don't think they will, because that's really something that's handled by the shell, but you could add the result in an environment variable and use that.

In this particular case, the $USER environment variable will probably give you the same.

Member

thaJeztah commented May 24, 2015

@k0377 I don't think they will, because that's really something that's handled by the shell, but you could add the result in an environment variable and use that.

In this particular case, the $USER environment variable will probably give you the same.

@askoretskiy

This comment has been minimized.

Show comment
Hide comment
@askoretskiy

askoretskiy May 26, 2015

@aanand Why not use any of existing template engines that are already present? Jinja2 is there and works fine.

As mentioned before -- implementing our own templating is non-trivial task (and regexps are not that cool) so that we should use already existing one, that proven to be solid.

askoretskiy commented May 26, 2015

@aanand Why not use any of existing template engines that are already present? Jinja2 is there and works fine.

As mentioned before -- implementing our own templating is non-trivial task (and regexps are not that cool) so that we should use already existing one, that proven to be solid.

@askoretskiy

This comment has been minimized.

Show comment
Hide comment
@askoretskiy

askoretskiy May 26, 2015

Alternatively we might use YAML ancors and references https://gist.github.com/bowsersenior/979804

But then we are limited on variables usage (inject variable name into middle of content).

askoretskiy commented May 26, 2015

Alternatively we might use YAML ancors and references https://gist.github.com/bowsersenior/979804

But then we are limited on variables usage (inject variable name into middle of content).

@DoubleMalt

This comment has been minimized.

Show comment
Hide comment
@DoubleMalt

DoubleMalt May 26, 2015

Contributor

+1 for Jinja2: it would certainly fit the mold and ansible uses it for
exactly that use case (templating in yml files)

On Tue, May 26, 2015 at 1:25 PM, tonnzor notifications@github.com wrote:

@aanand https://github.com/aanand Why not use any of existing template
engines that are already present? Jinja2 is there and works fine.

As mentioned before -- implementing our own templating is non-trivial task
(and regexps are not that cool) so that we should use already existing one,
that proven to be solid.


Reply to this email directly or view it on GitHub
#1377 (comment).

Contributor

DoubleMalt commented May 26, 2015

+1 for Jinja2: it would certainly fit the mold and ansible uses it for
exactly that use case (templating in yml files)

On Tue, May 26, 2015 at 1:25 PM, tonnzor notifications@github.com wrote:

@aanand https://github.com/aanand Why not use any of existing template
engines that are already present? Jinja2 is there and works fine.

As mentioned before -- implementing our own templating is non-trivial task
(and regexps are not that cool) so that we should use already existing one,
that proven to be solid.


Reply to this email directly or view it on GitHub
#1377 (comment).

@aanand

This comment has been minimized.

Show comment
Hide comment
@aanand

aanand May 26, 2015

Contributor

Jinja2 does a lot more than we need:

  • conditionals
  • looping
  • extension/inheritance
  • comments
  • filters

We're not adding any of that stuff to Compose. If Jinja2 can be configured to just interpolate variables, then it might be a candidate.

Contributor

aanand commented May 26, 2015

Jinja2 does a lot more than we need:

  • conditionals
  • looping
  • extension/inheritance
  • comments
  • filters

We're not adding any of that stuff to Compose. If Jinja2 can be configured to just interpolate variables, then it might be a candidate.

@DoubleMalt

This comment has been minimized.

Show comment
Hide comment
@DoubleMalt

DoubleMalt May 26, 2015

Contributor

Actually looping might be interesting.

Assume you have a list of customers that you want to start containers for
where you put some customer specific variables into the environment.

Extension/Inheritance might be interesting to enhance the current
rudimentary extension mechanism.

Filters can be great to do something with existing variables.

On Tue, May 26, 2015 at 1:56 PM, Aanand Prasad notifications@github.com
wrote:

Jinja2 does a lot more than we need:

  • conditionals
  • looping
  • extension/inheritance
  • comments
  • filters

We're not adding any of that stuff to Compose. If Jinja2 can be configured
to just interpolate variables, then it might be a candidate.


Reply to this email directly or view it on GitHub
#1377 (comment).

Contributor

DoubleMalt commented May 26, 2015

Actually looping might be interesting.

Assume you have a list of customers that you want to start containers for
where you put some customer specific variables into the environment.

Extension/Inheritance might be interesting to enhance the current
rudimentary extension mechanism.

Filters can be great to do something with existing variables.

On Tue, May 26, 2015 at 1:56 PM, Aanand Prasad notifications@github.com
wrote:

Jinja2 does a lot more than we need:

  • conditionals
  • looping
  • extension/inheritance
  • comments
  • filters

We're not adding any of that stuff to Compose. If Jinja2 can be configured
to just interpolate variables, then it might be a candidate.


Reply to this email directly or view it on GitHub
#1377 (comment).

@aanand

This comment has been minimized.

Show comment
Hide comment
@aanand

aanand May 26, 2015

Contributor

They might be interesting features, but they come with far more complexity than I'm comfortable introducing to both Compose and the file format, and we'd be tying both to a specific templating language with (as far as I know) a single implementation and no spec. It's simply not feasible.

Contributor

aanand commented May 26, 2015

They might be interesting features, but they come with far more complexity than I'm comfortable introducing to both Compose and the file format, and we'd be tying both to a specific templating language with (as far as I know) a single implementation and no spec. It's simply not feasible.

@askoretskiy

This comment has been minimized.

Show comment
Hide comment
@askoretskiy

askoretskiy May 26, 2015

@aanand Some notes here:

  1. Jinja2 is solid and it takes minutes to do pre-processing of YAML with it:

from jinja2 import Template
template = Template('Hello {{ name }}!')
template.render(name="Aanand")
Hello Aanand!

If you want more security -- you could use immutable sandbox:

from jinja2.sandbox import ImmutableSandboxedEnvironment
env = ImmutableSandboxedEnvironment()
template = env.from_string('Hello {{ name }}!')
template.render(name="Aanand")
Hello Aanand!

In our case it would be:

import os
from jinja2.sandbox import ImmutableSandboxedEnvironment
env = ImmutableSandboxedEnvironment()
template = env.from_string('Hello {{ name }}!')
template.render(**os.environ)

  1. Don't we want filters? With filter you could define default value easily ( {{ value|default("default") }} )
  2. Do we really need to care about users that use extended Jinja features to screw YAML file? In the same manner user could produce invalid YAML file manually. I think we should keep it simple -- try to process given Jinja template and return error if there was an error or produced YAML is invalid (the same as you do now).
  3. If you don't see Jinja2 as solution -- it would be great at least to use {{ variable }} as syntax.
  4. Django uses regexp to parse and generate template. It is production-grade for a long time and lives fine with it.

import os
import re
template = "Hello {{ name }}!"
re.sub("{{\s_([a-zA-Z0-9_]+?)\s_}}", lambda m: os.environ.get(m.group(1), ''), template)

In any case -- we need to have this feature rolling, whatever solution we take.

askoretskiy commented May 26, 2015

@aanand Some notes here:

  1. Jinja2 is solid and it takes minutes to do pre-processing of YAML with it:

from jinja2 import Template
template = Template('Hello {{ name }}!')
template.render(name="Aanand")
Hello Aanand!

If you want more security -- you could use immutable sandbox:

from jinja2.sandbox import ImmutableSandboxedEnvironment
env = ImmutableSandboxedEnvironment()
template = env.from_string('Hello {{ name }}!')
template.render(name="Aanand")
Hello Aanand!

In our case it would be:

import os
from jinja2.sandbox import ImmutableSandboxedEnvironment
env = ImmutableSandboxedEnvironment()
template = env.from_string('Hello {{ name }}!')
template.render(**os.environ)

  1. Don't we want filters? With filter you could define default value easily ( {{ value|default("default") }} )
  2. Do we really need to care about users that use extended Jinja features to screw YAML file? In the same manner user could produce invalid YAML file manually. I think we should keep it simple -- try to process given Jinja template and return error if there was an error or produced YAML is invalid (the same as you do now).
  3. If you don't see Jinja2 as solution -- it would be great at least to use {{ variable }} as syntax.
  4. Django uses regexp to parse and generate template. It is production-grade for a long time and lives fine with it.

import os
import re
template = "Hello {{ name }}!"
re.sub("{{\s_([a-zA-Z0-9_]+?)\s_}}", lambda m: os.environ.get(m.group(1), ''), template)

In any case -- we need to have this feature rolling, whatever solution we take.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah May 26, 2015

Member

I'm +1 on using a generic templating solution if templates are considered. E.g. http://mustache.github.io, which is available in many languages. This is just an example, other templating engines can be considered equally

Member

thaJeztah commented May 26, 2015

I'm +1 on using a generic templating solution if templates are considered. E.g. http://mustache.github.io, which is available in many languages. This is just an example, other templating engines can be considered equally

@DoubleMalt

This comment has been minimized.

Show comment
Hide comment
@DoubleMalt

DoubleMalt May 26, 2015

Contributor

@aanand I totally understand your point. I also like the simplicity and
succinctness of the compose dsl.

Maybe this should be done as an external project, say meta-composer. It
takes a compose.tpl.yml and a variables.yml, creates a docker-compose.yml
and off we go.
As @tonnzor showed t could be done with a tiny piece of python code.

This would deliver powerful templating to those who need it without
introducing complexity for simple tasks.

On Tue, May 26, 2015 at 4:52 PM, Sebastiaan van Stijn <
notifications@github.com> wrote:

I'm +1 on using a generic templating solution if templates are
considered. E.g. http://mustache.github.io, which is available in many
languages. This is just an example, other templating engines can be
considered equally


Reply to this email directly or view it on GitHub
#1377 (comment).

Contributor

DoubleMalt commented May 26, 2015

@aanand I totally understand your point. I also like the simplicity and
succinctness of the compose dsl.

Maybe this should be done as an external project, say meta-composer. It
takes a compose.tpl.yml and a variables.yml, creates a docker-compose.yml
and off we go.
As @tonnzor showed t could be done with a tiny piece of python code.

This would deliver powerful templating to those who need it without
introducing complexity for simple tasks.

On Tue, May 26, 2015 at 4:52 PM, Sebastiaan van Stijn <
notifications@github.com> wrote:

I'm +1 on using a generic templating solution if templates are
considered. E.g. http://mustache.github.io, which is available in many
languages. This is just an example, other templating engines can be
considered equally


Reply to this email directly or view it on GitHub
#1377 (comment).

@kojiromike

This comment has been minimized.

Show comment
Hide comment
@kojiromike

kojiromike May 26, 2015

Contributor

Hmm… So is the proposal now to use a templating language inside compose.yml (which is a descriptive language for composing Docker containers), for things like command and entrypoint, which already accept both exec and sh -c style values? This could be confusing, since after template rendering the resultant shell command would still presumably be interpreted, so if a variable happened to expand to * it would further be glob expanded. Escaping sequences in one language or another becomes tricky when you have this many layers of fall-through interpretation.

Contributor

kojiromike commented May 26, 2015

Hmm… So is the proposal now to use a templating language inside compose.yml (which is a descriptive language for composing Docker containers), for things like command and entrypoint, which already accept both exec and sh -c style values? This could be confusing, since after template rendering the resultant shell command would still presumably be interpreted, so if a variable happened to expand to * it would further be glob expanded. Escaping sequences in one language or another becomes tricky when you have this many layers of fall-through interpretation.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah May 26, 2015

Member

@kojiromike I'm not sure a templating engine is wanted, but if it's to be used! better use something well known. The basic question is; should docker-compose write the substitution from-scratch, or use something existing.

Member

thaJeztah commented May 26, 2015

@kojiromike I'm not sure a templating engine is wanted, but if it's to be used! better use something well known. The basic question is; should docker-compose write the substitution from-scratch, or use something existing.

@nafg

This comment has been minimized.

Show comment
Hide comment
@nafg

nafg May 26, 2015

On Tue, May 26, 2015, 11:02 AM Christoph Witzany notifications@github.com
wrote:

@aanand I totally understand your point. I also like the simplicity and
succinctness of the compose dsl.

Maybe this should be done as an external project, say meta-composer. It
takes a compose.tpl.yml and a variables.yml, creates a docker-compose.yml
and off we go.

You can do that today without any new project. I'm sure jinja can be
invoked from the command line somehow. Personally I just use the envsubst
command.

What would be really helpful is if compose could read the file from stdin.
That would eliminate the need for an intermediate file.

As @tonnzor showed t could be done with a tiny piece of python code.

This would deliver powerful templating to those who need it without
introducing complexity for simple tasks.

On Tue, May 26, 2015 at 4:52 PM, Sebastiaan van Stijn <
notifications@github.com> wrote:

I'm +1 on using a generic templating solution if templates are
considered. E.g. http://mustache.github.io, which is available in many
languages. This is just an example, other templating engines can be
considered equally

Reply to this email directly or view it on GitHub
#1377 (comment).

Reply to this email directly or view it on GitHub
#1377 (comment).

nafg commented May 26, 2015

On Tue, May 26, 2015, 11:02 AM Christoph Witzany notifications@github.com
wrote:

@aanand I totally understand your point. I also like the simplicity and
succinctness of the compose dsl.

Maybe this should be done as an external project, say meta-composer. It
takes a compose.tpl.yml and a variables.yml, creates a docker-compose.yml
and off we go.

You can do that today without any new project. I'm sure jinja can be
invoked from the command line somehow. Personally I just use the envsubst
command.

What would be really helpful is if compose could read the file from stdin.
That would eliminate the need for an intermediate file.

As @tonnzor showed t could be done with a tiny piece of python code.

This would deliver powerful templating to those who need it without
introducing complexity for simple tasks.

On Tue, May 26, 2015 at 4:52 PM, Sebastiaan van Stijn <
notifications@github.com> wrote:

I'm +1 on using a generic templating solution if templates are
considered. E.g. http://mustache.github.io, which is available in many
languages. This is just an example, other templating engines can be
considered equally

Reply to this email directly or view it on GitHub
#1377 (comment).

Reply to this email directly or view it on GitHub
#1377 (comment).

metal3d added a commit to metal3d/compose that referenced this issue Apr 19, 2016

Add command interpolation
Use $((cmd)) to be able to set a value from a shell command.

See #1377
@metal3d

This comment has been minimized.

Show comment
Hide comment
@metal3d

metal3d Apr 19, 2016

Contributor

This pull-request is a simple example that works for me. Maybe you can help me to do better.

I tried with environment and user directives from docker-compose.yml file. Works well for now.

Contributor

metal3d commented Apr 19, 2016

This pull-request is a simple example that works for me. Maybe you can help me to do better.

I tried with environment and user directives from docker-compose.yml file. Works well for now.

metal3d added a commit to metal3d/compose that referenced this issue Apr 19, 2016

Add command interpolation
Use $((cmd)) to be able to set a value from a shell command.

See #1377

Signed-off-by: Patrice Ferlet <metal3d@gmail.com>

metal3d added a commit to metal3d/compose that referenced this issue Apr 19, 2016

Add command interpolation
Use $((cmd)) to be able to set a value from a shell command.

See #1377

Signed-off-by: Patrice Ferlet <metal3d@gmail.com>

metal3d added a commit to metal3d/compose that referenced this issue Apr 19, 2016

Add command interpolation
Use $((cmd)) to be able to set a value from a shell command.

See #1377

Signed-off-by: Patrice Ferlet <metal3d@gmail.com>

metal3d added a commit to metal3d/compose that referenced this issue Apr 19, 2016

Add command interpolation
Use $((cmd)) to be able to set a value from a shell command.

See #1377

Signed-off-by: Patrice Ferlet <metal3d@gmail.com>

metal3d added a commit to metal3d/compose that referenced this issue Apr 19, 2016

Add command interpolation
Use $((cmd)) to be able to set a value from a shell command.

See #1377

Signed-off-by: Patrice Ferlet <metal3d@gmail.com>

metal3d added a commit to metal3d/compose that referenced this issue Apr 19, 2016

Add command interpolation
Use $((cmd)) to be able to set a value from a shell command.

See #1377

Signed-off-by: Patrice Ferlet <metal3d@gmail.com>

metal3d added a commit to metal3d/compose that referenced this issue Apr 19, 2016

Add command interpolation
Use $((cmd)) to be able to set a value from a shell command.

See #1377

Note: Seems to be posix specific now, Windows test break with a simple
"echo" command.

Signed-off-by: Patrice Ferlet <metal3d@gmail.com>

metal3d added a commit to metal3d/compose that referenced this issue Apr 20, 2016

Add command interpolation
Use $((cmd)) to be able to set a value from a shell command.

See #1377

Note: Seems to be posix specific now, Windows test break with a simple
"echo" command.

Signed-off-by: Patrice Ferlet <metal3d@gmail.com>

metal3d added a commit to metal3d/compose that referenced this issue Apr 20, 2016

Add command interpolation
Use $((cmd)) to be able to set a value from a shell command.

See #1377

Note: Seems to be posix specific now, Windows test break with a simple
"echo" command.

Signed-off-by: Patrice Ferlet <metal3d@gmail.com>
@marcellodesales

This comment has been minimized.

Show comment
Hide comment
@marcellodesales

marcellodesales May 3, 2016

The default value should be there... Very useful... Developers and OPS are using either the docker logs or syslog... So, They usually have to create the default LOG_FORMAT shown below... We could just have the default value and only use it when switching to syslog...

default:
  extends:
    file: base.yml
    service: base-${LOG_FORMAT:docker}
  labels:
    - "net.company.npmjs.datacenter=${DATA_CENTER}"
    - "net.company.npmjs.env=${ENV}"
    - "net.company.npmjs.hostname=${HOSTNAME}"
    - "net.company.npmjs.role=${NPMO_ROLE}"
    - "net.company.npmjs.log=${LOG_FORMAT}"

base-syslog:
  log_driver: syslog
  log_opt:
    tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"

base-docker:
  log_driver: json-file
  log_opt:
    max-size: "128m"
    max-file: "4"

When will this be available? I'm on Compose 1.7.0 and this is still not there :(

marcellodesales commented May 3, 2016

The default value should be there... Very useful... Developers and OPS are using either the docker logs or syslog... So, They usually have to create the default LOG_FORMAT shown below... We could just have the default value and only use it when switching to syslog...

default:
  extends:
    file: base.yml
    service: base-${LOG_FORMAT:docker}
  labels:
    - "net.company.npmjs.datacenter=${DATA_CENTER}"
    - "net.company.npmjs.env=${ENV}"
    - "net.company.npmjs.hostname=${HOSTNAME}"
    - "net.company.npmjs.role=${NPMO_ROLE}"
    - "net.company.npmjs.log=${LOG_FORMAT}"

base-syslog:
  log_driver: syslog
  log_opt:
    tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"

base-docker:
  log_driver: json-file
  log_opt:
    max-size: "128m"
    max-file: "4"

When will this be available? I'm on Compose 1.7.0 and this is still not there :(

@nottrobin

This comment has been minimized.

Show comment
Hide comment
@nottrobin

nottrobin May 4, 2016

Please please please give us default values!

nottrobin commented May 4, 2016

Please please please give us default values!

@pataquets

This comment has been minimized.

Show comment
Hide comment
@pataquets

pataquets May 4, 2016

@marcellodesales: Perhaps you can take advantage of using a docker-compose.override.yml file in some way. Check that feature.

Also +1 on env vars. It is our major pain point with docker-compose nowadays.

pataquets commented May 4, 2016

@marcellodesales: Perhaps you can take advantage of using a docker-compose.override.yml file in some way. Check that feature.

Also +1 on env vars. It is our major pain point with docker-compose nowadays.

@metal3d

This comment has been minimized.

Show comment
Hide comment
@metal3d

metal3d May 4, 2016

Contributor

I would insist on my PR #3367 to be able to get certain values from host. :)

Contributor

metal3d commented May 4, 2016

I would insist on my PR #3367 to be able to get certain values from host. :)

@marcellodesales

This comment has been minimized.

Show comment
Hide comment
@marcellodesales

marcellodesales May 5, 2016

@pataquets I don' think I want to create yet other override file... our base.yml file, as shown above, shows all the supported stuff in terms of logger driver, etc... I just want to switch and have default values. I would think that we would need to maintain yet more yml files. But I will keep that in mind just in case.

marcellodesales commented May 5, 2016

@pataquets I don' think I want to create yet other override file... our base.yml file, as shown above, shows all the supported stuff in terms of logger driver, etc... I just want to switch and have default values. I would think that we would need to maintain yet more yml files. But I will keep that in mind just in case.

@zerthimon

This comment has been minimized.

Show comment
Hide comment
@zerthimon

zerthimon commented Jul 13, 2016

+1

@trippel

This comment has been minimized.

Show comment
Hide comment
@trippel

trippel commented Sep 25, 2016

+1

@gbmeuk

This comment has been minimized.

Show comment
Hide comment
@gbmeuk

gbmeuk commented Sep 27, 2016

+1

@panga

This comment has been minimized.

Show comment
Hide comment
@panga

panga commented Oct 6, 2016

+1

@Modomu

This comment has been minimized.

Show comment
Hide comment
@Modomu

Modomu Oct 21, 2016

Any notice about use environment variables in docker-compose?

+1

Modomu commented Oct 21, 2016

Any notice about use environment variables in docker-compose?

+1

@ChrisCooney

This comment has been minimized.

Show comment
Hide comment

ChrisCooney commented Oct 22, 2016

+1

@darkn3rd

This comment has been minimized.

Show comment
Hide comment
@darkn3rd

darkn3rd Nov 7, 2016

FYI: Environment vars for docker-compose works as of 1.7.0. You can also set docker-compose default variables in .env in the same directory as your root docker-compose.yml file. This is not to be confused with docker engine envfiles, as that's a different thing.

darkn3rd commented Nov 7, 2016

FYI: Environment vars for docker-compose works as of 1.7.0. You can also set docker-compose default variables in .env in the same directory as your root docker-compose.yml file. This is not to be confused with docker engine envfiles, as that's a different thing.

@LouWii

This comment has been minimized.

Show comment
Hide comment
@LouWii

LouWii Dec 18, 2016

Is there a way to set the service name as a variable ?

Instead of writing this

services:
   site_db:
     image: mysql:5.7

We could write

services:
   ${CONTAINER_NAME}:
     image: mysql:5.7

My goal is to keep the same docker-compose.yml across multiple sites, and only change the .env file. Right now, I still need to modify the container name because I'm running several apps on the same host. And I'd like to have each service to have its own name for clarity.

LouWii commented Dec 18, 2016

Is there a way to set the service name as a variable ?

Instead of writing this

services:
   site_db:
     image: mysql:5.7

We could write

services:
   ${CONTAINER_NAME}:
     image: mysql:5.7

My goal is to keep the same docker-compose.yml across multiple sites, and only change the .env file. Right now, I still need to modify the container name because I'm running several apps on the same host. And I'd like to have each service to have its own name for clarity.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 18, 2016

Member

@LouWii you can use

services:
    site_db:
      container_name: "${CONTAINER_NAME}"
      image: mysql:5.7

or (compose-file format 2.1 and up)

services:
    site_db:
      container_name: "${CONTAINER_NAME:-defaultname}"
      image: mysql:5.7

But why not set the project-name? The project name is intended for that, as it prefixes/namespaces the container-names that are created to prevent conflicting with other projects on the same host. See https://docs.docker.com/compose/reference/envvars/#/composeprojectname

Member

thaJeztah commented Dec 18, 2016

@LouWii you can use

services:
    site_db:
      container_name: "${CONTAINER_NAME}"
      image: mysql:5.7

or (compose-file format 2.1 and up)

services:
    site_db:
      container_name: "${CONTAINER_NAME:-defaultname}"
      image: mysql:5.7

But why not set the project-name? The project name is intended for that, as it prefixes/namespaces the container-names that are created to prevent conflicting with other projects on the same host. See https://docs.docker.com/compose/reference/envvars/#/composeprojectname

@LouWii

This comment has been minimized.

Show comment
Hide comment
@LouWii

LouWii Dec 19, 2016

@thaJeztah Thanks ! I'm still learning how Docker and docker compose work. Setting the project name seems to be a better option, it makes total sense in my use.

LouWii commented Dec 19, 2016

@thaJeztah Thanks ! I'm still learning how Docker and docker compose work. Setting the project name seems to be a better option, it makes total sense in my use.

@SamHH

This comment has been minimized.

Show comment
Hide comment
@SamHH

SamHH Jan 24, 2017

Is there any way to script within the interpolated braces? For example ${HOST_PORT + 1}.

SamHH commented Jan 24, 2017

Is there any way to script within the interpolated braces? For example ${HOST_PORT + 1}.

@nafg

This comment has been minimized.

Show comment
Hide comment
@nafg

nafg Feb 1, 2017

nafg commented Feb 1, 2017

@logicminds

This comment has been minimized.

Show comment
Hide comment
@logicminds

logicminds Mar 31, 2017

Am I able to escape $?

environment:
   PATH: "$PATH:/home/appuser/.bundler/bin"

Currently this results in the host's PATH variable being interpolated and not the container

logicminds commented Mar 31, 2017

Am I able to escape $?

environment:
   PATH: "$PATH:/home/appuser/.bundler/bin"

Currently this results in the host's PATH variable being interpolated and not the container

@alaagalal

This comment has been minimized.

Show comment
Hide comment
@alaagalal

alaagalal Apr 17, 2017

is a docker-compose.yml file only one found ?
how can i add or modify it?
Thanks in advance

alaagalal commented Apr 17, 2017

is a docker-compose.yml file only one found ?
how can i add or modify it?
Thanks in advance

@samrocketman

This comment has been minimized.

Show comment
Hide comment
@samrocketman

samrocketman Feb 1, 2018

@logicminds though I couldn't find it documented anywhere, I found $$ interpolates to an escaped $.

environment:
   PATH: "$$PATH:/home/appuser/.bundler/bin"

samrocketman commented Feb 1, 2018

@logicminds though I couldn't find it documented anywhere, I found $$ interpolates to an escaped $.

environment:
   PATH: "$$PATH:/home/appuser/.bundler/bin"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment