Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Variable substitution #2380

gponsu opened this issue Nov 12, 2015 · 35 comments

Variable substitution #2380

gponsu opened this issue Nov 12, 2015 · 35 comments


Copy link

@gponsu gponsu commented Nov 12, 2015

This new feature is amazing, but I had a problem. I explain:

In our dev team, to enter in docker with a user with the same UID and GID that of the host machine, we made the following:

# docker-compose.yml
  image: rails:4.2.4
  working_dir: /app
    file: docker-custom.yml
    service: app

The docker-custom.yml file we have untracked of git, so everyone can define their own settings.

# docker-custom.yml
  # Set your user and group id.
  # View your user id with command 'id -u' and your group id with 'id -g'.
  user: '1000:1000'
    BUNDLE_PATH: /app/.bundle
    BUNDLE_APP_CONFIG: /app/.bundle
  # You can use env_file option instead of environment if you are using one file
  # to save environment variables
  # env_file: .env

With the new functionality of Variable substitution, I was excited, because we could spare us all this, and minimize it to this line in docker-compose.yml:

user: "${UID}:${GID}"

Unfortunately this does not work, because UID and GID are shell variables (not env variables), and os.environ do not get them.

Would it be possible get the shell envs too? Something like (I know nothing about python 😞):


He had also thought of another possibility, if the option user is user: host or similar, internally get the uid and gid with:


And setup docker with this config for user.

Thank you very much for everything, and apologies for my horrible English.

Copy link

@oinak oinak commented Nov 12, 2015


I have the exact same problem, as soon as I saw variable substitutio I tried for $UID to no avail

I could make the file ownership issues of containers mounting code folders go away.

Please please please enable shellvars along with envvars to be sunstituted on docker-compose.yml files.

Thank you for the excellent work.

Copy link

@dnephin dnephin commented Nov 12, 2015

The only different is that UID isn't exported. I don't think it's correct to go reading un-expected shell variables.

I would suggest export UID.

Copy link

@deeky666 deeky666 commented Dec 15, 2015

👍 I think this is a common use-case when mouting directories from the host and accessing them inside the container. Having to export extra environment variables on the host side is unnecessary extra configuration on the host side which more or less defeats this whole purpose.
Until using docker composer, I was evaluating the user and group id via id -u and id -g in the run command, which is not (yet) possible in docker composer configuration.
Some sort of solution to this would be very welcome.

Copy link

@cookandy cookandy commented Jan 29, 2016


Another use case would be trying to set the hostname of a container to the hostname of your host:

  hostname: ${HOSTNAME}

My workaround right now is to set it before calling docker-compose

HOSTNAME=$HOSTNAME docker-compose up

Copy link

@dvddarias dvddarias commented Apr 19, 2016

I have exactly the same problem. In our team all you need to do to setup a new developer is
to run:

git clone ....
cd ....   
export UID   
docker-compose up   

We need the same user on the container and the host because of the big amount of Ruby on Rails commands that generate new files on the project volume. The sad part is that you have to export the UID variable on every new shell session for docker-compose up to work or add the variable to your shell startup script, which is sub-optimal on such a common use case.

This problem is specific to programmers using docker-compose as a tool:

Using docker CLI directly you don't have this problem because it is assumed that you run them within a shell and there you can use the variable $UID without exporting it.

Using docker through the API is also possible getting the current user id with os.geteuid() or something similar.

When using docker-compose i haven't find the way of doing it without running commands, so the "same user as the host" service requirement can't be made effective on any of the repository files.

A solution may be to setup some specific environment variable (let's say COMPOSE_UID) to the same UID as the user running the docker-compose command with os.geteuid() and then use it within the file as ${COMPOSE_UID}.

Copy link

@mknapik mknapik commented Oct 4, 2016

Same use case, handling shared directories at CI server.

Copy link

@edannenberg edannenberg commented Mar 30, 2017

Can't spam enough 👍 for this. UID/GID of the current host process should be provided by docker-compose in some way.

Copy link

@ScreamingDev ScreamingDev commented Apr 11, 2017

So as my #4725 got closed due to duplicates I need to ask the remaining question here:

How to solve/bypass this?

I guess it can't be left that way it is as long as there is no alternative.
It really is something that makes work a lot easier for developers.
Doesn't matter on production server but for dev and test server it helps.

Does the world build a wrapper around docker-compose having export UID in it?
Is it feasible by docker compose to solve this?

Copy link

@durkode durkode commented Aug 3, 2017

I'm also going to add my voice to this. Developing with Django from within a container and would love for this to work out of the box without having to export UID and GID.

I personally am not concerned how it is done, but a way for files written to the mounted volume in the container to match the host user without having to export UID and GID is my end aim. Currently running the export isn't a big deal, it would just be nicer if this wasn't the case

Copy link

@sokratisg sokratisg commented Aug 3, 2017

To be honest we sort of worked around this issue by doing something similar to:

  1. Make sure UID is exported in each user's bash shell. Just create the following file and restart:
export UID
  1. Add the necessary arguments in the docker-compose.yml file which is used for local development purposes:
      context: .
        username: $USER
        userid: $UID
  1. Use these arguments in the respective Dockerfile which is also used solely for local development:
ARG username
ARG userid
RUN useradd --no-create-home --user-group --shell /bin/bash --home-dir $APP_HOME --uid $userid $username
RUN chown -R $username: $APP_HOME

Only drawback: UID conflicts with images that by default use the same one with any local user. (i.e. official node images have node user with UID=1000 and that conflicts with the initial UID a typical Ubuntu desktop assigns to its first user, tha one you're going to use post-installation)

Copy link

@edannenberg edannenberg commented Aug 3, 2017

Only drawback: UID conflicts with images that by default use the same one with any local user. (i.e. official node images have node user with UID=1000 and that conflicts with the initial UID a typical Ubuntu desktop assigns to its first user, tha one you're going to use post-installation)

Just use a static user like node in the image and change it's UID on container start as needed. The username is irrelevant when it comes to permissions, all that matters is a matching UID.

Copy link

@sokratisg sokratisg commented Aug 3, 2017

That is true if you use an image from a Dockerfile. Not true if you just want to start a node app by pulling the default image without any fiddling around or any custom shell script as your command.

Copy link

@edannenberg edannenberg commented Aug 3, 2017

@sokratisg You seem to use a local Dockerfile already, replacing useradd with usermod --uid $userid nodeshould do the trick.

Copy link

@sokratisg sokratisg commented Aug 3, 2017

@edannenberg excuse me but I think I may have been misunderstood, maybe my comment wasn't very clear. I'll try to give you an example.

Let's say your desktop user has UID=1000 (default for first user, at least on Ubuntu) and that you've written a node app that outputs or generates some local files. You want them to be written with the correct ownership (UID=1000) since you're also working locally and want to have read/write access on them without using sudo for your favorite editor/IDE. If you go with the solution I described then upon startup of the node container you will end up with a UiD conflict since official node images come pre-baked with a node user that has the same UID (1000).

This is what I was trying to describe; that this workaround is not universal and comes with some possibilities of a conflict so this issue is still relevant.

Copy link

@edannenberg edannenberg commented Aug 4, 2017

@sokratisg I got that, the problem is that your solution tries to add another user with an already existing UID. My point was that you can avoid that by just changing the UID of an existing static user in the image, i.e. node. In theory this will still leave the possibility of a conflict, but in practice the image will most likely only have one user with a UID that could conflict with the typical user UID space (>= 1000, everything below 1000 is usually reserved for system users).

A typical nodejs compose file in our projects looks like this:

version: '2.2'
      dockerfile: ${PWD}/docker/Dockerfile
      context: ./docker
        - NODEJS_UID=${UID}
        - NODEJS_GID=${GID}
    user: nodejs
    working_dir: ${PWD}
      - ~/.npm:/home/nodejs/.npm
      - ${PWD}:${PWD}

The nodejs user already exists in the pulled image, the local Dockerfile just changes the UID/GID of said user.

Copy link

@vipseixas vipseixas commented Oct 11, 2017

My workaround for the usecase of ownership of mounted volumes is to use a script as the entrypoint of the image, like this:



files=(/work/source/*) && DIR=${files[-1]}
USER_ID=$(stat -c "%u" ${DIR})

echo "Using ${USER_ID} from owner of dir ${DIR}"

id "${USER_NAME?}" >/dev/null 2>&1

if [[ $? -ne 0 ]]; then
	echo Creating user ${USER_NAME} with UID ${USER_ID}
	useradd --home-dir /work --uid ${USER_ID} --shell /bin/bash --no-create-home ${USER_NAME}

su --login --preserve-environment --command "/scripts/ $1 $2 $3" ${USER_NAME}

/work/source is where I mount my external source code inside the container.

Copy link

@38leinaD 38leinaD commented Jan 4, 2018


Copy link

@ChristianKreuzberger ChristianKreuzberger commented Apr 17, 2018

Django Developer here:
Using $UID and $GID within docker-compose.yml would be so great! Please please please!

Copy link

@CodeCorrupt CodeCorrupt commented May 27, 2018

One better and more versatile would be the ability to include shell script in the docker-compose. Something like this would be helpful

      - UID=$(id -u)
      - GID=$(id -g)

I'm currently accomplishing this by wrapping the docker-compose up in a shell script that sets those variables, however that's a little clunky. This could also be useful for dynamically setting other fields outside of just environment variables.

Edit: A lot of the comments are saying to just use $UID and $GID. I don't know about other distros but my Ubuntu Server 16.4 doesn't have that by default so adding the following to your .bashrc can add that and has been working well for me.

UID=$(id -u)
GID=$(id -g)

Copy link

@reduardo7 reduardo7 commented May 28, 2018

I use something like this:

version: "2.1"

      - /etc/passwd:/etc/passwd:ro
      - /etc/group:/etc/group:ro
    user: $UID:$GID

Copy link

@awilkins awilkins commented Jun 1, 2018

Using a .env file ; sadly, you can't use substitutions in a .env file, it's all literals.

Copy link

@rcdailey rcdailey commented Oct 10, 2018

Maybe I've got the wrong idea here, but why do folks use the user and group ID of the current user when they run their services? So far, I've personally been creating system users in Ubuntu for this, one per service. So for my plex container, I have a user called plex and a shared group called docker_services. My next service, sonarr, will have a user called sonarr and also use the group docker_services. This allows me to assign very specific permissions to bind mount directories, and prevent other services from touching files they aren't supposed to.

I have a shell script I run right now to launch my services with users in this way, but unfortunately since I can't run subshells inside the yml file, I haven't been able to use docker compose:

docker run -d \
    --name plex \
    --restart unless-stopped \
    --network host \
    -e PLEX_UID="$(id -u plex)" \
    -e PLEX_GID="$(id -g plex)" \

Is this not the recommended way to run permissions for containers? I'd rather not start them under my personal account I use to SSH into the machine itself, since this user account was not meant to be used by services running 24/7.

Assuming what I'm doing isn't completely bonkers, then I think there's still some ways to go before I use docker compose. Right now I use a series of bash scripts to start my services, semantically similar to docker-compose up. This is the only way I've been able to have the flexibility to do some system introspection while running docker containers.

Copy link

@javabrett javabrett commented Oct 10, 2018

Maybe I've got the wrong idea here, but why do folks use the user and group ID of the current user when they run their services?

Largely in a development context, in cases where it is desirable to use bind-mounts, and you can't, or it's not easy or practical to prevent the container from creating files. If such files are created with users/groups that don't match the host-container, they become painful to clean-up in development-cycles.

Copy link

@Wirone Wirone commented Dec 12, 2018

@gponsu We're using environment variable which is documented in our

Add export DOCKER_COMPOSE_RUN_AS_USER="$(id -u):$(id -g)" to your ~/.bashrc or other place that will define it every time when starting terminal.

and then in docker-compose.yml:

user: "${DOCKER_COMPOSE_RUN_AS_USER:?You must define user:group for permissions handling, look at README}"

The :? part enforces that variable, so if it's not defined, build will fail.

But yeah, I came here because I was looking for executing Bash script in docker-compose.yml because I wanted to pass git describe result as ENV variable.

Copy link

@lygstate lygstate commented Mar 3, 2019

looking for better solution

Copy link

@ivanbogomoloff ivanbogomoloff commented Apr 10, 2019

Working for me.


and run docker-compose with export like this

export APP_USER_UID=$UID && docker-compose up -d

Copy link

@MtDalPizzol MtDalPizzol commented May 18, 2019

Just ran into a scenario where I needed to pass the UID an GID to compose. I'll just live what I did here so maybe it helps someone out there. I'm running Mint (Ubuntu based distro) and here is the step by step that got this working for me:

  1. Edit ~/.bashrc file adding the following to the end of it:
export COMPOSE_UID=$(id -u)
export COMPOSE_GID=$(id -g)
  1. Add the environment vars to docker-compose.yml
version: "3"
    image: richarvey/nginx-php-fpm:latest
      - .:/var/www/html
      WEBROOT: /var/www/html/www
      - 8081:80
  1. Restart the terminal or start a fresh one.

  2. Now you can run compose without any additional magic:

$ docker-compose up

Copy link

@egdoc egdoc commented Jul 15, 2019

When using sudo to launch docker-compose, one can use the SUDO_UID and SUDO_GID environment variables which contain respectively the UID and GID of the user who launched the command:

version: '3'
            context: ./php-apache
                APACHE_UID: "${SUDO_UID}"
            -  8080:80
            -   ./DocumentRoot:/var/www/html:z

Copy link

@slykar slykar commented Jul 23, 2019

Use an .env file

I often have a sample .env file for my projects.
You can set required variables there.

# .env

docker-compose command picks this file automatically and makes the declared env vars available in the YAML file.

Ask users to create an override file and set user: manually.

# docker-compose.override.yaml
user: 1000:1000

Copy link

@rvdbogerd rvdbogerd commented Nov 12, 2019

After hours of searching, @slykar came with the missing piece. Finally I've got it working.

Small addition:

Why not add the uid and gid to the docker-compose.yml directly? Then you are sure it will work for everyone. Like:

# docker-compose.yml
user: "$UID:$GID"

Copy link

@awilkins awilkins commented Nov 18, 2019

The difficulty is that UID and GID are not really environment variables.

❯ env | grep UID

❯ echo $UID

They're inserted magically by your shell. I've used the .env hack mentioned by @slykar, usually with an .env.template committed to source ; this is basically the solution in this comment but solidified into a config file.

version: '2'
    image: ubuntu
    command: [/bin/echo, 'Hello, user : ${UID} in group ${GID}']
❯ docker-compose up
WARNING: The UID variable is not set. Defaulting to a blank string.
WARNING: The GID variable is not set. Defaulting to a blank string.
Creating network "env_default" with the default driver
Creating env_hello_world_1 ... done
Attaching to env_hello_world_1
hello_world_1  | Hello, user :  in group 
env_hello_world_1 exited with code 0

If you add the .env file suggested above by @slykar verbatim you just get exactly what's in it ... you have to fill it with your own values.

❯ docker-compose up
Recreating env_hello_world_1 ... done
Attaching to env_hello_world_1
hello_world_1  | Hello, user : $UID in group $GID
env_hello_world_1 exited with code 0

I usually do this with a shell script. (e.g. sed -i "s/\$UID/$UID/" .env)

What would be nice is if docker-compose called getuid() and getgid() on Linux and populated it's internal environment table with these vars, the same way that shells do.

Copy link

@Jim-Bar Jim-Bar commented Mar 12, 2020

As @Wirone signaled, it's possible to work around this issue pretty well by using:

user: "${UID:?Please export UID}:${GID:?Please export GID}"

docker-compose will immediately fail with the specified error message if the UID or GID is not exported.

But indeed it would be practical to have Compose either let us do a $(id -u):$(id -g) or provide us with the uid/gid of the caller. Maybe not using environment variables like UID and GID because that can be confusing if they are indeed not exported, but by any other means.

Copy link

@Snowshield Snowshield commented Mar 24, 2020

For people who are looking for a solution in the current version of docker-compose.

Default linux user have 1000 user id and 1000 group id. Using this values is most common case among developers.

So i found very practical to substitute these values as default in docker-compose to simplify usage of docker-compose CLI.

user: "${UID:-1000}:${GID:-1000}"

If developer needs some custom values of user id and group id or he needs full control over execution in automating tools so he can pass manually any values of UID and GID via environment variables, i.e.

  1. Custom values

    UID=1001 GID=1001 docker-compose up
  2. Full control

    UID=$(id -u) GID=$(id -g) docker-compose up

Copy link

@stale stale bot commented Sep 20, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Sep 20, 2020
Copy link

@stale stale bot commented Sep 27, 2020

This issue has been automatically closed because it had not recent activity during the stale period.

@stale stale bot closed this Sep 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet