Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Variable substitution #2380

Open
gponsu opened this issue Nov 12, 2015 · 29 comments

Comments

@gponsu
Copy link

commented Nov 12, 2015

This new feature is amazing, but I had a problem. I explain:

In our dev team, to enter in docker with a user with the same UID and GID that of the host machine, we made the following:

# docker-compose.yml
app:
  image: rails:4.2.4
  working_dir: /app
  ...
  extends:
    file: docker-custom.yml
    service: app

The docker-custom.yml file we have untracked of git, so everyone can define their own settings.

# docker-custom.yml
app:
  # Set your user and group id.
  # View your user id with command 'id -u' and your group id with 'id -g'.
  user: '1000:1000'
  environment:
    BUNDLE_JOBS: 4
    BUNDLE_PATH: /app/.bundle
    BUNDLE_APP_CONFIG: /app/.bundle
  # You can use env_file option instead of environment if you are using one file
  # to save environment variables
  # env_file: .env

With the new functionality of Variable substitution, I was excited, because we could spare us all this, and minimize it to this line in docker-compose.yml:

user: "${UID}:${GID}"

Unfortunately this does not work, because UID and GID are shell variables (not env variables), and os.environ do not get them.

Would it be possible get the shell envs too? Something like (I know nothing about python 😞):

os.system("set")

He had also thought of another possibility, if the option user is user: host or similar, internally get the uid and gid with:

os.getgid()
os.getuid()

And setup docker with this config for user.

Thank you very much for everything, and apologies for my horrible English.

@oinak

This comment has been minimized.

Copy link

commented Nov 12, 2015

👍

I have the exact same problem, as soon as I saw variable substitutio I tried for $UID to no avail

I could make the file ownership issues of containers mounting code folders go away.

Please please please enable shellvars along with envvars to be sunstituted on docker-compose.yml files.

Thank you for the excellent work.

@dnephin dnephin added the area/config label Nov 12, 2015
@dnephin

This comment has been minimized.

Copy link
Contributor

commented Nov 12, 2015

The only different is that UID isn't exported. I don't think it's correct to go reading un-expected shell variables.

I would suggest export UID.

@deeky666

This comment has been minimized.

Copy link

commented Dec 15, 2015

👍 I think this is a common use-case when mouting directories from the host and accessing them inside the container. Having to export extra environment variables on the host side is unnecessary extra configuration on the host side which more or less defeats this whole purpose.
Until using docker composer, I was evaluating the user and group id via id -u and id -g in the run command, which is not (yet) possible in docker composer configuration.
Some sort of solution to this would be very welcome.

@cookandy

This comment has been minimized.

Copy link

commented Jan 29, 2016

+1

Another use case would be trying to set the hostname of a container to the hostname of your host:

myapp:
  hostname: ${HOSTNAME}

My workaround right now is to set it before calling docker-compose

HOSTNAME=$HOSTNAME docker-compose up
@dvddarias

This comment has been minimized.

Copy link

commented Apr 19, 2016

I have exactly the same problem. In our team all you need to do to setup a new developer is
to run:

git clone ....
cd ....   
export UID   
docker-compose up   

We need the same user on the container and the host because of the big amount of Ruby on Rails commands that generate new files on the project volume. The sad part is that you have to export the UID variable on every new shell session for docker-compose up to work or add the variable to your shell startup script, which is sub-optimal on such a common use case.

This problem is specific to programmers using docker-compose as a tool:

Using docker CLI directly you don't have this problem because it is assumed that you run them within a shell and there you can use the variable $UID without exporting it.

Using docker through the API is also possible getting the current user id with os.geteuid() or something similar.

When using docker-compose i haven't find the way of doing it without running commands, so the "same user as the host" service requirement can't be made effective on any of the repository files.

A solution may be to setup some specific environment variable (let's say COMPOSE_UID) to the same UID as the user running the docker-compose command with os.geteuid() and then use it within the file as ${COMPOSE_UID}.

@mknapik

This comment has been minimized.

Copy link

commented Oct 4, 2016

👍
Same use case, handling shared directories at CI server.

@edannenberg

This comment has been minimized.

Copy link

commented Mar 30, 2017

Can't spam enough 👍 for this. UID/GID of the current host process should be provided by docker-compose in some way.

@ScreamingDev

This comment has been minimized.

Copy link

commented Apr 11, 2017

So as my #4725 got closed due to duplicates I need to ask the remaining question here:

How to solve/bypass this?

I guess it can't be left that way it is as long as there is no alternative.
It really is something that makes work a lot easier for developers.
Doesn't matter on production server but for dev and test server it helps.

Does the world build a wrapper around docker-compose having export UID in it?
Is it feasible by docker compose to solve this?

@durkode

This comment has been minimized.

Copy link

commented Aug 3, 2017

I'm also going to add my voice to this. Developing with Django from within a container and would love for this to work out of the box without having to export UID and GID.

I personally am not concerned how it is done, but a way for files written to the mounted volume in the container to match the host user without having to export UID and GID is my end aim. Currently running the export isn't a big deal, it would just be nicer if this wasn't the case

@sokratisg

This comment has been minimized.

Copy link

commented Aug 3, 2017

To be honest we sort of worked around this issue by doing something similar to:

  1. Make sure UID is exported in each user's bash shell. Just create the following file and restart:
    /etc/profile.d/docker.sh
export UID
  1. Add the necessary arguments in the docker-compose.yml file which is used for local development purposes:
...
  app:
    build:
      context: .
      args:
        username: $USER
        userid: $UID
...
  1. Use these arguments in the respective Dockerfile which is also used solely for local development:
...
ARG username
ARG userid
RUN useradd --no-create-home --user-group --shell /bin/bash --home-dir $APP_HOME --uid $userid $username
RUN chown -R $username: $APP_HOME
...

Only drawback: UID conflicts with images that by default use the same one with any local user. (i.e. official node images have node user with UID=1000 and that conflicts with the initial UID a typical Ubuntu desktop assigns to its first user, tha one you're going to use post-installation)

@edannenberg

This comment has been minimized.

Copy link

commented Aug 3, 2017

Only drawback: UID conflicts with images that by default use the same one with any local user. (i.e. official node images have node user with UID=1000 and that conflicts with the initial UID a typical Ubuntu desktop assigns to its first user, tha one you're going to use post-installation)

Just use a static user like node in the image and change it's UID on container start as needed. The username is irrelevant when it comes to permissions, all that matters is a matching UID.

@sokratisg

This comment has been minimized.

Copy link

commented Aug 3, 2017

That is true if you use an image from a Dockerfile. Not true if you just want to start a node app by pulling the default image without any fiddling around or any custom shell script as your command.

@edannenberg

This comment has been minimized.

Copy link

commented Aug 3, 2017

@sokratisg You seem to use a local Dockerfile already, replacing useradd with usermod --uid $userid nodeshould do the trick.

@sokratisg

This comment has been minimized.

Copy link

commented Aug 3, 2017

@edannenberg excuse me but I think I may have been misunderstood, maybe my comment wasn't very clear. I'll try to give you an example.

Let's say your desktop user has UID=1000 (default for first user, at least on Ubuntu) and that you've written a node app that outputs or generates some local files. You want them to be written with the correct ownership (UID=1000) since you're also working locally and want to have read/write access on them without using sudo for your favorite editor/IDE. If you go with the solution I described then upon startup of the node container you will end up with a UiD conflict since official node images come pre-baked with a node user that has the same UID (1000).

This is what I was trying to describe; that this workaround is not universal and comes with some possibilities of a conflict so this issue is still relevant.

@edannenberg

This comment has been minimized.

Copy link

commented Aug 4, 2017

@sokratisg I got that, the problem is that your solution tries to add another user with an already existing UID. My point was that you can avoid that by just changing the UID of an existing static user in the image, i.e. node. In theory this will still leave the possibility of a conflict, but in practice the image will most likely only have one user with a UID that could conflict with the typical user UID space (>= 1000, everything below 1000 is usually reserved for system users).

A typical nodejs compose file in our projects looks like this:

version: '2.2'
services:
  app:
    build:
      dockerfile: ${PWD}/docker/Dockerfile
      context: ./docker
      args:
        - NODEJS_UID=${UID}
        - NODEJS_GID=${GID}
    user: nodejs
    working_dir: ${PWD}
    volumes:
      - ~/.npm:/home/nodejs/.npm
      - ${PWD}:${PWD}

The nodejs user already exists in the pulled image, the local Dockerfile just changes the UID/GID of said user.

@vipseixas

This comment has been minimized.

Copy link

commented Oct 11, 2017

My workaround for the usecase of ownership of mounted volumes is to use a impersonate.sh script as the entrypoint of the image, like this:

#!/bin/bash

USER_NAME="deployer"

files=(/work/source/*) && DIR=${files[-1]}
USER_ID=$(stat -c "%u" ${DIR})

echo "Using ${USER_ID} from owner of dir ${DIR}"

id "${USER_NAME?}" >/dev/null 2>&1

if [[ $? -ne 0 ]]; then
	echo Creating user ${USER_NAME} with UID ${USER_ID}
	useradd --home-dir /work --uid ${USER_ID} --shell /bin/bash --no-create-home ${USER_NAME}
fi

su --login --preserve-environment --command "/scripts/deploy.sh $1 $2 $3" ${USER_NAME}

/work/source is where I mount my external source code inside the container.

@38leinaD

This comment has been minimized.

Copy link

commented Jan 4, 2018

+1

@ChristianKreuzberger

This comment has been minimized.

Copy link

commented Apr 17, 2018

Django Developer here:
Using $UID and $GID within docker-compose.yml would be so great! Please please please!

@CodeCorrupt

This comment has been minimized.

Copy link

commented May 27, 2018

One better and more versatile would be the ability to include shell script in the docker-compose. Something like this would be helpful

services:
  service_name:
    environment:
      - UID=$(id -u)
      - GID=$(id -g)

I'm currently accomplishing this by wrapping the docker-compose up in a shell script that sets those variables, however that's a little clunky. This could also be useful for dynamically setting other fields outside of just environment variables.

Edit: A lot of the comments are saying to just use $UID and $GID. I don't know about other distros but my Ubuntu Server 16.4 doesn't have that by default so adding the following to your .bashrc can add that and has been working well for me.

UID=$(id -u)
GID=$(id -g)
@reduardo7

This comment has been minimized.

Copy link

commented May 28, 2018

I use something like this:

version: "2.1"

services:
  app:
    volumes:
      - /etc/passwd:/etc/passwd:ro
      - /etc/group:/etc/group:ro
    user: $UID:$GID
@awilkins

This comment has been minimized.

Copy link

commented Jun 1, 2018

Using a .env file ; sadly, you can't use substitutions in a .env file, it's all literals.

@rcdailey

This comment has been minimized.

Copy link

commented Oct 10, 2018

Maybe I've got the wrong idea here, but why do folks use the user and group ID of the current user when they run their services? So far, I've personally been creating system users in Ubuntu for this, one per service. So for my plex container, I have a user called plex and a shared group called docker_services. My next service, sonarr, will have a user called sonarr and also use the group docker_services. This allows me to assign very specific permissions to bind mount directories, and prevent other services from touching files they aren't supposed to.

I have a shell script I run right now to launch my services with users in this way, but unfortunately since I can't run subshells inside the yml file, I haven't been able to use docker compose:

docker run -d \
    --name plex \
    --restart unless-stopped \
    --network host \
    -e PLEX_UID="$(id -u plex)" \
    -e PLEX_GID="$(id -g plex)" \
   #...etc....

Is this not the recommended way to run permissions for containers? I'd rather not start them under my personal account I use to SSH into the machine itself, since this user account was not meant to be used by services running 24/7.

Assuming what I'm doing isn't completely bonkers, then I think there's still some ways to go before I use docker compose. Right now I use a series of bash scripts to start my services, semantically similar to docker-compose up. This is the only way I've been able to have the flexibility to do some system introspection while running docker containers.

@javabrett

This comment has been minimized.

Copy link
Contributor

commented Oct 10, 2018

Maybe I've got the wrong idea here, but why do folks use the user and group ID of the current user when they run their services?

Largely in a development context, in cases where it is desirable to use bind-mounts, and you can't, or it's not easy or practical to prevent the container from creating files. If such files are created with users/groups that don't match the host-container, they become painful to clean-up in development-cycles.

@Wirone

This comment has been minimized.

Copy link

commented Dec 12, 2018

@gponsu We're using environment variable which is documented in our README.md:

Add export DOCKER_COMPOSE_RUN_AS_USER="$(id -u):$(id -g)" to your ~/.bashrc or other place that will define it every time when starting terminal.

and then in docker-compose.yml:

user: "${DOCKER_COMPOSE_RUN_AS_USER:?You must define user:group for permissions handling, look at README}"

The :? part enforces that variable, so if it's not defined, build will fail.

But yeah, I came here because I was looking for executing Bash script in docker-compose.yml because I wanted to pass git describe result as ENV variable.

@lygstate

This comment has been minimized.

Copy link

commented Mar 3, 2019

looking for better solution

@ivanbogomoloff

This comment has been minimized.

Copy link

commented Apr 10, 2019

Working for me.

environment:
    APP_USER_UID: $APP_USER_UID

and run docker-compose with export like this

export APP_USER_UID=$UID && docker-compose up -d
@MtDalPizzol

This comment has been minimized.

Copy link

commented May 18, 2019

Just ran into a scenario where I needed to pass the UID an GID to compose. I'll just live what I did here so maybe it helps someone out there. I'm running Mint (Ubuntu based distro) and here is the step by step that got this working for me:

  1. Edit ~/.bashrc file adding the following to the end of it:
export COMPOSE_UID=$(id -u)
export COMPOSE_GID=$(id -g)
  1. Add the environment vars to docker-compose.yml
version: "3"
services:
  storage:
    image: richarvey/nginx-php-fpm:latest
    volumes:
      - .:/var/www/html
    environment:
      WEBROOT: /var/www/html/www
      PUID: ${COMPOSE_UID}
      PGID: ${COMPOSE_GID}
    ports:
      - 8081:80
  1. Restart the terminal or start a fresh one.

  2. Now you can run compose without any additional magic:

$ docker-compose up
@egdoc

This comment has been minimized.

Copy link

commented Jul 15, 2019

When using sudo to launch docker-compose, one can use the SUDO_UID and SUDO_GID environment variables which contain respectively the UID and GID of the user who launched the command:

version: '3'
services:
    php-apache:
        build:
            context: ./php-apache
            args:
                APACHE_UID: "${SUDO_UID}"
        ports:
            -  8080:80
        volumes:
            -   ./DocumentRoot:/var/www/html:z
@slykar

This comment has been minimized.

Copy link

commented Jul 23, 2019

Use an .env file

I often have a sample .env file for my projects.
You can set required variables there.

# .env
UID=$UID
GID=$GID

docker-compose command picks this file automatically and makes the declared env vars available in the YAML file.

Ask users to create an override file and set user: manually.

# docker-compose.override.yaml
...
user: 1000:1000
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.