New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`docker stack deploy` in 1.13 doesn't load `.env` file as `docker-compose up` does #29133

Open
twang2218 opened this Issue Dec 5, 2016 · 63 comments

Comments

Projects
None yet
@twang2218

twang2218 commented Dec 5, 2016

To test docker stack deploy --compose-file function, I load one of my sample docker-compose.yml:

version: '3'
services:
    nginx:
        image: "${DOCKER_USER}/lnmp-nginx:v1.2"
        build:
            context: .
            dockerfile: Dockerfile.nginx
        ports:
            - "80:80"
        networks:
            - frontend
        depends_on:
            - php
    php:
        image: "${DOCKER_USER}/lnmp-php:v1.2"
        build:
            context: .
            dockerfile: Dockerfile.php
        networks:
            - frontend
            - backend
        environment:
            MYSQL_PASSWORD: Passw0rd
        depends_on:
            - mysql
    mysql:
        image: mysql:5.7
        volumes:
            - mysql-data:/var/lib/mysql
        environment:
            TZ: 'Asia/Shanghai'
            MYSQL_ROOT_PASSWORD: Passw0rd
        command: ['mysqld', '--character-set-server=utf8']
        networks:
            - backend
volumes:
    mysql-data:

networks:
    frontend:
    backend:

In the image section of service nginx and php, I used ${DOCKER_USER} to get the docker id from environment variables. And if I use docker-compose up, it will load .env file as default envvar files, which content is:

DOCKER_USER=twang2218

However, if I use docker stack to deploy this docker-compose.yml, I will got following errors:

$ docker stack deploy --compose-file docker-compose.yml lnmp
Ignoring unsupported options: build

Creating network lnmp_frontend
Creating network lnmp_backend
Creating network lnmp_default
Creating service lnmp_php
Error response from daemon: rpc error: code = 3 desc = ContainerSpec: "/lnmp-php:v1.2" is not a valid repository/tag

As you can see, as docker stack deploy command didn't load .env file, the ${DOCKER_USER} was replaced by empty string, which cause image name become invalid.

If .env file was loaded, the final image name should be twang2218/lnmp-php:v1.2.

The environment substitution is actually working, if I run the command this way:

$ DOCKER_USER=twang2218 docker stack deploy --compose-file docker-compose.yml lnmp
Ignoring unsupported options: build

Creating network lnmp_frontend
Creating network lnmp_backend
Creating network lnmp_default
Creating service lnmp_mysql
Creating service lnmp_nginx
Creating service lnmp_php

And we can verify it's working by docker service inspect command:

$ docker service inspect lnmp_php | grep Image
                    "Image": "twang2218/lnmp-php:v1.2@sha256:4f1aef1350aeef3f757f6b6da8f2e1a79ff849f61382320e4b668bfe2b0d1c5a",

The image name is twang2218/lnmp-php:v1.2, which is correct.

I tested this feature on Digtial Ocean droplet, which installed docker 1.13.0-rc2 via docker-machine.

Here is the version:

$ docker version
Client:
 Version:      1.13.0-rc2
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   1f9b3ef
 Built:        Wed Nov 23 06:32:39 2016
 OS/Arch:      linux/amd64

Server:
 Version:             1.13.0-rc2
 API version:         1.25
 Minimum API version: 1.12
 Go version:          go1.7.3
 Git commit:          1f9b3ef
 Built:               Wed Nov 23 06:32:39 2016
 OS/Arch:             linux/amd64
 Experimental:        false

Here is the docker info:

root@d1:~/docker-lnmp# docker info
Containers: 7
 Running: 1
 Paused: 0
 Stopped: 6
Images: 4
Server Version: 1.13.0-rc2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 43
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: vyf3mgcj3uonrnh5xxquasp38
 Is Manager: true
 ClusterID: jb8rxvd6ptrn3psfkiixxed7r
 Managers: 1
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 138.197.195.206
 Manager Addresses:
  138.197.195.206:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 51371867a01c467f08af739783b8beafc154c4d7
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-51-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 488.5 MiB
Name: d1
ID: E6UB:PHX6:I2KY:Q35T:PCCI:MFDQ:ZMMN:2X7K:DEOZ:PAP7:4BUC:FP6X
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Labels:
 provider=digitalocean
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

@twang2218 twang2218 changed the title from `docker stack deploy` in 1.13 doen't load `.env` file as `docker-compose up` does to `docker stack deploy` in 1.13 doesn't load `.env` file as `docker-compose up` does Dec 5, 2016

@vdemeester vdemeester added this to the 1.13.0 milestone Dec 5, 2016

@dnephin dnephin removed the version/1.13 label Dec 5, 2016

@dnephin dnephin removed this from the 1.13.0 milestone Dec 5, 2016

@dnephin

This comment has been minimized.

Member

dnephin commented Dec 5, 2016

This is by design. The .env support is a feature of Compose, not of the file format.

We can discuss adding this for a future release, but I don't know if it's really the best option.

@twang2218

This comment has been minimized.

twang2218 commented Dec 6, 2016

The .env support is quite useful in Compose, we used it in many of our compose files. It separates dynamic parts and static parts of docker-compose.yml file. With load .env by default, we can just provides different .env file for different environments, and keeping the docker-compose.yml and related scripts static.

It's not possible to use env_file or environment in docker-compose.yml to achieve the same envvars substitution result. .env is the most easy way to do it.

Otherwise, we have to prefix export in each line of .env file and manually source .env every time before loading the docker-compose.yml, and the extra steps sometimes are prone to mistake.

@vovimayhem

This comment has been minimized.

vovimayhem commented Dec 13, 2016

I've just noticed this.
I assumed/expected it to work in the same way it does for Compose, but is not the case for docker deploy.

In my particular case, I'm was expecting to use the .env file to store sensitive data (passwords, API keys, etc) used in the services I'll be creating in the stack:

version: '3'

volumes:
  data:
    driver: local

networks:
  backend:
    driver: overlay

services:
  rabbitmq:
    image: rabbitmq:${EXCHANGE_RABBITMQ_TAG}
    volumes: [ "data:/var/lib/rabbitmq" ]
    logging: { driver: gelf, options: { gelf-address: "udp://0.0.0.0:12201" } }
    networks: [ "backend" ]
    ports: [ "15672:15672", "5672:5672" ]
    environment:
      RABBITMQ_DEFAULT_USER: ${EXCHANGE_RABBITMQ_USER}
      RABBITMQ_DEFAULT_PASS: ${EXCHANGE_RABBITMQ_PASS}
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.labels.queue-host == true

While this Compose file will be checked in on Git, the .env file would be ignored.

I've followed the whole stuff/history of the *.dab vs compose files, and I feel you guys are trying to avoid something - or proposing a better solution - but I lost track of the whole discussion...

@dnephin dnephin removed their assignment Dec 27, 2016

@akhildangore

This comment has been minimized.

akhildangore commented Jan 6, 2017

Hello All,

Docker version: v1.13.0-rc4

I am getting error : Ignoring unsupported options: build, Does it mean it will not create build from Dockerfile ?

And also getting same error for network_mode: Ignoring unsupported options: network_mode.

Below is command:
DOCKER_IMAGE=akhil123 docker stack deploy -c docker-compose.yml foo

Many thanks in advance.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jan 6, 2017

@akhildangore please don't comment on issues with questions that are not directly related. The docker stack deploy feature, by design does not perform builds. Reason for this is that a build is performed on the host that the command is run from, so the image will only be available on that node. When deploying that image in a Swarm, the service cannot be started on other nodes. To deploy services, make sure the image you're deploying is pushed to a registry (or for testing; make sure the image is available on every node in the swarm)

@vovimayhem

This comment has been minimized.

vovimayhem commented Jan 24, 2017

Are there any cases/discussion in favor of/against supporting this behavior on docker deploy?

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jan 24, 2017

@vovimayhem no decision was made yet on .env file, but you may be interested in #30144, which adds support for secrets to the compose file

@vovimayhem

This comment has been minimized.

vovimayhem commented Jan 24, 2017

I've just found that discussion. Excellent!

@vdemeester vdemeester removed their assignment Jan 27, 2017

@whoan

This comment has been minimized.

whoan commented Feb 3, 2017

I am using a bash function as a workaround.

You may adapt it to your needs until the secrets feature be released:

dsd() {
    stack=${1:-${PWD##*/}} # by default, the name of the cointaining folder
    compose_file=${2:-docker-compose.yml}

    if [ ! -f $compose_file ]; then
        echo "Misses compose file: $compose_file" >&2
        return 1
    fi

    # execute as a subcommand in order to avoid the variables remain set
    (
        # export variables excluding comments
        [ -f .env ] && export $(sed '/^#/d' .env)

        # Use dsd your_stack your_compose_file to override the defaults
        docker stack deploy --compose-file $compose_file $stack
    )
}
@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Feb 4, 2017

For those subscribing to this issue; secrets support for docker-compose files will be included in the 1.13.1 release, which should be not too far in the future

@vovimayhem

This comment has been minimized.

vovimayhem commented Feb 8, 2017

@whoan I'm using this as a workaround:

env $(cat .env | grep ^[A-Z] | xargs) docker stack deploy --compose-file docker-compose.yml [STACK_NAME]

That way, variables don't get stuck on the terminal window

@ntwrkguru

This comment has been minimized.

ntwrkguru commented Dec 20, 2017

Thanks, but any word on simply supporting variable substitution from a file?

[edit] This shouldn't be an issue (but could be) as I'm using Ansible to perform the task with become: yes, but the user is still the same user who sources the environment vars.

@gaui

This comment has been minimized.

gaui commented Dec 20, 2017

I can't believe this isn't possible. We would like to keep a single stack file and be able to change deployment options using dynamic variables.

This would work but is ugly and a hack:

echo "$(docker-compose -f stack.yml config 2>/dev/null)" | docker stack deploy -c- stack_name

@ntwrkguru

This comment has been minimized.

ntwrkguru commented Dec 21, 2017

I've been messing with this for many hours and looks like doing some background script work (I'll be using CI to do this) to create a DAB file from these various pieces. It would be nice to be have a docker tool that would do this... :-) Maybe add a --env-file version.env added to docker-compose bundle?

In my use case, I want to use a version.env "manifest" to track and control the versions of containers/images that comprise a platform. My process looks like this:

  • Update .env file
  • Source .env file to populate the shell vars
  • Run docker-compose -f stack.yml config > docker-compose.yml
  • Run docker-compose pull to register the image digests
  • Run docker-compose bundle -o stack.version.dab DAB file

At this point, ideally I would be able to docker deploy --host manager.swarm.com --bundle-file stack.version.dab --with-registry-auth --resolve-image always stack-name, but I get that it's not possible now. In this case, I would scp the DAB file to the manager and execute the docker deploy.

Is this the workflow Docker had in mind @thaJeztah? Is there a better way?

[edit] This does not work since a DAB ignores volume:, network:, and deploy: directives. I've taken to simply sourcing an env file to the shell, rebuilding a temp compose file, then deploying that.

@gaui

This comment has been minimized.

gaui commented Jan 11, 2018

I would love if .env files were read by default like in Docker Compose and --env-file (short -e) would be added for docker stack deploy to be able to override environment variables and the defaults from the .env file.

@ntelisil

This comment has been minimized.

ntelisil commented Jan 11, 2018

+1
It would be quite convenient to read the .env in the first place as docker-compose does.
Either by default or just pass the env file as parameter in docker stack command line.

The two already mentioned workarounds (find them also below) seem to work although a little bit ugly compared to the proposed 'fix'.

echo "$(docker-compose -f stack.yml config 2>/dev/null)" | docker stack deploy -c- stack_name

env $(cat .env | grep ^[A-Z] | xargs) docker stack deploy --compose-file docker-compose.yml [STACK_NAME]

@blop

This comment has been minimized.

blop commented Jan 11, 2018

I would add that having docker-compose installed only to be able to process these variables can also be quite dangerous actually.

If anyone deploy the stack using docker-compose up -d by mistake instead of using docker stack deploy, it would most probably induce applications errors and file corruption.

@rnickle

This comment has been minimized.

rnickle commented Feb 16, 2018

Having environment variable interpolation in deploy is the best solution for continuous integration.

One issue with the use of '.env' is that it overlaps with similar file names used by, for example, Ruby on Rails, having the ability to specify an environment file by argument is superior.

@ntelisil's suggestions are well taken, they could be supplemented or modified by:

https://unix.stackexchange.com/questions/294835/replace-environment-variables-in-a-file-with-their-actual-values:

You could use envsubst (part of gnu gettext):
envsubst < infile

I used:

$ envsubst < docker-compose.yml-template > docker-compose.yml

And that worked fine, but I needed to add 3Mb or so to my container.

@ntwrkguru

This comment has been minimized.

ntwrkguru commented Feb 16, 2018

Nice @rnickle, but still "hacky" compared to Docker being able to do this the way it used to. It seems (and please Docker dudes correct me if I'm wrong) that there is very little effort going into swarm-mode since the announcement of K8s support, so I'm doubtful that any of these regressions will be addressed.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Feb 16, 2018

And that worked fine, but I needed to add 3Mb or so to my container.

If the environment variables are set, docker slack deploy should already interpolate them

compared to Docker being able to do this the way it used to.

Docker never had this feature; docker compose had; docker compose and docker stack deploy share the file format but not nescessarily all behaviors of the software itself.

that there is very little effort going into swarm-mode since the announcement of K8s support

Compose file handling is all done client side, and used both for k8s and swarmkit, so nothing slowing down in this area, other than “there’s only so much we can do at any given time”

For those waiting for that feature; the upcoming release will have support for multiple compose files for docker stack deploy, so if you’re currently using that feature in docker compose: the upcoming docker release will support this as well

@ntwrkguru

This comment has been minimized.

ntwrkguru commented Feb 16, 2018

Semantics aside, thanks for the update. When I say Docker, I don't mean necessarily the daemon or engine, but the company that builds the tools. The genesis for the comment about there being little effort really stems from the fact that swarm-mode has been out for over a year and is still WAY behind where we were with a single-node using compose.

I can appreciate that they are different, but use the same file format, but that actually leads to more confusion, and potentially disappoint, when users discover that they have to trade features for using swarm. That and the fact that the DAB files are STILL being listed as experimental. Unfortunately, we made the decision to use swarm-mode in a big project, or I wouldn't even care. Fortunately, we won't make that mistake again.

[edit]

If the environment variables are set, docker slack deploy should already interpolate them

Doing this in a CI environment is a royal ass pain. Essentially, I ended up installing compose, sourcing the envvars, washing the compose file through (after pulling the images) and then spitting out a new compose file for use with stack deploy. (And I won't even start in on how painful it is when :latest doesn't mean :latest in swarm-mode, since this issue is specifically about variable interpolation.)

@djbingham

This comment has been minimized.

djbingham commented Feb 26, 2018

EDIT: I hope this post doesn't come across aggressively. I love the Docker Stack concept and I do think the whole Docker suite is pretty well supported. It just seems that Docker (the organisation) currently doesn't view its products the same way users do, which is unfortunate and appears to be the cause of most issues I've run into using Compose and Stack.

Docker never had this feature; docker compose had; docker compose and docker stack deploy share the file format but not nescessarily all behaviors of the software itself.

I feel like this statement reveals a fundamental disconnect between the way Docker's developers and users view these products. As a user, whether I'm working with Docker Engine, Docker Compose or Docker Swarm, it's all Docker and should behave consistently as far as possible.

My understanding of the purpose of these products is:

  • Docker Engine is for building and running individual containers
  • Docker Compose is for running a suite of containers in development
  • Docker Stack is for deploying containers to production

One of the key selling points of Docker (and containers in general) is easy reuse of the same app in multiple environments. Therefore, Compose should be additive to Engine and Stack should be additive to Compose. Particularly as they share the same file format, Stack not supporting features of Compose leads to confusion for developers and added complexity in CI processes.

@ntwrkguru

This comment has been minimized.

ntwrkguru commented Feb 26, 2018

I wouldn't say compose is for dev and stack for prod, necessarily. I view compose as being for a multi-container application or "system" and stack for multi-host deployments using swarm-mode as the scheduler/orchestrator. Otherwise, your comment is 100% spot on. I also am frustrated (if it weren't obvious by my other comments) that there is a disconnect between the developers and the users as to how the product(s) are being used.

@aardelean

This comment has been minimized.

aardelean commented Feb 27, 2018

oh, wow. I joined the confusion and pain while evaluating docker swarm for a big project as well.
The secrets are all fun and games, just that not all external docker images support reading secrets from files. I do not want to modify for each of them and place a custom image. Might use them for my projects, yes, but for external projects would be a no go.
So that leaves env variables. But hey, they behave differently from compose to stack! Imagine my surprise when I noticed that they don't really work.
How is it possible to have docker stack without env support?best would be in my opinion --env-file=dev.env , so it wouldn't interfere with other env files... but the support for this is horrible. I must go back to k8s unfortunately, as the stack missing this from the start, could prove to be a horrible choice while in production, who knows how many other things will be missing.

@biels

This comment has been minimized.

biels commented Mar 6, 2018

It's been two years and we are still like that... I don't want to come across as rude, but come on! It's matter of reusing a bit of code from docker-compose!

@jmunson

This comment has been minimized.

jmunson commented Mar 27, 2018

Another similar problem I just ran in to is that even if you use env_file, the behavior is still not exactly the same.

I use env_file to load paths specified from the project root, but my compose file is in a different subdirectory. To launch with docker-compose, I just use --project-directory . -f path/to/docker-compose.yml

docker stack does not allow me to specify the project-directory, causing all of my env_files to fail to load.

Changing env_file to "../../path/to/envrc" does not work with docker-compose, as these files must be inside the project root dir

@thiagolsfortunato

This comment has been minimized.

thiagolsfortunato commented May 22, 2018

+1

1 similar comment
@zdanos

This comment has been minimized.

zdanos commented May 22, 2018

+1

@robwithhair

This comment has been minimized.

robwithhair commented Jul 21, 2018

+1 this would be really useful. Don't see a downside really.

@joao-fidalgo

This comment has been minimized.

joao-fidalgo commented Jul 26, 2018

It's incredibly dumb that docker doesn't support this yet.

@biels

This comment has been minimized.

biels commented Jul 26, 2018

+1 @joao-fidalgo It definitely is.

@sachnk

This comment has been minimized.

sachnk commented Aug 6, 2018

+1

@mariotacke

This comment has been minimized.

mariotacke commented Aug 14, 2018

Just ran into the same thing as we are trying out swarm. I've read through all of the messages in various threads and am left extremely frustrated. IMO, some really good points and strong arguments were made by @ntwrkguru and others. I hope we can consider them to get this feature in; hacks won't cut it.

@trocho

This comment has been minimized.

trocho commented Sep 12, 2018

.env file

export MYSQL_USER=user

source .env && docker stack deploy

It will also work with docker-compose up for bash

@thernstig

This comment has been minimized.

thernstig commented Nov 4, 2018

.env file

export MYSQL_USER=user

source .env && docker stack deploy

It will also work with docker-compose up

That would only work for bash and not other shells.

@robbyemmert

This comment has been minimized.

robbyemmert commented Nov 7, 2018

This confused me like crazy. My app would come up using docker-compose, but not with docker stack deploy. There should at least be a disclaimer prominently displayed in the docs

But IMO this is a bug, and is degrading the docker experience.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment