New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENV variables not available at build time for Dockerfile deploys #1860

Closed
craigbeck opened this Issue Jan 12, 2016 · 33 comments

Comments

Projects
None yet
9 participants
@craigbeck

craigbeck commented Jan 12, 2016

The docs say "The variables are available both at run time and during the application build/compilation step." but I've not seen this actually happen at build time.

actual app config:

$ ssh dokku@test config adminapp                                                                                                                                                                
=====> adminapp config vars
API_BASE:              //adminapp-api.test.zipwhip.com
DOKKU_APP_RESTORE:     1
DOKKU_DOCKERFILE_PORT: 80
DOKKU_NGINX_PORT:      80
NODE_ENV:              production
PATH_PREFIX:           /admin

I'm logging from my build script, so at build time I see:

Step 1 : RUN NODE_ENV=production webpack -p
 ---> Running in 0826574bc4e0
===> Building with NODE_ENV 'production'
===> Building with PATH_PREFIX 'undefined'
===> Building with API_BASE 'undefined'
Hash: 54683fe2b93e4d330eed
Version: webpack 1.12.11
Time: 170802ms
...

The NODE_ENV var is only set because I prefixed the Dockerfile build step with NODE_ENV=production while the other vars I expect to be set (PATH_PREFIX and API_BASE) are still undefined.

This is with dokku version 0.4.7

$ ssh dokku@test version
0.4.7
@craigbeck

This comment has been minimized.

Show comment
Hide comment
@craigbeck

craigbeck Jan 12, 2016

This is using a Dockerfile based build

craigbeck commented Jan 12, 2016

This is using a Dockerfile based build

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jan 12, 2016

Member

We only support build-time env vars for buildpack deploys.

@michaelshobbs Was this a conscious decision? I know we implemented dockerfile deploys way after build-env, so it's possible we just missed it (given that dockerfile-deploys aren't in the forefront of our development). Seems like they could be useful.

Member

josegonzalez commented Jan 12, 2016

We only support build-time env vars for buildpack deploys.

@michaelshobbs Was this a conscious decision? I know we implemented dockerfile deploys way after build-env, so it's possible we just missed it (given that dockerfile-deploys aren't in the forefront of our development). Seems like they could be useful.

@josegonzalez josegonzalez changed the title from ENV variables not available at build time to ENV variables not available at build time for Dockerfile deploys Jan 12, 2016

@craigbeck

This comment has been minimized.

Show comment
Hide comment
@craigbeck

craigbeck Jan 12, 2016

I have a Dockerfile based deployment that runs nginx and serves a statically-built web app, so the standard env vars aren't available at run time because the app is static (i.e. no process at run time to modify the response). The documentation isn't clear on these not being available (honestly thought it was just a bug in the older version of Dokku we had been running, but it was still an issue after upgrading)

craigbeck commented Jan 12, 2016

I have a Dockerfile based deployment that runs nginx and serves a statically-built web app, so the standard env vars aren't available at run time because the app is static (i.e. no process at run time to modify the response). The documentation isn't clear on these not being available (honestly thought it was just a bug in the older version of Dokku we had been running, but it was still an issue after upgrading)

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jan 12, 2016

Member

Yeah I agree the docs aren't clear. I'm just trying to figure out if we should support it, and if so, what the best way to do that is.

We'll get this issue sorted out soon enough (@michaelshobbs is probably at lunch) and figure out if the only thing we should do is update docs, or if we need to update docs/implement the functionality. IMO this is a bug and we need to implement it (though potentially for the upcoming 0.5.0 since it might break existing dockerfile apps).

I'm on our irc room as savant if you'd like to chat there :)

Member

josegonzalez commented Jan 12, 2016

Yeah I agree the docs aren't clear. I'm just trying to figure out if we should support it, and if so, what the best way to do that is.

We'll get this issue sorted out soon enough (@michaelshobbs is probably at lunch) and figure out if the only thing we should do is update docs, or if we need to update docs/implement the functionality. IMO this is a bug and we need to implement it (though potentially for the upcoming 0.5.0 since it might break existing dockerfile apps).

I'm on our irc room as savant if you'd like to chat there :)

@craigbeck

This comment has been minimized.

Show comment
Hide comment
@craigbeck

craigbeck Jan 12, 2016

I did some poking around in the Docker docs and repo. @Jaon-Grant's comment was useful: moby/moby#6822 in pointing me to the Docker --build-arg options and ARG command... which renders my Dockerfile onbuild a lot less elegant than it was but I'm not sure what to do about it right now.

I'm trying a combination of Dokku docker-options with build-args set on the build phase with changes to my Dockerfile.

craigbeck commented Jan 12, 2016

I did some poking around in the Docker docs and repo. @Jaon-Grant's comment was useful: moby/moby#6822 in pointing me to the Docker --build-arg options and ARG command... which renders my Dockerfile onbuild a lot less elegant than it was but I'm not sure what to do about it right now.

I'm trying a combination of Dokku docker-options with build-args set on the build phase with changes to my Dockerfile.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jan 12, 2016

Member

Yeah, my thinking is that we would use our docker-options integration to make a small pluginhook that adds them as a build-arg for dockerfile deploys. I'm not sure how that would work, but it seems possible.

Member

josegonzalez commented Jan 12, 2016

Yeah, my thinking is that we would use our docker-options integration to make a small pluginhook that adds them as a build-arg for dockerfile deploys. I'm not sure how that would work, but it seems possible.

@michaelshobbs

This comment has been minimized.

Show comment
Hide comment
@michaelshobbs

michaelshobbs Jan 12, 2016

Member

See the discussion here: #1255 (comment)

Member

michaelshobbs commented Jan 12, 2016

See the discussion here: #1255 (comment)

@michaelshobbs

This comment has been minimized.

Show comment
Hide comment
@michaelshobbs

michaelshobbs Jan 12, 2016

Member

I'm not sure when --build-arg was introduced but perhaps that can now be used in the build-env plugin to provide build time env vars.

Member

michaelshobbs commented Jan 12, 2016

I'm not sure when --build-arg was introduced but perhaps that can now be used in the build-env plugin to provide build time env vars.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jan 12, 2016

Member

Looks like 1.9.0, so fairly recent... I wonder if we can optionally support this? I don't remember where we left off re: supported docker versions in 0.5.0.

Member

josegonzalez commented Jan 12, 2016

Looks like 1.9.0, so fairly recent... I wonder if we can optionally support this? I don't remember where we left off re: supported docker versions in 0.5.0.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jan 12, 2016

Member

Somewhat relevant: moby/moby#13490

Member

josegonzalez commented Jan 12, 2016

Somewhat relevant: moby/moby#13490

@michaelshobbs

This comment has been minimized.

Show comment
Hide comment
@michaelshobbs

michaelshobbs Jan 12, 2016

Member

Specifically

Build-time environment variables (#9176, #15182). The build-time environment variables were not designed to handle secrets. By lack of other options, people are planning to use them for this. To prevent giving the impression that they are suitable for secrets, it's been decided to deliberately not encrypt those variables in the process.

Member

michaelshobbs commented Jan 12, 2016

Specifically

Build-time environment variables (#9176, #15182). The build-time environment variables were not designed to handle secrets. By lack of other options, people are planning to use them for this. To prevent giving the impression that they are suitable for secrets, it's been decided to deliberately not encrypt those variables in the process.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jan 12, 2016

Member

If we're not going to support this for dockerfile deploys, we should definitely document that. We only do so for buildpack builds because we are following heroku on that path.

Member

josegonzalez commented Jan 12, 2016

If we're not going to support this for dockerfile deploys, we should definitely document that. We only do so for buildpack builds because we are following heroku on that path.

@michaelshobbs

This comment has been minimized.

Show comment
Hide comment
@michaelshobbs

michaelshobbs Jan 12, 2016

Member

Agreed

Member

michaelshobbs commented Jan 12, 2016

Agreed

@craigbeck

This comment has been minimized.

Show comment
Hide comment
@craigbeck

craigbeck Jan 13, 2016

After looking into this some more, the Dockerfile needs to be changed to accept the build args and any build args passed need to be used by the Dockerfile... so it's not feasible to just pass the app's environment variables to docker build because if the Dockerfile doesn't expect them the build fails.

The workaround is to use the docker-options on the build phase to pass the specific args required by the Dockerfile.

craigbeck commented Jan 13, 2016

After looking into this some more, the Dockerfile needs to be changed to accept the build args and any build args passed need to be used by the Dockerfile... so it's not feasible to just pass the app's environment variables to docker build because if the Dockerfile doesn't expect them the build fails.

The workaround is to use the docker-options on the build phase to pass the specific args required by the Dockerfile.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jan 13, 2016

Member

@craigbeck Other than stating that build-env only works for buildpacks deploys, what should I put in the documentation to help other users in your situation?

Member

josegonzalez commented Jan 13, 2016

@craigbeck Other than stating that build-env only works for buildpacks deploys, what should I put in the documentation to help other users in your situation?

@craigbeck

This comment has been minimized.

Show comment
Hide comment
@craigbeck

craigbeck Jan 13, 2016

@josegonzalez thinking about that right now to write something up

craigbeck commented Jan 13, 2016

@josegonzalez thinking about that right now to write something up

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jan 13, 2016

Member

<3 :)

Member

josegonzalez commented Jan 13, 2016

<3 :)

@craigbeck

This comment has been minimized.

Show comment
Hide comment
@craigbeck

craigbeck Jan 13, 2016

I wrote something up about how I solved the problem of build args: http://craigbeck.io/blog/2016/01/13/dokku-plus-dockerfile-deployments/

  • dokku build-options:add myapp build '--build-arg "PATH_PREFIX=/myapp"'
  • modify your dockerfile
FROM baseimage

ARG PATH_PREFIX                            # declare PATH_PREFIX as build arg
RUN PATH_PREFIX=${PATH_PREFIX} ./dobuild   # use PATH_PREFIX (in this case via environment variable)

craigbeck commented Jan 13, 2016

I wrote something up about how I solved the problem of build args: http://craigbeck.io/blog/2016/01/13/dokku-plus-dockerfile-deployments/

  • dokku build-options:add myapp build '--build-arg "PATH_PREFIX=/myapp"'
  • modify your dockerfile
FROM baseimage

ARG PATH_PREFIX                            # declare PATH_PREFIX as build arg
RUN PATH_PREFIX=${PATH_PREFIX} ./dobuild   # use PATH_PREFIX (in this case via environment variable)
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jan 20, 2016

For anyone running into issues passing NODE_ENV as a build-arg...try just passing it as NODE instead, such as:

docker build --build-arg NODE=development --rm -t some/name .

and in Dockerfile:

ARG NODE=production
ENV NODE_ENV ${NODE}
RUN npm start

ghost commented Jan 20, 2016

For anyone running into issues passing NODE_ENV as a build-arg...try just passing it as NODE instead, such as:

docker build --build-arg NODE=development --rm -t some/name .

and in Dockerfile:

ARG NODE=production
ENV NODE_ENV ${NODE}
RUN npm start
@makenosound

This comment has been minimized.

Show comment
Hide comment
@makenosound

makenosound Feb 4, 2016

@josegonzalez This is also something I’d love to see for Dockerfile based deploys now that --build-args is supported in Docker.

@craigbeck Thanks for the post with your workaround so far!

makenosound commented Feb 4, 2016

@josegonzalez This is also something I’d love to see for Dockerfile based deploys now that --build-args is supported in Docker.

@craigbeck Thanks for the post with your workaround so far!

@tobru

This comment has been minimized.

Show comment
Hide comment
@tobru

tobru Feb 16, 2016

Contributor

@craigbeck thanks a lot for your post, it really helps me! But one thing is not yet clear to me: The build/release process I have in mind when having a static HTML5/JS app:

  1. Build docker image for building the static app, containing NodeJS, NPM and Grunt (using the file Dockerfile, which is also used for development). Here I pass the --build-arg "ENVIRONMENT=production" option to configure the static app
  2. Run build process (grunt build:production) in previously built Docker container, mapping the dist/ directory to the host to have access to the build artifacts
  3. Build production image with only Nginx and the static build artifacts with a different Dockerfile Dockerfile.nginx

How are you doing that with Dokku? Seems a bit complicated to me, but probably I have not yet find the correct way. How does your build process look like?

Contributor

tobru commented Feb 16, 2016

@craigbeck thanks a lot for your post, it really helps me! But one thing is not yet clear to me: The build/release process I have in mind when having a static HTML5/JS app:

  1. Build docker image for building the static app, containing NodeJS, NPM and Grunt (using the file Dockerfile, which is also used for development). Here I pass the --build-arg "ENVIRONMENT=production" option to configure the static app
  2. Run build process (grunt build:production) in previously built Docker container, mapping the dist/ directory to the host to have access to the build artifacts
  3. Build production image with only Nginx and the static build artifacts with a different Dockerfile Dockerfile.nginx

How are you doing that with Dokku? Seems a bit complicated to me, but probably I have not yet find the correct way. How does your build process look like?

@craigbeck

This comment has been minimized.

Show comment
Hide comment
@craigbeck

craigbeck Feb 16, 2016

@tobru I have a Docker image that has node + nginx setup to serve my statically built app from /app/dist. Dokku invokes the Dockerfile to build the app then runs the resulting image. Sounds like you are trying to use two images (build container/built app, and nginx)?

craigbeck commented Feb 16, 2016

@tobru I have a Docker image that has node + nginx setup to serve my statically built app from /app/dist. Dokku invokes the Dockerfile to build the app then runs the resulting image. Sounds like you are trying to use two images (build container/built app, and nginx)?

@tobru

This comment has been minimized.

Show comment
Hide comment
@tobru

tobru Feb 17, 2016

Contributor

@craigbeck Yes, that's true. The idea was to only have the needed parts in the resulting Docker image, the statically built HTML5/JS app doesn't need NodeJS to run, only for building it. I want to have only the really needed things in the image: nginx + app. But probably this is not possible with Dokku without modification or plugin?

Contributor

tobru commented Feb 17, 2016

@craigbeck Yes, that's true. The idea was to only have the needed parts in the resulting Docker image, the statically built HTML5/JS app doesn't need NodeJS to run, only for building it. I want to have only the really needed things in the image: nginx + app. But probably this is not possible with Dokku without modification or plugin?

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Feb 17, 2016

Member

I don't think we can set alternative dockerfiles for the dokku run - @michaelshobbs correct me if I am wrong - but it should be possible for you to do this with one Dockerfile and a custom ENTRYPOINT that either runs nginx to serve your stuff or just bails out. I would personally always compress assets upon deployment, but thats just me.

Member

josegonzalez commented Feb 17, 2016

I don't think we can set alternative dockerfiles for the dokku run - @michaelshobbs correct me if I am wrong - but it should be possible for you to do this with one Dockerfile and a custom ENTRYPOINT that either runs nginx to serve your stuff or just bails out. I would personally always compress assets upon deployment, but thats just me.

@michaelshobbs

This comment has been minimized.

Show comment
Hide comment
@michaelshobbs

michaelshobbs Feb 17, 2016

Member

dokku run is definitely bound to the built app image.

@tobru If the concern is having node and other binaries necessary for the building of the app but not the running of the app, why not include some cleanup RUN directive towards the end of your dockerfile that remove the unnecessary bits?

Member

michaelshobbs commented Feb 17, 2016

dokku run is definitely bound to the built app image.

@tobru If the concern is having node and other binaries necessary for the building of the app but not the running of the app, why not include some cleanup RUN directive towards the end of your dockerfile that remove the unnecessary bits?

@tobru

This comment has been minimized.

Show comment
Hide comment
@tobru

tobru Feb 17, 2016

Contributor

@michaelshobbs yeah, this will be probably the way to go. (sorry for hijacking this thread for this particularly unrelated question)

Contributor

tobru commented Feb 17, 2016

@michaelshobbs yeah, this will be probably the way to go. (sorry for hijacking this thread for this particularly unrelated question)

@tobru

This comment has been minimized.

Show comment
Hide comment
@tobru

tobru Feb 24, 2016

Contributor

Just for clarification: the correct command is dokku docker-options:add myapp build '--build-arg "PATH_PREFIX=/myapp"' (docker-options vs. build-options)

Contributor

tobru commented Feb 24, 2016

Just for clarification: the correct command is dokku docker-options:add myapp build '--build-arg "PATH_PREFIX=/myapp"' (docker-options vs. build-options)

josegonzalez added a commit that referenced this issue Mar 18, 2016

Merge pull request #2007 from dokku/josegonzalez-patch-1
Document when configuration variables are available. Closes #1860
@michael-px

This comment has been minimized.

Show comment
Hide comment
@michael-px

michael-px Jun 30, 2016

I tried doing a build with --build-arg VERSION="string" and docker 1.11.2 barfed.

The docs don't say it supported, so did this RFE die or it just hasn't shown up in the docker yet.

Modifying a config file inside a container with the Jenkins build and GIT small SHA1 becomes a major task without being able to pass environment variables to builds.

michael-px commented Jun 30, 2016

I tried doing a build with --build-arg VERSION="string" and docker 1.11.2 barfed.

The docs don't say it supported, so did this RFE die or it just hasn't shown up in the docker yet.

Modifying a config file inside a container with the Jenkins build and GIT small SHA1 becomes a major task without being able to pass environment variables to builds.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jun 30, 2016

Member

@michael-px can you open up a new issue and include all of the information we ask for in our issue template? It is helpful for debugging problems. Thanks.

Member

josegonzalez commented Jun 30, 2016

@michael-px can you open up a new issue and include all of the information we ask for in our issue template? It is helpful for debugging problems. Thanks.

@blocka

This comment has been minimized.

Show comment
Hide comment
@blocka

blocka Oct 21, 2016

For anybody running into this problem:

dokku docker-options:add myapp build '--build-arg "PATH_PREFIX=/myapp"'

was creating a build arg "PATH_PREFIX instead of PATH_PREFIX. This took me a long time to realize.

blocka commented Oct 21, 2016

For anybody running into this problem:

dokku docker-options:add myapp build '--build-arg "PATH_PREFIX=/myapp"'

was creating a build arg "PATH_PREFIX instead of PATH_PREFIX. This took me a long time to realize.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Oct 21, 2016

Member

@blocka do you mind making a pull request for our docker-options documentation to make that more clear?

Member

josegonzalez commented Oct 21, 2016

@blocka do you mind making a pull request for our docker-options documentation to make that more clear?

@diogocaetano

This comment has been minimized.

Show comment
Hide comment
@diogocaetano

diogocaetano May 31, 2017

After hours trying to make it work...
Work for me in this way:

dokku docker-options:add myapp build '--build-arg NODE_ENV=production'

dockerfile

ARG NODE_ENV=production
ENV NODE_ENV ${NODE_ENV}
RUN echo $NODE_ENV

Use ARG for default value even you don't need.

Hopeful that helps someone. #2776

diogocaetano commented May 31, 2017

After hours trying to make it work...
Work for me in this way:

dokku docker-options:add myapp build '--build-arg NODE_ENV=production'

dockerfile

ARG NODE_ENV=production
ENV NODE_ENV ${NODE_ENV}
RUN echo $NODE_ENV

Use ARG for default value even you don't need.

Hopeful that helps someone. #2776

@templth

This comment has been minimized.

Show comment
Hide comment
@templth

templth Jun 6, 2017

@diogocaetano thanks very much for the hint! I tried your approach but it doesn't work for me :-( My dokku version is 0.9.4 and docker 17.05.0-ce. The default value I specified isn't replaced by the value configured using docker-options.

Do you know if it's possible to have the list of set variables? I try to find out a way to debug this problem... Thanks!

templth commented Jun 6, 2017

@diogocaetano thanks very much for the hint! I tried your approach but it doesn't work for me :-( My dokku version is 0.9.4 and docker 17.05.0-ce. The default value I specified isn't replaced by the value configured using docker-options.

Do you know if it's possible to have the list of set variables? I try to find out a way to debug this problem... Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment