Docker build args sourced from Shell environment variables

[flyinprogrammer]

So we'd like to have copilot source environment variables from the shell that copilot is run in as build arguments into the docker build command. And it either already does this, and I am dumb, or it does not.

In my mind some form of the following might work:

manifest.yml:

image:
  build:
    dockerfile: path/to/dockerfile
    context: context/dir
    args:
      SECRET_PASSWORD: {{ .Env.SECRET_PASSWORD_IN_SHELL_ENV }}

In my shell:

export SECRET_PASSWORD_IN_SHELL_ENV="ssshthisissecret"
copilot deploy
# at some point this would happen:  docker build -f Dockerfile --build-arg SECRET_PASSWORD=ssshthisissecret

I thought maybe, perhaps, we were clever and ran the manifest file through go template before shoving it into a struct, but i'm either stupid, or we aren't doing this? Happy to provide a PR and shame on me for not running this in a debugger before opening this issue, but alas here we are.

[bvtujo]

Hey @flyinprogrammer, you're right that Copilot doesn't do this currently. This is a super interesting use case though, so thank you so much for bringing it to our attention.

Also, you're right that we run docker build under the hood and it's not great to require that you expose your secrets in the manifest if you have to pass them in as build args.

Depending on your use case, though I can think of a couple of workarounds of varying complexity that will allow you to keep using Copilot for infrastructure management 😄

If you need secrets inside your deployed container:

We recommend you use SSM and the secrets section of the manifest to specify credentials that you need to access from inside your running containers.

If you need secrets from the terminal environment as build arguments

You can run docker build manually using whatever environment variables suit you, then use the image.location field of the manifest to specify an external repository (Dockerhub or ECR, both work 😄 ) from which to grab the built secret-containing image.

HUGE CAVEAT that if you do this, be really careful not to specify any secrets in the variables section of the manifest as they'll be exposed as plaintext in the Environment Variables section of the Container Definition inside the Task Definition on the AWS console.

Do either of those work for you?

[flyinprogrammer]

This is a super interesting use case though, so thank you so much for bringing it to our attention.

Yah, the primary use case is that we have private python/java repos that tools like pip and gradle need access to at build time, as we build our code in the container. Building our image outside of Copilot seems reasonable for now.

image.location

So this would require us to manage the version in the manifest.yaml file - which is fine, but it still might be nice to have a way to make that dynamic since most of our use cases already have CI tools providing Git hashes and Tag names as environment variables. At some point it might be nice to have copilot acknowledge these ideas as Default Environment variables, but perhaps it's out of scope.

I'll close this for now as your backlog of issues will either force this requirement naturally, or it will be able to be delayed indefinitely, which is no problem either way.

[aaroncarlucci]

I would also like to see this feature.

[perevernihata]

bump on this one, things like nextjs require build time env vars, hence deploying from local using copilot deploy becomes impossible

[zackdotcomputer]

Same here and could love to see this issue reopened @flyinprogrammer since, backlog or no, this is a real issue for anyone who needs a secret build arg (such as an access token to a private package repo).

[tsingh2k15]

same here, I was looking to be able to pass ARTIFACT_VERSION (not a secret) as build args. The point here is that ARTIFACT_VERSION has to be read from a source of truth and could not be hardcoded in the manifest file. I am looking to be able to specify something like below:

image:
  build:
    dockerfile: path/to/dockerfile
    context: context/dir
    args:
      ARTIFACT_VERSION: {{ get_artifact_version() }} // invocation to a specific script -  shell, python, groovy...

Could this be supported? or is it already supported?

[iamhopaul123]

Hello @tsingh2k15 it is not supported yet. Reopen this issue as it gains more popularity.

[tsingh2k15]

@iamhopaul123 thank you for considering this!

On another note, it would also be easier if copilot svc deploy could allow me to pass build-args. What do you think about that as another alternative? Specifying args in the manifest file and having the values assigned programmatically might not be always that easy for every user.

[iamhopaul123]

@tsingh2k15 Good question. Normally we don't want to spam our flag set by adding any field that's already available in our manifest. However, if there does exist loads of users wanting to add the flag we'll reconsider about it!

[iamhopaul123]

Hello friends! We have a proposal for supporting manifest env vars in #2912. Feel free to check it out and leave your thought on it. Appreciate for your input on this features 🙇