Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ECS] Need Host environment variable resolution to pass some information to a container #369

Open
owaaa opened this issue Jan 6, 2015 · 108 comments
Assignees
Labels

Comments

@owaaa
Copy link

@owaaa owaaa commented Jan 6, 2015

Before moving to ECS, I need to be able to make sure that I can run consul in that environment. To run consul, I need to pass some Host environment variables as environment variables to the container. I would normally do this as -e HOST=$HOST etc. However, it seems that that if I define a variable with $HOST in a task definition, this is treated as a static string.

Related, I also typically would set docker command line options on the run command for several features I need. I may be able to get around some of this by overriding options in the docker daemon itself, but as far as passing host information I am not aware of any workaround.

@euank
Copy link

@euank euank commented Jan 6, 2015

I'd appreciate a few clarifications around what you need here.

When you mention passing in host environment variables, what are you thinking about passing in and how are they getting set on the host? Do you just need e.g. the hostname or ip, or do you want to access arbitrary host environment variables?

Also, if you could pass in environment variables as container overrides when you call start-task, would that help?

As it is, you can access the EC2 Metadata Service from within your containers to get the IP, Instance ID, and other information.

I'd like to understand the use-case a little better and any other possible solutions.

Best,
Euan

@owaaa
Copy link
Author

@owaaa owaaa commented Jan 6, 2015

Thank you for the quick reply!

My end goal is to get several docker "side cars" working with ECS (service discovery, log shipping, etc). Some of these containers expect some host information to be passed in at run time, like the host ip, hostname, etc. I think currently most or all of the items I need are host specific. Currently all our deployments are automated via CloudFormation and the environment variables are set during cloud-init which also launches the containers. You raise a good point about the ec2 metadata service. I could probably fork some of these projects to use it, the only drawback being the containers are more tied to AWS so harder to run offline, etc.

The other option I was just considering is to load these containers which require special config on each of the ECS hosts I create via Cloud Formation via current process. ECS would then only manage my other containers. I am wondering though if that would muck with the CPU and Memory allocation available to tasks?

I have a related question in that I have many containers that need to pass docker run arguments such as --dns -h $HOST --restart=always etc. I think some of these I can set in the docker daemon startup options in /etc/syconfig/ but some can't be set that way. Are you guys thinking about how docker run options like these might be used or specified?

Thanks,

Andrew

@euank
Copy link

@euank euank commented Jan 14, 2015

Sorry for the slow response on this; passing through host environment variables is a tricky subject.

First of all, you have a few other questions that are easier to address.

  1. CPU / Memory allocation available:

Yes, running things on your instance that ECS is unaware of would cause issues there, but we want to support that use-case. Being able to override the resources ECS can use at launch makes a lot of sense to me. Good question.

  1. Docker run arguments

You're correct that anything that can be passed to the docker daemon (such as the ones you reference) can be edited by using the following user-data:

#cloud-boothook
#!/bin/sh
echo 'OPTIONS="--dns=8.8.8.8"' > /etc/sysconfig/docker

If there are any specific docker run features you want that are not covered by our task definition, we'd be interested in hearing about them.

  1. Host environment variables

To clarify a little further, when you say your environment variables are "set with cloud init" you mean you've put the desired data in /etc/environment or similar, correct?

Would being able to just specify the equivilant of docker's "--env-file" as part of a container definition fit your use case, or would that be insufficient for some reason?

@radenui
Copy link

@radenui radenui commented Mar 3, 2015

Hello,

I re-lauch the thread as I'm interested too, and I believe my use-case is close to the one described here.

In the task definition, I can statically set anything related to the environment of the running task (domain, ...), but I also need to run my docker container with arguments depending on the host on which it is launched (host IP address, AWS instance Id, ...).

My docker run command would be something like:

docker run -d -p=80:80 -e HOST=$HOST myrepo/myimage java -Dmy.local.ip=$(curl 169.254.169.254/latest/meta-data/local-ipv4 2> /dev/null) -jar /path/to/my/jarfile.jar

And would correspond to:

docker run -d -p=80:80 -e HOST=myhostname myrepo/myimage java -Dmy.local.ip=10.0.0.5 -jar /path/to/my/jarfile.jar

Is there a way to reproduce this kind of behaviour with a task definition ?

Thanks,

Arthur

@euank
Copy link

@euank euank commented Mar 3, 2015

@radenui Currently there's not a direct way to reproduce that with a task definition.

However, if you change your image to have an entrypoint file where you resolve the above, you can get similar behavior.
For example, such an entrypoint might look like:

#!/bin/sh
export HOST=$(curl -s 169.254.169.254/latest/meta-data/local-hostname)
export LOCAL_IP=$(curl -s 169.254.169.254/latest/meta-data/local-ipv4)
exec "$@"

With the above entrypoint, you can reference those environment variables in your command.

I'm still leaving this issue open since I think the above is probably not the right long-term solution (especially for common things like instance ip).

Best,
Euan

@owaaa
Copy link
Author

@owaaa owaaa commented May 28, 2015

I had to table this temporarily but the end goal is I need consul and registrator working in ECS if I'm going to migrate our services. The consul / registrator docker containers require the $HOST and $HOSTNAME to be passed in to correctly register services and advertise the host machine IP. I was trying to avoid forking the Docker container with a solution like the entrypoint as that ties the container to AWS.

The other original blocker is on my own containers I need to set the DNS to the docker bridge via the --dns flag. I think #2 might work for that if I run the ECS agent on my own tailored machined with that setting.

@johndvorakHR
Copy link

@johndvorakHR johndvorakHR commented Jun 4, 2015

+1

@tj
Copy link

@tj tj commented Jun 5, 2015

would love to see this as well, along with most of the docker-run flags, necessary for logging & metrics unless you want to run those via ansible/terraform etc

@dgalichet
Copy link

@dgalichet dgalichet commented Jun 11, 2015

+1

having HOST and LOCAL_IP directly defined would be much appreciated. My other option would have to link directly to the container that I wan't to access, but it's not actually possible to link to a container not defined in a Task (https://forums.aws.amazon.com/thread.jspa?threadID=179653&tstart=0)

@pschorf
Copy link

@pschorf pschorf commented Jun 25, 2015

I'm also interested in this to pass database connection information

@juliencruz
Copy link

@juliencruz juliencruz commented Jul 1, 2015

+1

Also interested in this feature as it fills in some gaps for auto-registration of containers in our environment as well.

@rainulf
Copy link

@rainulf rainulf commented Jul 8, 2015

+1 👍

I would like to have this feature as well.

@pikeas
Copy link

@pikeas pikeas commented Jul 17, 2015

+1, same use case as @owaaa - I want to use consul + registrator, so I need a clean way to pass host-specific config into a container.

@nexus49
Copy link

@nexus49 nexus49 commented Jul 17, 2015

+1
Being able to pass in the host or other dynamic values specific to the host is valuable either through a environment variable or possibly through a CMD statement.

Specifically when using third party containers where you don't want to create your own version of.

@vsudilov
Copy link

@vsudilov vsudilov commented Jul 30, 2015

+1; My use case is also described above

@ccit-spence
Copy link

@ccit-spence ccit-spence commented Aug 10, 2015

@euank I have been trying your thought on the entrypoint and can't get your bash script to work. Do you have a working example?

@euank
Copy link

@euank euank commented Aug 13, 2015

@ccit-spence I put together a short example that worked for me which I've posted in this gist.

@iainlbc
Copy link

@iainlbc iainlbc commented Aug 17, 2015

+1, consul & registrator use case as well!

@tdensmore
Copy link

@tdensmore tdensmore commented Aug 17, 2015

+1. Need support for docker-run flags.ECS not supporting Docker defaults is puzzling (at best).

Why has this not been implemented yet? On March 3 the proposed workaround to ECS shortcomings was identified as "probably not the right long-term solution (especially for common things like instance ip)". Five months have passed...

@scatterbrain
Copy link

@scatterbrain scatterbrain commented Sep 18, 2015

+1

2 similar comments
@bs-thomas
Copy link

@bs-thomas bs-thomas commented Oct 4, 2015

+1

@Lakret
Copy link

@Lakret Lakret commented Oct 29, 2015

+1

@urog
Copy link

@urog urog commented Oct 29, 2015

+1 to being able to execute arbitrary shell to define values for vars

edit: on the host node passing this to the container.

@filosganga
Copy link

@filosganga filosganga commented Oct 29, 2015

@urog Not all the images have a shell.

@tdensmore
Copy link

@tdensmore tdensmore commented Oct 29, 2015

This issue has been open for close to 10 months. Does anyone know if this issue is getting any official attention (other than the +1 support from the audience) ?

@samuelkarp
Copy link
Member

@samuelkarp samuelkarp commented Oct 29, 2015

@tdensmore The +1s and use-case details are very helpful for us.

@brentonannan
Copy link

@brentonannan brentonannan commented Oct 30, 2015

+1 - need to be able to advertise consul as running on the host IP.

@sebest
Copy link

@sebest sebest commented Oct 30, 2015

need HOST_NAME and HOST_IP

I think that being able to pass informations available in EC2 metadata (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) as environment variable would solve most of the use case.

Maybe extending the JSON for the task definition currently it is like this:

"environment" : [
    { "name" : "HOST_NAME", "value" : "myhost" },
    { "name" : "HOST_IP", "value" : "42.42.42.42" }
]

adding support for a syntax like that:

"environment" : [
    { "name" : "HOST_NAME", "metadata" : "hostname" },
    { "name" : "HOST_IP", "metadata" : "local-ipv4" }
]

this would get the values from http://169.254.169.254/latest/meta-data/hostname and http://169.254.169.254/latest/meta-data/local-ipv4

@kyriesent
Copy link

@kyriesent kyriesent commented Nov 5, 2015

+1 would love to see this. Also looking to use consul and registrator and advertise the host IP

@thomaspeitz
Copy link

@thomaspeitz thomaspeitz commented Nov 11, 2015

+1 need this for consul (have to advertise the internal ip of the docker host to other clients, otherwise consul will announce the internal docker container ip e.g. 172..) 👍

@rjosephwright
Copy link

@rjosephwright rjosephwright commented Sep 19, 2017

@danielhanold it isn't necessary unless you need information passed in that is unique to the underlying host, such as its IP address.

Example using CloudFormation (pardon if the syntax is not 100% correct):

Parameters:
  NodeEnv:
    Type: String
    AllowedValues:
      - QA
      - Staging
      - Production

Resources:
  Type: AWS::ECS::TaskDefinition
  Properties:
    ContainerDefinitions:
      - Image: my-image:v1.0
        Environment:
          - Name: NODE_ENV
            Value: !Sub NodeEnv
@danielhanold
Copy link

@danielhanold danielhanold commented Sep 19, 2017

@rjosephwright I'm currently using Travis to build and upload the Docker image to ECR, then generate an updated JSON representation of the task definition, followed by an AWS CLI call to update a specific CloudFormation stack using the new task definition.

I don't know why I never thought of defining the task definition in CloudFormation itself the way you described, which sounds like a much cleaner way to manage & update containers with CFN. Thanks for the great suggestion.

@simonslater
Copy link

@simonslater simonslater commented Sep 21, 2017

@danielhanold what I like about your solution is that it allows for a task definition to be used by multiple enviroments.

@adamrbennett
Copy link

@adamrbennett adamrbennett commented Oct 3, 2017

Here's my use-case:
Create a task definition and run it in dev stack.
Test, iterate, etc.
When ready to promote, reuse the same task definition in prod stack.

It seems this is the intended approach, since task definitions are not cluster-specific. Furthermore, the task definition includes run-time properties, and contains things which may affect run-time behavior. It makes the most sense to promote the run-time definition, rather than just the build-time definition (i.e.: the Docker image) and then re-create, or try to match, the run-time definition with what was tested in dev.

In order to achieve this, we must be able to override the container's environment variables depending on the stack it is deployed to. This would be simple if we could use host environment variables to set values on container environment variables.

EDIT: I suppose the most elegant solution is to actually be able to override container environment variables in the service definition (which is cluster-specific), much like you can when running an ad-hoc task.

@shailesh2088
Copy link

@shailesh2088 shailesh2088 commented Oct 19, 2017

+1
This is possible when you execute docker run on bash. I don't see why it's not possible using task definitions.

@zfletcher
Copy link

@zfletcher zfletcher commented Nov 7, 2017

New, from AWS... https://aws.amazon.com/about-aws/whats-new/2017/11/amazon-ecs-allows-containers-to-directly-access-environmental-metadata/

Now, applications running in containers managed by Amazon ECS can directly query environmental metadata such as the Docker container and Docker image name and id, the container’s networking configuration and port mapping, as well as the ECS Task’s and Container Instance Amazon Resource Name (ARN). This makes applications running on Amazon ECS environment-aware, and enables them to self-register in Service Discovery and Configuration Management solutions.

ECS_CONTAINER_METADATA_FILE=/opt/ecs/metadata/e6e6d129-b432-43aa-bbf5-3dc2498142r0/ecs-container-metadata.json

👍

@nmeyerhans
Copy link

@nmeyerhans nmeyerhans commented Nov 7, 2017

Thanks @zfletcher for calling that out here. Hopefully this container metadata API addresses most of the use cases captured in this issue. You had previously called out dynamically mapped host ports as one particularly interesting to you. We do support that, but be aware of #1052 for now.

I'd love to resolve this issue at this point, but I don't want to do so if there are still scenarios that aren't addressed by the container metadata feature. This issue has been open for a long time and quite a few people have contributed to it. If you've been waiting for this functionality, please give it a try and let us know if we've addressed your needs. If we haven't, please provide details. (Note that we'd also be happy to have people open new issues; it might help us keep better track of the various requests.)

@danielhanold
Copy link

@danielhanold danielhanold commented Nov 9, 2017

@zfletcher Thanks for adding this update, but @nmeyerhans: I don't think this address or resolves the problem raised in this issue. @adamrbennett describes (what I believe to be) the most common use case that is currently not possible with ECS:

Create a task definition and run it in dev stack.
Test, iterate, etc.
When ready to promote, reuse the same task definition in prod stack.

The issue is pretty close to the actual title of this issue: "Need Host environment variable resolution to pass some information to a container". Here's my use case, which is not solved by this recent addition:

  • CFN Stack A that creates an ECS cluster with 1 ECS instance and sets an environment variable ENV_NAME=DEV on the ECS instance
  • CFN Stack B that creates an ECS cluster with 1 ECS instance and sets an environment variable ENV_NAME=PROD on the ECS instance
  • Task Definition X that is running a container on Stack A, which uses ENV_NAME from the ECS Cluster created by Stack A => ENV_NAME = DEV
  • Once tested successfully, the very same task definition X is then promoted to Stack B and the container uses the ENV_NAME from the ECS Cluster created by Stack B => ENV_NAME = PROD

Other than this use case, it's frustrating that we can't use variables for the container environment variables in the task definition and are limited to the hardcoded values set in a task definition.

@SilverGhostBS
Copy link

@SilverGhostBS SilverGhostBS commented Nov 10, 2017

@danielhanold Thanks, that's EXACTLY my use case.

What's more frustrating is that this is allow through command line on the vanilla docker, just put -e VAR_NAME and it will passthrough the env var from host to container.

But the task definition schema absolutely insists on us setting a value for the parameter, whereas just allowing to omit it might work.

@pikeas
Copy link

@pikeas pikeas commented Dec 7, 2017

Agree with @danielhanold!

It doesn't make sense to have separate task definitions for dev/staging/prod. I'd like to define my needed env vars once in the task definition, then populate values in each environment's service definition.

@simonslater
Copy link

@simonslater simonslater commented Dec 7, 2017

@mceg
Copy link

@mceg mceg commented Dec 15, 2017

The workaround that I am using:

{
    "name": "kafka",
    "image": "confluentinc/cp-kafka:4.0.0",
    "essential": true,
    "memoryReservation": 2048,
    "command": [
      "/bin/bash",
      "-c",
      "export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://$(hostname -i):9092; /etc/confluent/docker/run"
    ],
    "environment": [
      {
        "name": "KAFKA_ZOOKEEPER_CONNECT",
        "value": "..."
      }
    ],
...
}

a little modification of entrypoint approach but without the need to manage your own containers, and without running containers from user data scripts, just registering them as services as it should be.

@akrymets
Copy link

@akrymets akrymets commented Mar 6, 2018

If you're using the official consul image then its entrypoint.sh file allows you to specify CONSUL_BIND_INTERFACE env variable and if it's set the script will determine the ip in it and substitute this ip to the "-bind" parameter of consul startup. If you set this env variable, say, to "eth0" then your consul agent will be automatically configured with the required ipv4 address belonging to eth0 during its startup.
Somebody has already mentioned this solution here.

@pb0101
Copy link

@pb0101 pb0101 commented Mar 16, 2018

The real problem comes when you are using third part docker images that you can't modify and add another ENTRYPOINT.
Is there any other solution besides modifying ENTRYPOINT

@zfletcher
Copy link

@zfletcher zfletcher commented Mar 16, 2018

You can usually do docker run --entrypoint "/bin/bash" ... or similar.

@rash805115
Copy link

@rash805115 rash805115 commented Nov 11, 2018

I faced this problem today and after hours of research I found this thread. Wow, the discussion here is very informative and it helped me find a solution for myself.

As suggested in this thread I modified my Dockerfile to execute a script at entrypoint and in that script I made the call to metadata service to get LOCAL_IP.
Please refer to my StackOverflow post for more details.

Dockerfile

FROM gliderlabs/registrator:v7

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh && \
    apk update && \
    apk add curl

ENTRYPOINT ["/entrypoint.sh"]

entrypoint.sh

#!/bin/sh

LOCAL_IP=$(curl -s --connect-timeout 3 169.254.169.254/latest/meta-data/local-ipv4)
if [ $? -eq 0 ]; then
  /bin/registrator -ip $LOCAL_IP $@
else
  /bin/registrator $@
fi

I hope this issue is fixed the right way, but until then this solution would suffice for me.

@chaitan94
Copy link

@chaitan94 chaitan94 commented Nov 15, 2018

The entrypoint solution suggested by multiple people here definitely works, but really just feels like a hacky way.

This issue has been open for a long time now, is there any specific reason why there has been no elegant solution for this so far? I'm really curious to know what are the hurdles here. I'll be even glad to contribute myself if someone can explain what's going wrong here.

@mfortin
Copy link

@mfortin mfortin commented Nov 22, 2018

Can't the labels be leveraged ? The agent can define them, and they can be retrieved using the metadata URI. I am looking at container instance tags to retrieve the information somehow, haven't found a clean solution yet.

@stefanjauker
Copy link

@stefanjauker stefanjauker commented Dec 29, 2018

+1

@varxy20k
Copy link

@varxy20k varxy20k commented Apr 4, 2019

Just want to add my 2 cents here as this is something that would be really useful for us if it worked the same way it works in docker: -e ENV_VAR without value uses the value from the host. Would it work if we could set environment variables without values in the task definition?

@bvanderveen
Copy link

@bvanderveen bvanderveen commented Apr 12, 2019

+1.

What I want to do is to build my set of containers, build a task def that orchestrates that set of containers, and then launch that taskdef into a cluster with different environment variables (say, to distinguish between dev/prod).

Seems my only choices are to a) maintain separate dev/prod task defs, where the only difference is some env var that indicates dev vs. prod or b) bake the fact of being dev or prod into the containers (which seems completely absurd).

To be honest, I feel like I've been sold a bill of goods, but I'm far enough into the finger trap and Stockholm syndrome that I have no choice but to press on with my 3 weeks of sunk cost and my dreams for a better world.

And this is after a buddy of mine took his startup's infrastructure down by deploying during a scale-out event two days before getting a Letter of Intent from a megacorp learning "the hard way" that ECS Blue/Green deploys don't support Auto-Scaling.

Guys. This is borderline bad-faith.

@coultn coultn transferred this issue from aws/amazon-ecs-agent Jun 12, 2019
@coultn coultn changed the title Need Host environment variable resolution to pass some information to a container [ECS] Need Host environment variable resolution to pass some information to a container Jun 12, 2019
@coultn
Copy link

@coultn coultn commented Jun 12, 2019

@bvanderveen would #193 address your use case? In other words, suppose you could define environment variables on a per-cluster basis; every task run in that cluster would use those environment variables (overriding task definition env variables if necessary).

@coultn
Copy link

@coultn coultn commented Jun 13, 2019

The entrypoint solution suggested by multiple people here definitely works, but really just feels like a hacky way.

This issue has been open for a long time now, is there any specific reason why there has been no elegant solution for this so far? I'm really curious to know what are the hurdles here. I'll be even glad to contribute myself if someone can explain what's going wrong here.

For all those who have been requesting this, one of the main obstacles to providing the IP address in an environment variable is that, with EC2 networking, the IP address of an EC2 instance (or ENI attached to a task) can change. Therefore, the value of the IP address when the task started is not necessarily correct at all times.

@DavidMorton
Copy link

@DavidMorton DavidMorton commented Sep 3, 2019

sh -c 'aws ecs describe-clusters --cluster $(curl -s $ECS_CONTAINER_METADATA_URI | jq -r ".Labels[\"com.amazonaws.ecs.cluster\"]") --include TAGS | jq -r ".clusters[0].tags" | jq -r "map([.key,.value] | join(\"=\")) | join(\"\n\")"'

Okay, folks, I know this is CRAZY long for a thing... but basically, it'll go and get the tags from the cluster and output them in
KEY=VALUE
format, one line at a time. This could then be used to set the environment variables for your application before it starts running, based on the tags that are on the cluster.

How it works:

The ECS_CONTAINER_METADATA_URI environment variables returns back a URL that gives some JSON back. You can read more about it here. After that, it's a simple matter of using jq (you'll have to include it in your container) to fetch the cluster information from AWS, and trim down to just the tags on the cluster.

Prerequisites:

  1. You have to have jq installed on your instance, to parse things out.
  2. Your aws cli should be logged in and have the proper policies attached.

Hope this helps people who want to avoid adding volumes. I fully agree with anyone who might say this is a hack, but sometimes, you just gotta figure it out, in the absence of anything better, which AWS doesn't seem to be providing anytime soon.

@srrengar srrengar self-assigned this Apr 1, 2020
@mkozjak
Copy link

@mkozjak mkozjak commented Oct 8, 2020

@srrengar - What is the status on this one? We need this feature to pass values generated from one container that's needed for the next one to run in order to use it as a variable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.