[ECS] Need Host environment variable resolution to pass some information to a container

[owaaa]

Before moving to ECS, I need to be able to make sure that I can run consul in that environment. To run consul, I need to pass some Host environment variables as environment variables to the container. I would normally do this as -e HOST=$HOST etc. However, it seems that that if I define a variable with $HOST in a task definition, this is treated as a static string.

Related, I also typically would set docker command line options on the run command for several features I need. I may be able to get around some of this by overriding options in the docker daemon itself, but as far as passing host information I am not aware of any workaround.

[euank]

I'd appreciate a few clarifications around what you need here.

When you mention passing in host environment variables, what are you thinking about passing in and how are they getting set on the host? Do you just need e.g. the hostname or ip, or do you want to access arbitrary host environment variables?

Also, if you could pass in environment variables as container overrides when you call start-task, would that help?

As it is, you can access the EC2 Metadata Service from within your containers to get the IP, Instance ID, and other information.

I'd like to understand the use-case a little better and any other possible solutions.

Best,
Euan

[owaaa]

Thank you for the quick reply!

My end goal is to get several docker "side cars" working with ECS (service discovery, log shipping, etc). Some of these containers expect some host information to be passed in at run time, like the host ip, hostname, etc. I think currently most or all of the items I need are host specific. Currently all our deployments are automated via CloudFormation and the environment variables are set during cloud-init which also launches the containers. You raise a good point about the ec2 metadata service. I could probably fork some of these projects to use it, the only drawback being the containers are more tied to AWS so harder to run offline, etc.

The other option I was just considering is to load these containers which require special config on each of the ECS hosts I create via Cloud Formation via current process. ECS would then only manage my other containers. I am wondering though if that would muck with the CPU and Memory allocation available to tasks?

I have a related question in that I have many containers that need to pass docker run arguments such as --dns -h $HOST --restart=always etc. I think some of these I can set in the docker daemon startup options in /etc/syconfig/ but some can't be set that way. Are you guys thinking about how docker run options like these might be used or specified?

Thanks,

Andrew

[euank]

Sorry for the slow response on this; passing through host environment variables is a tricky subject.

First of all, you have a few other questions that are easier to address.

1. CPU / Memory allocation available:

Yes, running things on your instance that ECS is unaware of would cause issues there, but we want to support that use-case. Being able to override the resources ECS can use at launch makes a lot of sense to me. Good question.

2. Docker run arguments

You're correct that anything that can be passed to the docker daemon (such as the ones you reference) can be edited by using the following user-data:

#cloud-boothook
#!/bin/sh
echo 'OPTIONS="--dns=8.8.8.8"' > /etc/sysconfig/docker

If there are any specific docker run features you want that are not covered by our task definition, we'd be interested in hearing about them.

3. Host environment variables

To clarify a little further, when you say your environment variables are "set with cloud init" you mean you've put the desired data in /etc/environment or similar, correct?

Would being able to just specify the equivilant of docker's "--env-file" as part of a container definition fit your use case, or would that be insufficient for some reason?

[radenui]

Hello,

I re-lauch the thread as I'm interested too, and I believe my use-case is close to the one described here.

In the task definition, I can statically set anything related to the environment of the running task (domain, ...), but I also need to run my docker container with arguments depending on the host on which it is launched (host IP address, AWS instance Id, ...).

My docker run command would be something like:

docker run -d -p=80:80 -e HOST=$HOST myrepo/myimage java -Dmy.local.ip=$(curl 169.254.169.254/latest/meta-data/local-ipv4 2> /dev/null) -jar /path/to/my/jarfile.jar

And would correspond to:

docker run -d -p=80:80 -e HOST=myhostname myrepo/myimage java -Dmy.local.ip=10.0.0.5 -jar /path/to/my/jarfile.jar

Is there a way to reproduce this kind of behaviour with a task definition ?

Thanks,

Arthur

[euank]

@radenui Currently there's not a direct way to reproduce that with a task definition.

However, if you change your image to have an entrypoint file where you resolve the above, you can get similar behavior.
For example, such an entrypoint might look like:

#!/bin/sh
export HOST=$(curl -s 169.254.169.254/latest/meta-data/local-hostname)
export LOCAL_IP=$(curl -s 169.254.169.254/latest/meta-data/local-ipv4)
exec "$@"

With the above entrypoint, you can reference those environment variables in your command.

I'm still leaving this issue open since I think the above is probably not the right long-term solution (especially for common things like instance ip).

Best,
Euan

[owaaa]

I had to table this temporarily but the end goal is I need consul and registrator working in ECS if I'm going to migrate our services. The consul / registrator docker containers require the $HOST and $HOSTNAME to be passed in to correctly register services and advertise the host machine IP. I was trying to avoid forking the Docker container with a solution like the entrypoint as that ties the container to AWS.

The other original blocker is on my own containers I need to set the DNS to the docker bridge via the --dns flag. I think #2 might work for that if I run the ECS agent on my own tailored machined with that setting.

[johndvorakHR]

+1

[tj]

would love to see this as well, along with most of the docker-run flags, necessary for logging & metrics unless you want to run those via ansible/terraform etc

[dgalichet]

+1

having HOST and LOCAL_IP directly defined would be much appreciated. My other option would have to link directly to the container that I wan't to access, but it's not actually possible to link to a container not defined in a Task (https://forums.aws.amazon.com/thread.jspa?threadID=179653&tstart=0)

[pschorf]

I'm also interested in this to pass database connection information

[juliencruz]

+1

Also interested in this feature as it fills in some gaps for auto-registration of containers in our environment as well.

[rainulf]

+1 👍

I would like to have this feature as well.

[pikeas]

+1, same use case as @owaaa - I want to use consul + registrator, so I need a clean way to pass host-specific config into a container.

[nexus49]

+1
Being able to pass in the host or other dynamic values specific to the host is valuable either through a environment variable or possibly through a CMD statement.

Specifically when using third party containers where you don't want to create your own version of.

[vsudilov]

+1; My use case is also described above

[ccit-spence]

@euank I have been trying your thought on the entrypoint and can't get your bash script to work. Do you have a working example?

[euank]

@ccit-spence I put together a short example that worked for me which I've posted in this gist.

[iainlbc]

+1, consul & registrator use case as well!

[tdensmore]

+1. Need support for docker-run flags.ECS not supporting Docker defaults is puzzling (at best).

Why has this not been implemented yet? On March 3 the proposed workaround to ECS shortcomings was identified as "probably not the right long-term solution (especially for common things like instance ip)". Five months have passed...

[scatterbrain]

+1

[bs-thomas]

+1

[Lakret]

+1

[urog]

+1 to being able to execute arbitrary shell to define values for vars

edit: on the host node passing this to the container.

[filosganga]

@urog Not all the images have a shell.

[tdensmore]

This issue has been open for close to 10 months. Does anyone know if this issue is getting any official attention (other than the +1 support from the audience) ?

[samuelkarp]

@tdensmore The +1s and use-case details are very helpful for us.

[brentonannan]

+1 - need to be able to advertise consul as running on the host IP.

[sebest]

need HOST_NAME and HOST_IP

I think that being able to pass informations available in EC2 metadata (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) as environment variable would solve most of the use case.

Maybe extending the JSON for the task definition currently it is like this:

"environment" : [
    { "name" : "HOST_NAME", "value" : "myhost" },
    { "name" : "HOST_IP", "value" : "42.42.42.42" }
]

adding support for a syntax like that:

"environment" : [
    { "name" : "HOST_NAME", "metadata" : "hostname" },
    { "name" : "HOST_IP", "metadata" : "local-ipv4" }
]

this would get the values from http://169.254.169.254/latest/meta-data/hostname and http://169.254.169.254/latest/meta-data/local-ipv4

[kyriesent]

+1 would love to see this. Also looking to use consul and registrator and advertise the host IP

[thomaspeitz]

+1 need this for consul (have to advertise the internal ip of the docker host to other clients, otherwise consul will announce the internal docker container ip e.g. 172..) 👍

[danielhanold]

@zfletcher Thanks for adding this update, but @nmeyerhans: I don't think this address or resolves the problem raised in this issue. @adamrbennett describes (what I believe to be) the most common use case that is currently not possible with ECS:

Create a task definition and run it in dev stack.
Test, iterate, etc.
When ready to promote, reuse the same task definition in prod stack.

The issue is pretty close to the actual title of this issue: "Need Host environment variable resolution to pass some information to a container". Here's my use case, which is not solved by this recent addition:

• CFN Stack A that creates an ECS cluster with 1 ECS instance and sets an environment variable ENV_NAME=DEV on the ECS instance
• CFN Stack B that creates an ECS cluster with 1 ECS instance and sets an environment variable ENV_NAME=PROD on the ECS instance
• Task Definition X that is running a container on Stack A, which uses ENV_NAME from the ECS Cluster created by Stack A => ENV_NAME = DEV
• Once tested successfully, the very same task definition X is then promoted to Stack B and the container uses the ENV_NAME from the ECS Cluster created by Stack B => ENV_NAME = PROD

Other than this use case, it's frustrating that we can't use variables for the container environment variables in the task definition and are limited to the hardcoded values set in a task definition.

[SilverGhostBS]

@danielhanold Thanks, that's EXACTLY my use case.

What's more frustrating is that this is allow through command line on the vanilla docker, just put -e VAR_NAME and it will passthrough the env var from host to container.

But the task definition schema absolutely insists on us setting a value for the parameter, whereas just allowing to omit it might work.

[pikeas]

Agree with @danielhanold!

It doesn't make sense to have separate task definitions for dev/staging/prod. I'd like to define my needed env vars once in the task definition, then populate values in each environment's service definition.

[simonslater]

I agree with @danielhanild too!

…

On Wed, Dec 6, 2017 at 7:46 PM Aris Pikeas ***@***.***> wrote: Agree with @danielhanold <https://github.com/danielhanold>! It doesn't make sense to have separate task definitions for dev/staging/prod. I'd like to define my needed env vars once in the task definition, then populate values in each environment's service definition. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/aws/amazon-ecs-agent/issues/3#issuecomment-349824092>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGE0db1XrZRy3v7FZPgnUGTi-9818rJlks5s9zVbgaJpZM4DO_hG> .

[mceg]

The workaround that I am using:

{
    "name": "kafka",
    "image": "confluentinc/cp-kafka:4.0.0",
    "essential": true,
    "memoryReservation": 2048,
    "command": [
      "/bin/bash",
      "-c",
      "export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://$(hostname -i):9092; /etc/confluent/docker/run"
    ],
    "environment": [
      {
        "name": "KAFKA_ZOOKEEPER_CONNECT",
        "value": "..."
      }
    ],
...
}

a little modification of entrypoint approach but without the need to manage your own containers, and without running containers from user data scripts, just registering them as services as it should be.

[akrymets]

If you're using the official consul image then its entrypoint.sh file allows you to specify CONSUL_BIND_INTERFACE env variable and if it's set the script will determine the ip in it and substitute this ip to the "-bind" parameter of consul startup. If you set this env variable, say, to "eth0" then your consul agent will be automatically configured with the required ipv4 address belonging to eth0 during its startup.
Somebody has already mentioned this solution here.

[pb0101]

The real problem comes when you are using third part docker images that you can't modify and add another ENTRYPOINT.
Is there any other solution besides modifying ENTRYPOINT

[zfletcher]

You can usually do docker run --entrypoint "/bin/bash" ... or similar.

[rash805115]

I faced this problem today and after hours of research I found this thread. Wow, the discussion here is very informative and it helped me find a solution for myself.

As suggested in this thread I modified my Dockerfile to execute a script at entrypoint and in that script I made the call to metadata service to get LOCAL_IP.
Please refer to my StackOverflow post for more details.

Dockerfile

FROM gliderlabs/registrator:v7

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh && \
    apk update && \
    apk add curl

ENTRYPOINT ["/entrypoint.sh"]

entrypoint.sh

#!/bin/sh

LOCAL_IP=$(curl -s --connect-timeout 3 169.254.169.254/latest/meta-data/local-ipv4)
if [ $? -eq 0 ]; then
  /bin/registrator -ip $LOCAL_IP $@
else
  /bin/registrator $@
fi

I hope this issue is fixed the right way, but until then this solution would suffice for me.

[chaitan94]

The entrypoint solution suggested by multiple people here definitely works, but really just feels like a hacky way.

This issue has been open for a long time now, is there any specific reason why there has been no elegant solution for this so far? I'm really curious to know what are the hurdles here. I'll be even glad to contribute myself if someone can explain what's going wrong here.

[mfortin]

Can't the labels be leveraged ? The agent can define them, and they can be retrieved using the metadata URI. I am looking at container instance tags to retrieve the information somehow, haven't found a clean solution yet.

[stefanjauker]

+1

[varxy20k]

Just want to add my 2 cents here as this is something that would be really useful for us if it worked the same way it works in docker: -e ENV_VAR without value uses the value from the host. Would it work if we could set environment variables without values in the task definition?

[bvanderveen]

+1.

What I want to do is to build my set of containers, build a task def that orchestrates that set of containers, and then launch that taskdef into a cluster with different environment variables (say, to distinguish between dev/prod).

Seems my only choices are to a) maintain separate dev/prod task defs, where the only difference is some env var that indicates dev vs. prod or b) bake the fact of being dev or prod into the containers (which seems completely absurd).

To be honest, I feel like I've been sold a bill of goods, but I'm far enough into the finger trap and Stockholm syndrome that I have no choice but to press on with my 3 weeks of sunk cost and my dreams for a better world.

And this is after a buddy of mine took his startup's infrastructure down by deploying during a scale-out event two days before getting a Letter of Intent from a megacorp learning "the hard way" that ECS Blue/Green deploys don't support Auto-Scaling.

Guys. This is borderline bad-faith.

[coultn]

@bvanderveen would #193 address your use case? In other words, suppose you could define environment variables on a per-cluster basis; every task run in that cluster would use those environment variables (overriding task definition env variables if necessary).

[coultn]

The entrypoint solution suggested by multiple people here definitely works, but really just feels like a hacky way.

This issue has been open for a long time now, is there any specific reason why there has been no elegant solution for this so far? I'm really curious to know what are the hurdles here. I'll be even glad to contribute myself if someone can explain what's going wrong here.

For all those who have been requesting this, one of the main obstacles to providing the IP address in an environment variable is that, with EC2 networking, the IP address of an EC2 instance (or ENI attached to a task) can change. Therefore, the value of the IP address when the task started is not necessarily correct at all times.

[DavidMorton]

sh -c 'aws ecs describe-clusters --cluster $(curl -s $ECS_CONTAINER_METADATA_URI | jq -r ".Labels[\"com.amazonaws.ecs.cluster\"]") --include TAGS | jq -r ".clusters[0].tags" | jq -r "map([.key,.value] | join(\"=\")) | join(\"\n\")"'

Okay, folks, I know this is CRAZY long for a thing... but basically, it'll go and get the tags from the cluster and output them in
KEY=VALUE
format, one line at a time. This could then be used to set the environment variables for your application before it starts running, based on the tags that are on the cluster.

How it works:

The ECS_CONTAINER_METADATA_URI environment variables returns back a URL that gives some JSON back. You can read more about it here. After that, it's a simple matter of using jq (you'll have to include it in your container) to fetch the cluster information from AWS, and trim down to just the tags on the cluster.

Prerequisites:

1. You have to have jq installed on your instance, to parse things out.
2. Your aws cli should be logged in and have the proper policies attached.

Hope this helps people who want to avoid adding volumes. I fully agree with anyone who might say this is a hack, but sometimes, you just gotta figure it out, in the absence of anything better, which AWS doesn't seem to be providing anytime soon.

[mkozjak]

@srrengar - What is the status on this one? We need this feature to pass values generated from one container that's needed for the next one to run in order to use it as a variable.

[shawnweeks]

+1 from me too. I work with a number of containers that need to be able to assume the host name of the EC2 they're running on without running in network=host mode.