Way to disable per-container memory limit in task definition

[robbyt]

According to the docs1, the per-container memory limit is required. This is inconvenient, because I use the same ElasticBeanstalk Dockerrun.aws.json file with multiple instance types.

Is there a way to disable this setting?

[euank]

You're correct that there's currently no way to disable this requirement.

There is some discussion in #124 about memory limits and we're internally tracking the feature request of percentage / proportional memory limits.

Would being able to specify a proportional amount of memory and/or being able to overcommit memory resolve the issue for you?
Do you want to just be able to opt out of setting memory and having it tracked/scheduled upon at all?

[robbyt]

@euank yes, a percentage/proportional amount of memory would be the best solution. I can imagine that overcommit might confuse people.

Disabling the memory setting requirement would be a quick fix for my use case.

[0xadada]

i'm also interested in this discussion.

[pmcjury]

👍

[morgante]

Having this is kind of a deal-breaker for us.

We're running multiple services in our cluster. Each service by itself usually uses very little memory but occasionally spikes while processing a request. When it spikes, it needs nearly the entire instance's memory. However, services never spike at the same time.

As it is, there doesn't seem to be any way to provision this kind of setup using ECS.

Being able to disable resource limits would be hugely beneficial for us.

[vad]

We need this too, because the OS enables the oom-killer when containers go above the threshold, and the oom-killer implements an heuristic to kill containers "randomly" (and even processes in the host, not only containers!). This happened us twice: a machine (host) stopped responding because a container went above the threshold!

[chenliu0831]

+1. Need this

[gabrianoo]

+1. Need this

[leftclickben]

👍 Please make the memory limit optional. I have a similar situation as described above; services can spike but not simultaneously.

[mr337]

+1

[jawadst]

+1
It's often the case that we run 1 task per EC2 instance and want the container to simply have access to all the memory

[razitz]

+1

[allanharris]

+1 Please allow us to skip memory limit setting.
This is default docker behavior!

[parajanoff]

+1

[echery]

👍

[TheTweak]

+1

[alexmac]

+1 for opting out of memory limit enforcement - I want to tightly pack services onto a small "dev" ec2 instance for debugging purposes

[bribroder]

+1

[pparth]

+1

[ynelin]

+1
In Docker there is everything for our request
https://docs.docker.com/engine/reference/run/#memory-constraints

[kyr7]

+1

[CameronGo]

+1

[ChrisRut]

+1

[mjaverto]

+1

[rjdavis3]

+1

[kswope]

+1

I'm not sure if this is so easy for them. Having a memory limit lets them figure out where a container will fit neatly in a cluster.

[allanharris]

Almost 6 moths passed since robbyt opened this issue...

[ccosgroveOtreva]

+1

[akvadrako]

We also need this feature and though the proposed design with memoryReservation would give us enough control, the naming and interaction between the two fields is confusing. More intuitively, I would propose minimum and maximum fields, either like:

{
  "memoryMinimum": 5,
  "memoryMaximum": 10
}

or even better:

{
  "memory": { "min": 5, "max": 10 }
}

So a system must have at least min for the task to be scheduled and that amount will be reserved. Also, the task will be limited to max usage.

[tallavi]

So a system must have at least min for the task to be scheduled and that amount will be reserved. Also, the task will be limited to max usage.

@akvadrako, I agree it's allot clearer like this.

@samuelkarp, please don't leave us - compose users - hanging with no support for this feature!

[PepijnK]

Hmmm, I'm doubting if this feature request is realistic. How would this soft limit work with regards to auto-scaling? When to decide to pop another instance? If the docker host decides to add another container to the instance, would it have to kill and restart the others in order to max-out memory consumption?

I'm actually in favor of a pre-determined bahavior, as all proposals here would imply a dynamic behaviour that you don't want -- especially not in production environments. So I doubt the usefulnes of it since you want to keep dev/test/prod as equal as possible (that's why you chose Docker in the first place).

Off course it sounds very handy as if you don't have to think about memory consumption of your application at all.. Probably most of the time you run your app/webservice on an machine with plenty of memory. Now suddenly you have to think about a max, and that we don't like.

But it is a good thing. For example we found out our backend service requires lot of mem, but never noticed it since it was running on EB with a micro-instance fully at its disposal. It makes you think about the architecture and I'm glad we have that insight now. Tweaking the memory setting for a ECS task is a bit of hit-and-miss, but with enough testing you can find it out.

[mpestritto]

Hi @samuelkarp

Was this just rolled out to the ECS web console? If so, I'm not sure what isn't supporting the change - the ecs-agent or the aws-cli? Do you know which this could be related to?

Thanks.

Parameter validation failed: Unknown parameter in containerDefinitions[0]: "memoryReservation", must be one of: name, image, cpu, memory, links...

[tedder]

Sir Jeff put up a post: https://aws.amazon.com/about-aws/whats-new/2016/08/amazon-ec2-container-service-now-supports-networking-modes-and-memory-reservation/

It's explained in the docs: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definitions

Now waiting for it to trickle down to CloudFormation so I can use it.

[CameronGo]

The hard limit memory is still a required field. Need to do some experimentation to try and figure out how these two work together and impact the placement/availability of placement for tasks on a host.

[tedder]

Based on the docs, I don't think setting a memory field will affect placement if the memoryReservation is set.

[CameronGo]

Ahh - I didn't read closely enough. I think you are right. Also, the phrasing of this seems to address my comment:

You must specify a non-zero integer for one or both of memory or memoryReservation in container definitions.

[tedder]

Ha, and I missed the 'one or both' clause.

[mpestritto]

Yes - I am setting both. The only thing I changed was adding the memoryReservation to my config file. Unfortunately, the aws ecs register-task-definition call from command line fails with what is below. Any ideas if this is an aws api issue? aws-cli issue?

Curious to see if anyone was able to get it working from command line. As a side note -- works fine from aws web console.

Will gladly transfer this discussion to another thread.

"memory": 384, "memoryReservation": 128,

jenkins@:~/workspace/project-name$ aws ecs register-task-definition --family api-workflow-www-staging --cli-input-json file://project-name-v_189.json

Parameter validation failed:
Unknown parameter in containerDefinitions[0]: "memoryReservation", must be one of: name, image, cpu, memory, links, portMappings, essential, entryPoint, command, environment, mountPoints, volumesFrom, hostname, user, workingDirectory, disableNetworking, privileged, readonlyRootFilesystem, dnsServers, dnsSearchDomains, extraHosts, dockerSecurityOptions, dockerLabels, ulimits, logConfiguration

[tedder]

I bet you'll need to wait for the awscli to be updated. Typically the reference implementations (java, ruby, botocore/awscli/boto3) get updated within the same day of new features, others can take longer. Here's where to watch for a new version of awscli: https://aws.amazon.com/releasenotes/CLI

You can watch all of them here: https://aws.amazon.com/releasenotes

And I publish an RSS feed of it here: http://tedder.me/rss/aws-release-notes.xml

[mpestritto]

Great -- Thanks @tedder I'll look around. Appreciate it.

[CameronGo]

Wonder what version of the ECS agent has to be running.

[samuelkarp]

Hey all,

As you might have noticed, we've now launched the ability to set a soft memory limit as per my proposal. You can see the launch announcement here: https://aws.amazon.com/about-aws/whats-new/2016/08/amazon-ec2-container-service-now-supports-networking-modes-and-memory-reservation/.

To use the new memoryReservation field in your task definition, you must be running ECS agent version 1.5.0 or greater and Docker version 1.9.0 or greater.

You can set memoryReservation in the ECS console today. The AWS CLI and SDKs will be updated with the new field soon.

Please let us know if you have any further questions.

Thanks,
Sam

[tallavi]

Still wondering if there's a way to use the new feature in a ecs-cli compose command. Anyone knows?

[samuelkarp]

@tallavi Can you open a separate issue for tracking this on the Amazon ECS CLI project?

[tallavi]

Done.

[mpestritto]

Following-up on exposing the parameter to aws ecs.... It was a boto issue. PR is up.
boto/botocore#1006

[benjaminwai]

Taking a stab on Elastic Beanstalk I tried the "memoryReservation" param in multi-containers' Dockerrun.aws.json and it is returning "Service:AmazonECS...: Invalid setting for container... At least one of 'memory' or 'memoryReservation' must be specified'. I guess eb is not yet passing "memoryReservation"? Looking forward to be able to use it in eb.

[ssorathia]

Any update on when this would be available for EB? I'm still seeing the following: Service:AmazonECS, Code:ClientException, Message:Invalid setting for container 'app1'. At least one of 'memory' or 'memoryReservation' must be specified., Class:com.amazonaws.services.ecs.model.ClientException

And here is my Dockerrun.aws.json file showing that it's configured:
`"essential": true,

  "memoryReservation": 64,
  "logConfiguration": {

`

But still no luck. Considering this was in ECS quite some time ago, I'm wondering if something is broken with EB.

Anybody know?

[vinayan3]

@ssorathia Please see this forum post https://forums.aws.amazon.com/message.jspa?messageID=747465#747465

ElasticBeanStalk is not broken but does not support the new parameter yet. As the AWS developer says they do not have a timeline to it being fixed.

I was really hoping for this feature to be accessible through ElasticBeanStalk because that's how I deploy things onto ECS these days.

[ssorathia]

@vinayan3 Thanks so much for that link! For some reason my search didn't yield that post! Thanks for helping me out, at least now I know I'm not crazy. I must have checked my spelling a dozen times! :)

[danilobuerger]

Sadly this is still missing from Cloudformation :-(

[brewboy]

+1

[samuelkarp]

@danilobuerger @brewboy The MemoryReservation field is available in the AWS::ECS::TaskDefinition type in CloudFormation.

[yololftw]

@samuelkarp Is this option exposed in aws batch?