New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swap options #124

Closed
tj opened this Issue Jun 25, 2015 · 45 comments

Comments

Projects
None yet
@tj

tj commented Jun 25, 2015

The default of no swap seems a bit heavy-handed, for some programs it's really hard to pin-point the usage you'll need, and often 90% of the time they use 10% of that limit until a spike. Maybe an option to disable the limit all together would be nice, not like we want swap at all haha, but for spiky behaviour it's tricky. Let me know what you think!

@euank

This comment has been minimized.

Contributor

euank commented Jun 26, 2015

I think this really raises two separate issues. Swap support seams like a reasonable thing to add to me.

Personally, I think being able to set an acceptable amount of overcommit and then an oom-kill priority on tasks would be useful for the spiky case and many cases. There's a docker issue for making the priority easy to set and I think without that overcommit is much less valuable.

I'll have to run the above by other people on the team, but if you think that overcommit would help or have other ideas for how to handle that sort of thing, we'd value the feedback.

Best,
Euan

@tj

This comment has been minimized.

tj commented Jun 26, 2015

Hmm thinking about it more, it's sort of a program design problem on our end and me under-scheduling to make sure we have good utilization on the hosts. A bit of swap would be nicer than crashing, at least that would give us a chance to alert and raise the limit. So far we haven't had any issues actually exhausting memory on the host, just the mis-configurations for containers.

Semi-related but most of the unpredictable ones are more cron-like, we actually use go-cron in a few of them which is partly why it feels like a huge hack to allocate say 3GB to something that uses a tiny fraction of that most of the time. I know we could use start/run-task but if there's any internal debates regarding cron support I'd give that a big +1 :D.

@nzoschke

This comment has been minimized.

nzoschke commented Jul 17, 2015

+1 for an option to enable swap.

I have been testing a high memory worker service on ECS. On a large instance running a single container with a large memory parameter I saw the Python worker crash:

worker: OSError: [Errno 12] Cannot allocate memory

In this simple case with 1 container per instance, my expectation would be that the container can spike and use swap, and really any amount of the host resources, and not get killed.

But I don't see any way in the APIs to tune the behavior.

@nzoschke

This comment has been minimized.

nzoschke commented Aug 6, 2015

We worked around this in Convox by monitoring Docker and modifying the cgroups directly after containers start

https://github.com/convox/agent/blob/master/monitor.go#L120-L145

@bcwp

This comment has been minimized.

bcwp commented Aug 13, 2015

+1 for option to enable swap and over-provision memory.

@iameli

This comment has been minimized.

iameli commented Aug 21, 2015

Cross-referencing this with very similar discussion on the AWS forums: https://forums.aws.amazon.com/thread.jspa?messageID=671642&#671642

My understanding is somewhat limited, but if I understand everything correctly allowing for configuration of the memory-swap parameter would implement the "optional memory limit" feature requested there, yes?

(I'm "ekmallon" on that thread, FWIW)

@gavannewell

This comment has been minimized.

gavannewell commented Aug 26, 2015

+1 for option to enable swap

Our use case involves automatically scaling our ECS Cluster and automatically scaling our ECS Services to respond to both the normal daily rise and fall in demand as well as spikes in demand. Some services are CPU heavy and others are memory heavy, so we need to be able to scale up/down on both.

Enabling a fixed amount of swap per service, just like we can with physical memory currently, would be ideal for us. That would allow services to spike into swap and give us enough time to scale up while still serving requests. A fixed amount of swap per service would protect us against rogue containers, without having to deal with the OOM issues involved in allowing overcommit on physical memory.

The advantage of this approach is the ECS Cluster can still be scaled as we know how much swap/CPU/memory is available cluster-wide. Allowing overcommit on physical memory would be great for utilization and cron/batch jobs though...

@euank

This comment has been minimized.

Contributor

euank commented Oct 29, 2015

I'd like to add a little more information here since I've investigated this further. Although it's not intentional, right now the default Docker options for swap memory are used. That means that a container does get some amount (in recent Docker versions, default swap is the same as specified memory).
On ECS, your containers will be given swap. docker inspect will show 0, which really means default.

The above only applies if the AMI you're running has swap available, however, and most AMIs / instance types do not have swap by default. There are also kernel configurations that can prevent swap from being used.

Proper swap support should still be modeled and variable per task, so the issue remains, but I wanted to clarify what the current behavior actually is.
Best,
Euan

@leftclickben

This comment has been minimized.

leftclickben commented Nov 15, 2015

👍 for allowing specification of docker memory options without ECS imposing further restrictions.

@benweet

This comment has been minimized.

benweet commented Jan 31, 2016

👍

@tuusberg

This comment has been minimized.

tuusberg commented Feb 7, 2016

+1 for memory-swap option. Rabbit MQ container needs it badly.

@tallavi

This comment has been minimized.

tallavi commented Feb 8, 2016

+1

1 similar comment
@Ouwen

This comment has been minimized.

Ouwen commented Feb 11, 2016

+1

@aldarund

This comment has been minimized.

aldarund commented Feb 12, 2016

+1. kind of sad that such thing not implemented yet

@nzoschke

This comment has been minimized.

nzoschke commented Feb 12, 2016

Here's one line of userdata for the ECS AMI to make a 5 GB swapfile:

fallocate -l 5G /swapfile && chmod 0600 /swapfile && mkswap /swapfile && swapon /swapfile

(In context: https://github.com/convox/rack/blob/master/api/dist/kernel.json#L644)

I'm not sure if this is enough to make the default docker swap usage that @euank mentioned sufficient.

It would be nice if it is, as I'd love to remove the extra container monitor and cgroup modification hack that I'm using here:

https://github.com/convox/agent/blob/master/containers.go#L173

@paulrutter

This comment has been minimized.

paulrutter commented Mar 3, 2016

+1

2 similar comments
@Ouwen

This comment has been minimized.

Ouwen commented Mar 5, 2016

+1

@mixja

This comment has been minimized.

mixja commented Mar 5, 2016

+1

@ryanwalls

This comment has been minimized.

ryanwalls commented Mar 15, 2016

This is definitely needed. Would love if the ECS group responded with at least an acknowledgement.

@JanBednarik

This comment has been minimized.

JanBednarik commented Mar 23, 2016

+1

3 similar comments
@devotox

This comment has been minimized.

devotox commented Apr 15, 2016

+1

@ghost

This comment has been minimized.

ghost commented Apr 15, 2016

+1

@edgaredel

This comment has been minimized.

edgaredel commented Apr 21, 2016

+1

@ooleem

This comment has been minimized.

ooleem commented May 1, 2016

+1

@bryceray1121

This comment has been minimized.

bryceray1121 commented May 26, 2016

+1

1 similar comment
@gabadi

This comment has been minimized.

gabadi commented Jun 8, 2016

+1

@theikkila

This comment has been minimized.

theikkila commented Jun 12, 2016

+1

1 similar comment
@mjaverto

This comment has been minimized.

Contributor

mjaverto commented Jun 15, 2016

+1

@igorescobar

This comment has been minimized.

igorescobar commented Jul 8, 2016

The main thing that need this to be ungently fixed is that we ended having to pay for a larger instance just because we need more memory during the build/deployment process. If we could use memory swap we can still be using smaller instances and the build process will use swap memory to complete the process if needed. Win win.

@samuelkarp

This comment has been minimized.

Member

samuelkarp commented Aug 25, 2016

Hi all,

We recently added support for memory reservation to allow more flexibility in memory settings (see #155 (comment) for a discussion of what we built). Reading over all the use-cases in this thread again, it looks like they're all covered by the memory reservation support we added.

I'm going to close this issue for now. If anyone still has a need for swap specifically and isn't covered by either the memory reservation feature or #124 (comment), please open a new issue specifically focused on swap.

Thanks,
Sam

@samuelkarp samuelkarp closed this Aug 25, 2016

@PepijnK

This comment has been minimized.

PepijnK commented Sep 9, 2016

I think swap VS memory reservation is something different. With swap I explicitly make the choice to trade performance for stability.

@jamesjnadeau

This comment has been minimized.

jamesjnadeau commented Sep 22, 2016

2nd this, they are two different things. I have a scenario where I don't care if I'm using RAM, I have a background worker that can spike on certain jobs to a lot of memeory, don't care if it's in ram, etc. Don't need it to be fast, and would appreciate not having to pay for the larger instance I don't need 99% of the time.

I think you should support the same options as docker:
https://docs.docker.com/engine/reference/run/#/user-memory-constraints

specifically the memory-swap option.

@igorescobar

This comment has been minimized.

igorescobar commented Sep 22, 2016

@jamesjnadeau exactly!! What @samuelkarp said didn't solve the problem... at least for what comes to be memory swapping. (i.e: #124 (comment))

@igorescobar

This comment has been minimized.

igorescobar commented Sep 22, 2016

Seems that the only thing they need to do is to read the standard $DOCKER_OPTS environment variable inside of the /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh. Right on this line:
screen shot 2016-09-22 at 13 46 19

But kinda sucks having to overwrite the original file just for adding this and basically lose any further oficial update for it.

@bs-thomas

This comment has been minimized.

bs-thomas commented Nov 5, 2016

+1

1 similar comment
@ivistvanvarga

This comment has been minimized.

ivistvanvarga commented Feb 7, 2017

+1

@albertocubeddu

This comment has been minimized.

albertocubeddu commented Mar 29, 2017

memory-swap will be really useful... +1

@mhowell-ims

This comment has been minimized.

mhowell-ims commented Apr 19, 2017

+1

@mhowell-ims

This comment has been minimized.

mhowell-ims commented Apr 19, 2017

As suggested by @samuelkarp above, I've opened a new issue #771 specifically for swap support. @PepijnK @jamesjnadeau @igorescobar @bs-thomas @ivistvanvarga @albertocubeddu please contribute your thoughts to #771.

@mhowell-ims

This comment has been minimized.

mhowell-ims commented Apr 19, 2017

@nzoschke @bcwp @gavannewell @leftclickben @tuusberg please contribute your thoughts to #771 as well if you can.

@rubenhak

This comment has been minimized.

rubenhak commented Oct 1, 2017

+1 for having ability to use swap memory with ECS

@gdrte

This comment has been minimized.

gdrte commented Mar 2, 2018

+1 for swap

@evgeny-myasishchev

This comment has been minimized.

evgeny-myasishchev commented Apr 19, 2018

+1

@newhoggy

This comment has been minimized.

newhoggy commented May 10, 2018

+1 for swap

1 similar comment
@cat-turner

This comment has been minimized.

cat-turner commented Jun 14, 2018

+1 for swap

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment