Docker drops containers with OOM without any particular reason

[denismakogon]

Here's the thing.

Docker errors with OOM on container even if there's enough free memory. Docker allows to disable OOM kills for those containers that are memory-bound by adjusting container host config (https://docs.docker.com/engine/admin/resource_constraints/#limit-a-containers-access-to-memory). So, all our functions are memory bound (not sure if we follow same limitation as Docker - not less than 4MB RAM) why not use this feature, in any case we'd kill container right after timeout exceeds.

Env: Ubuntu server 16.04 10GB RAM, 2 vCPU, 200GB HDD
Function:

• required RAM: up to 2GB
• container size 2.5GB

This is 3 lines of code fix, but i want to get an input from all: @carimura @rdallman @treeder @hibooboo2.

[rdallman]

we've discussed this before and the behavior modes of an OS / docker container when it's out of RAM are not exactly straight forward. it's much better to kill and let user know that their task has no RAM instead of it trying to swap and that failing / failing mallocs / other mysterious blowups. then the user can look into adjusting ram, debugging what is causing mem usage, etc. vs containers mysteriously running into the timeout. so, gonna close this one, thanks for PR though.

[rdallman]

also, yea, the OOM killer is not the greatest piece of tech ever. it will also take into account the system ram and can kill when there is free space and also based on other heuristics that can be really strange. also, linux isn't necessarily great at managing its cache and freeing up space for malloc. have seen stuff like this happen on previous projects.

the best solution for getting around this really is to add padding to our process. in the past we have tried disabling OOM killer altogether and also changing its aggressiveness but it doesn't really help all that much. you can do things like assign affinity of the OOM killer to processes but i think all of that is kind of a bridge we can cross if we get there. even though it's not very efficient, adding an extra 500-1000 MB of padding can help a lot (also, we aren't taking into account RAM usage of our current process when doling out RAM for containers. this could likely be added relatively easily, and we use a non-trivial amount of RAM with caches for apps and routes).

[denismakogon]

Ok, at least we need doc here that will help to configure OS to make OOM less aggressive, like disabling OOM, extending swap to N GBs, or whatever we think will make life easier.

[rdallman]

i'm not sure we should make any notes about it until the time comes that we actually figure out a preferable way to configure it ourselves, then we end up recommending things we don't understand / recommend / operate ourselves. if we add padding then it kind of falls out a little bit but idk what we need to say other than fn relies on the OOM killer to stop processes that reach their configured RAM capacity, as OOM killer is not perfect this may not always be the case. If this is happening to you often, please file a bug with details of your configuration so that we can try to improve our solution to this -- this day may come, but we are so far from that.

[rdallman]

gonna close this, OOM is plumbed to user pretty well now (was a bug when this was filed)