docker: "fatal error: runtime: out of memory" when container produces lots of logs

[kayrus]

Reproducible in 647.0.0, 723.3.0 and 815.0.0

Use this test systemd unit:

[Unit]
Description=MyApp
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill busybox1
ExecStartPre=-/usr/bin/docker rm busybox1
ExecStartPre=/usr/bin/docker pull busybox
ExecStart=/usr/bin/docker run --name busybox1 busybox /bin/sh -c 'while true; do echo "ANEP5iXIZOC/7Ta5jit8YQTXBnCzLPQYPsv5nXsjxfla1YEHgSG5F7tgGaBPA2g5SIhOMacfqVgW\
IrQyQ69OEoSj5KWNOKY+IDARNb3LBtuLnnD4s9FeXwS/6JOX3/b9IIuJMjVXsoWPgTs4T0yqNfDu\
pFPUjH/YJwVwCzwiFP4RQmq1F37eVcITQHlnlGOtzezqX8a4gdVK4C7sAR8Cou5/WZYtu0Zo0Vip\
M6O92bwx2j4JGQ1RevDZu8gAuaikxiWlyRYqhcXratEvwNThfxoJF5epvXEjWaXUIo0opPXxWTJD\
5omO4vYYgh9LprCGbxFH7OtCZ0agdqBeOZf4Taky85MRl42zTEvhWp9XO6Oj9dcE+P4GOzDuCVZg\
bPSKZis8xN5eBlJo+lzrvD/c3FVCFg/05/2PcemiZNmMKngG8dbiiEOrHiCXIRGbLaRFvMckUDOK\
ZnFC1liROZYwrx8pthtMnsMpovik3AggfX3RouihySlDmNU7aUwgCZAIjIhOVYgkPcG9Q/TFs27v\
YnQn3y4lF1UsHLShqrFPSrs1t23+sG/qcNbQLqBRm1iJqCIHaDIuiuNDeIJTLejPTHL4pIeGNrky\
BXmZ5bV/OL1B6IvJntl/Fd/UMdwI07EDVgpYpAg1FGRkLBhQGlBpX+hAbOHoGj381mpVhbwEhsp1\
7CJQAolcfjLBD32W9oPQ9zhLpMj4TIBRAAlourtrl700euzkAjoCcfuTrWESkFSygGqzNmzl4URA\
cLVbuwSIs4aLyTM9rJ/2Wbr/CpIuwosWQdxIxlDeXJEWOCgTGPNTVN/IFHKtw0i+dG3dCTqrGbnX\
6F8hSTX3BHb0VzP/2hrrprpV4A5J2HFWDODSZ6ww91+IVzLUPuK9V0BQZC0XxUURuhJolFN1ex/y\
J7MrPmgDhBjKEmiKqMwQk3bjN7FNjhgWaBgfhgAB58KD4w3Tr5Y2OwXz4CcsHxoeuDfcO5JVtdnF\
kpwbNBWMYaB2ZM7rK4SUrtL5Nbgob9x8jK+sob9V32gCFF2JB9NEOc8MzETBxc9l8ANK1TnedQdh\
gwwi93TdKQE3TyBooNyOwbNQEzDlT4D3FMhdcTn+9CI5KvxX5T1G4a8PwCQBH63UauXbmWnK1kfB\
A2KUUGaMPk9iFO7psGzKMAKybZvTq9AZ+EsHORz0s2zW7auJk0D5+K20rFezPxim5vhJdC9Bcq1M\
ixYDKlWMyUi+w7i2NumZrYmDqkD0ojGcSyGiq2oQSmhfL8eWRJ7luWqyPdlWDSJR7px+DRBmuiys\
/1VA49KPbQlgcR6BdrHZoN75VZg5lXLrwwbMltdH+I1hxfLAK7UYHqB/LZG32b1lpQwtk/7RQRUW\
6MbRnp7/7D+dCrWpfXAR2Q56vz04z34Imf7KNvSBgk/iUNhtRxoYh8CDWZH4g63cbPhF7vID/sai\
8OLa7GpAEkXWmOB3AnVCny5jHtk9WHBjVQ37ZiAz/TLYDn0ZfppZSEu1BoMZdM+kZ1oA6FNkiyJy\
mCcJ5Oac1DjYq+66JN0BM9DEGxyecEF0dqYubGJ69t5Pzm2zwss1l5O2vPeJxcwTKoT8WoELoWw6\
ATLaZNwe8eXh9q8R0Y2n3ZJdSjxXKQUni6IMLO3FmhYiHEG/dYJ6rQOhDCeaY3PVGvebIDBfezEe\
648N669UMHTLAeRT3HsH1R+/KR7BjBNptc23p9AdWQDKA1HnR/Gu994nsPnUNw+MYLvNvARkG00R\
U3vusldK7Vfbk8Gg+DqtzQFfKcECDFXcNsf7D5kY/we874mAnGCjlmuOhwGyLKgdSynN3e374g/d\
vxnaThUaqCO07K5b4BqDrd9HWXKmtgadxZt6EhXIEYJNeW/Hl9HYcKz4z1NQ76qiqIqbsDSZdMjx\
yxVfNlUt96YqvJmTgOUK5ooPSSYfNYgZszwG6RXSPBqFMQluW04KO61CTkM2qOn7uPYcXThpIn09\
FzyuA0Y1r7VTy6rx+Gs0KkpO1PvckTMHftT98tLSBQPkGEr5sqz1Q8zmDNLrY89VM64QJRDNCnuU"; done'
ExecStop=/usr/bin/docker stop busybox1

Run it and you'll receive fatal error: runtime: out of memory error message

/cc @philips @crawford

UPDATE: weird... busybox+sh fails docker container. ubuntu+sh fails container. ubuntu+bash - not.
UPDATE1: increased burst and rate interval (RateLimitInterval=120s and RateLimitBurst=10000) produces same docker issue even with ubuntu container + bash.

[dongsupark]

I can also easily see this error, with docker 1.8.2, kernel 4.3-rc4 on a virtual machine with 1G RAM.
I suppose it's a similar issue as moby/moby#14460. However, with docker 1.8.2 the crash seems to still happen.
As a workaround, it would be possible to raise system memory to 2G or higher. That'll work according to my tests.
In contrast to other issues like #489, it's interesting that kernel OOM killer is not necessarily involved in this case. So it might make sense to limit memory utilization on the docker side.

[dongsupark]

Docker 1.9.0-dev 52de250 (current master) seems to be a little different from 1.8.2. Now with 1.9.0-dev the dockerd process doesn't die immediately on the internal buffer, but it runs for a while, until the kernel oom-killer actually kills that daemon.

[dongsupark]

I think we can close this issue, as the problem was already solved by moby/moby#16190, which reimplements NewBufReader(). See also https://github.com/LK4D4/docker/commit/24310b5b4ab600ab5cb046d2f2ceaaa086b30be3 and https://github.com/LK4D4/docker/commit/56b70bf84e253bfedef94248a0b6b1f8d730091c.
With those changes, docker daemon effectively introduces a maximum limit to the internal buffer, making use of a new type BytesPipe. That looks good to me.
All we should do with this issue is waiting for the next release of docker 1.9.

[jonathanperret]

I would be careful closing this. The reason moby/moby@52de250 behaves differently, is that it includes moby/moby#16190 . However, as @dongsupark witnessed, it still allocates unlimited memory (the internal buffer is not capped even after moby/moby#16190 , despite appearances ) when logging faster than logs are consumed.

[vcaputo]

There's a new PR up re: bounded buffer: moby/moby#17877

[chr0n1x]

bump & 👍

[mischief]

moby/moby#17877 is merged and is in 1.10.0. it could potentially be backported to 1.9.1. i was not able to reproduce this problem on coreos 983.0.0.

[jonathanperret]

moby/moby#17877 should definitely prevent this issue, indeed.

[crawford]

This should no longer be an issue now that a newer Docker is being shipped.