Really high memory usage #579

hnioche · 2019-07-19T12:38:43Z

Hi,
I'm using docker for a development environment which has a mysql image.
On my current computer, running arch linux up to date with the default docker setup (community/docker and community/docker-compose).
All the containers I use locally works fine (a ruby container, a nodejs one, a few dotnet core, memcahed).

I only have issues with mysql that uses all the memory available that, each time I start the container, uses immediately all the memory of my computer.
Even the most basic use of the docker image with no database uses 16GB.

I've tried the docker library mysql image, version 8 and 5.7, the oracle version, the percona version, they all have the same issue.
I've tried mariadb and it works as it's supposed, using 100 something MB.
I've tried the same mysql image with podman and had no issue, it uses around 200 MB.

My version of docker is:

Docker version 18.09.8-ce, build 0dd43dd87f

Here's the Dockerfile

FROM mysql:5.7
ENV MYSQL_ROOT_PASSWORD rootpassword
ENV MYSQL_ALLOW_EMPTY_PASSWORD=yes
ENV MYSQL_DATABASE=database

Here's docker stats

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
d66ccdbd03aa        boring_haibt        0.20%               14.27GiB / 15.51GiB   92.00%              1.34kB / 0B         439MB / 299MB       27

And top inside the container

top - 16:11:06 up  7:32,  0 users,  load average: 0.84, 2.05, 1.42
Tasks:   2 total,   1 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.4 us,  3.0 sy,  0.2 ni, 87.1 id,  3.3 wa,  0.9 hi,  0.2 si,  0.0 st
KiB Mem:  16262332 total, 16100992 used,   161340 free,    10252 buffers
KiB Swap:  8388604 total,  6197880 used,  2190724 free.   443820 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 mysql     20   0 17.005g 0.013t      0 S   0.0 85.0   0:05.59 mysqld
  172 root      20   0   24152   2336   2000 R   0.0  0.0   0:00.15 top

I tried to limit the memory usage of the container using -m and mysqld refuses to start when it is bellow 10g with the error:

ERROR: mysqld failed while attempting to check config
command was: "mysqld --verbose --help"

I found something even weirder.
On the same machine, I run a Windows virtual machine using qemu-kvm.
When this machine is started, the mysql container behave normally. When this machine is not started, it uses all the memory of my computer.

I'm not entirely sure the issue is due to this docker image, but I'm a bit lost and don't know how to troubleshoot this problem further. It seems to be specific to mysql running in a container on docker.

The text was updated successfully, but these errors were encountered:

wglambert · 2019-07-19T17:57:29Z

When this machine [windows vm] is started, the mysql container behave normally. When this machine is not started, it [mysql] uses all the memory of my computer.

That's an interesting issue, and the fact that the process in the container is what's actually using the memory.

Getting a quick look on the normal metrics from a mysql start:

$ docker run -d --rm --name mysql -e MYSQL_ROOT_PASSWORD=root mysql:5.7                    
3f6d13a3418954dfde81727b908e084f5ccd29b57cd1f063bd53a9aac39699e0

After about 30 seconds it settles

CONTAINER ID       NAME     CPU %      MEM USAGE / LIMIT     MEM %    NET I/O             BLOCK I/O           PIDS
3f6d13a34189       mysql    0.15%      183.9MiB / 6.578GiB   2.73%    3.24kB / 0B         98.3kB / 760MB        27

So some other unusual differences are your BLOCK I/O metrics, the input stays high and the output is low

With it being affected by QEMU I'm thinking it might be something with the Docker engine and then something with your environment that's causing an edge case.
So I'd file an issue over at https://github.com/moby/moby/issues

Also what if you try using a host-mounted volume for /var/lib/mysql?

hnioche · 2019-07-22T13:01:08Z

Indeed, you're totally right concerning the IO, I had totally missed this, I'm gonna look this way and try to find what this IO is about.

I think you're right, it's more likely to be due to my environment as I was not able to recreate the same issue on a really similar environment (same os, docker version, amount of memory).

I'll follow your advice and open an issue on moby as soon as I have troubleshoot the IO part.
Thanks for your input!

hnioche · 2019-07-24T08:28:33Z

I tried to look at the IO side, it seems really random and couldn't find anything relevant. I wonder if it can be due to swapping due to the really high memory usage.
Also, starting the Windows VM has no impact anymore, I can't think of anything I've changed on this side.
I tried mounting /var/lib/data without seeing any impact, either as a volume or a folder from the host.
Finally I tried to limit memory at the docker daemon level, without any impact, it is totally discarded.

I'll open an issue on Moby project later today and will link it here before closing this issue

hschne · 2019-07-29T14:44:19Z

Just chiming in because I'm experiencing this issue as well. Also on Arch, also exactly the same symptoms that @hnioche described.

Until recently it I could run the container without issues by waiting some time after booting up, but for some reason that no longer works 🤷‍♂️

hnioche · 2019-08-02T17:38:27Z

I had to jump on some other thing so I didn't pursue investigating this issue further. Just out of curiosity, what's your CPU @hschne ?
On my laptop I have a Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
On a desktop setup with Arch as well but with and AMD Zen CPU, I don't have the issue at all.

hschne · 2019-08-05T14:38:10Z

Mine is a Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz.
Can confirm that I haven't experienced the issue on any of my other devices. Real mystery right here 👻

hnioche · 2019-08-05T15:35:10Z

Are your other devices on arch too?
Do you have any specific docker daemon settings on your i7 computer? Any specific kernel parameter at boot time or using sysctl?

hschne · 2019-08-05T15:48:11Z

Yes, the other devices run Arch as well, specifically Antergos. Am not aware of any specific daemon or kernel settings, everything should be vanilla. If you tell me what you are looking for specifically, e.g. which commands to run I can post the output here.

Right now I'm using Podman to run the MySQL container, and that works without any issues :)

evolbug · 2019-08-05T16:49:10Z

Hello, I'm experiencing the same issue described here, running on Arch, my CPU is the same Intel Core i7-8565U.
Some possibly relevant additional info:

I have an NVME SSD
root and home are on separate ext4 partitions
kernel flags: rootfstype=ext4 add_efi_memmap pci=noaer nouveau.modeset=0 pci=biosirq

evolbug · 2019-08-06T13:28:32Z

Relevant report on Redhat Bugzilla that was apparently fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1708115

hnioche · 2019-08-07T13:48:49Z

Relevant report on Redhat Bugzilla that was apparently fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1708115

Oh, good find! The problem seems really close, but the version mentioned in the ticket that supposedly fixes the issue is older than the one I've tried. I wonder if it's in moby itself or in the integration in the distribution.

hnioche · 2019-08-08T12:24:05Z

Thank you so much @evolbug!
The bug you found put me on the right tracks. The issue is caused by the ulimit nofile.
By default, on arch, the value is too low I guess.
running the container with --ulimit nofile=262144:262144 solve the issue and mysql behaves normally. I guess the best way to fix this would be to set this option by default.

Should this be documented on docker images prone to this issue?

Edit:
Actually, it's weird, I think the default limit is way to high, this is what I get by default:

> docker run --rm centos:7 /bin/bash -c 'ulimit -Hn && ulimit -Sn'
1073741816
1073741816

So lowering the value fixes the issue

hschne · 2019-08-08T12:55:21Z

running the container with --ulimit nofile=262144:262144 solve the issue and mysql behaves normally.

You're a god. Thanks to @evolbug as well!

I can confirm that this fixes the issue 👌

evolbug · 2019-08-09T13:05:37Z

I agree as well that this should be set by default on containers exhibiting this behaviour, as it's quite difficult to track down and could happen across docker versions, as seen in the redhat report

wglambert · 2019-08-09T17:25:54Z

The Docker from pacman uses LimitNOFILE=1048576 in the docker.service which is also the default on my Ubuntu install and matches the host's ulimit. So is something changing your guys' host/docker ulimit to be 1073741816?

Curiously the ulimit value of 1073741816 is 1023.999*1048576. Not exactly 1024 but close.
It seems when it's set to infinity then that's the value it sets https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=920913

Looks like Fedora has something related to that exact ulimit size https://bugzilla.redhat.com/show_bug.cgi?id=1715254

the default ulimit value (1073741816 on my Fedora 30)

And an Ubuntu issue was filed that might be relevant? https://www.mail-archive.com/ubuntu-bugs@lists.ubuntu.com/msg5628533.html

due to PermissionsStartOnly=true, systemd runs ExecStartPre commands with insane limits

evolbug · 2019-08-09T17:35:17Z

Running ulimit -Hn && ulimit -Sn on my host (pure Arch) shows

evol~ > ulimit -Hn && ulimit -Sn
524288
1024

Which appears to be half of service's limit, could it be this mismatch?

Edit: another bit of odd behaviour:

evol~ > docker run ubuntu /bin/bash -c ulimit -Hn
unlimited
evol~ > docker run ubuntu /bin/bash -c ulimit -Sn
unlimited
evol~ > docker run ubuntu /bin/bash -c 'ulimit -Hn && ulimit -Sn'
1073741816
1073741816

wglambert · 2019-08-09T17:46:25Z

What's your cat /proc/$(pgrep dockerd)/limits?

$ cat /proc/$(pgrep dockerd)/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             unlimited            unlimited            processes 
Max open files            1048576              1048576              files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       46429                46429                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

It could be your systemd/init file that's changing the ulimit

Also your docker run line is being interpreted by the host's shell when not in single quotes
Example:

$ /bin/bash -c ulimit -Hn
unlimited

evolbug · 2019-08-09T17:50:13Z

I see, here it is

evol~ > cat /proc/$(pgrep dockerd)/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             unlimited            unlimited            processes 
Max open files            1048576              1048576              files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       63448                63448                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

hnioche · 2019-08-09T17:57:31Z

On this note, I had tried to reduce the number of opened files by adding this in my systemd docker service override:

LimitNOFILE=49152

And it seems to be applied properly

➜  ~ cat /proc/$(pgrep dockerd)/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             unlimited            unlimited            processes
Max open files            49152                49152                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       63209                63209                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

But it had no impact on the mysql container ulimits

baj1210 · 2019-12-01T05:59:03Z

I'm encountering memory issues too when running within a docker container. I found the following info suggesting that docker is not releasing threads to mysql. [Info] (https://stackoverflow.com/questions/52439641/mysql-in-docker-container-never-releases-memory-and-regularly-crashes-cause-out)

Quoting Wilson's comment:

"You have already identified your problem. Docker never RELEASES memory. SELECT @@threads_connected; will confirm your suspicion. When you know everything has completed, threads_connected should reduce when the Docker software asks for CLOSE() and release the resources. I suspect someone missed the CLOSE() or equivalent request in the docker software. Suggestions for MySQL configuration are on the way today. Wilson"

yosifkit · 2019-12-03T01:18:11Z

@baj1210, that quote does not seem correct. Docker does not intercept memory allocations. That is provided by whatever libc is in the image (and any container limits are enforced by the kernel, not dockerd).

If you are talking about a bug in docker-userland-proxy that it doesn't close the connection, then that should be reported to https://github.com/docker/libnetwork

chiqui3d · 2021-04-18T12:41:06Z

Is it normal that Mysql(8.0.23) in Docker consumes so much memory?
I have about 20 other containers and MYSQL is the one that consumes the most, how is that possible, what is the solution?

I am WSL2 with Ubuntu 18

chiqui3d · 2021-04-18T18:30:52Z

The same project with Mariadb's latest image consumes 80 MB, compared to 340 MB it is much better, but I still think it's a huge amount.

Piemontez · 2021-06-22T20:10:15Z

Some problem here. Mysql 8.0.25 don't dispache memory on docker and consume 500Mg of memory all time of my virtual machine.

fenKss · 2021-07-27T13:46:45Z

Some problem here. Mysql 8.0.25 don't dispache memory on docker and consume 500Mg of memory all time of my virtual machine.

Faced the same problem. Kernel 5.13.5-arch1-1. mysql 8 consumes a minimum of memory, but mysql 5 - almost all existing

giammin · 2021-12-07T15:35:15Z

same problem on a t4g ec2 instance with Mysql 8.0.23 container

karakani · 2022-01-21T06:22:12Z

mysqld 8.0.13 on Docker version 20.10.8 on Fedora 34 causes same problem.
After updating the image to mysqld 8.0.28, I don't see any problem so far. still consumes 372.6MiB, but it is reasonable to official document.

…s with ulimit-nofile too high Ref docker-library/mysql#579)

…s with ulimit-nofile too high Ref: docker-library/mysql#579

mikehenrty · 2023-02-16T06:07:17Z

I believe for linux users, the issue is related to this: docker/for-linux#73

Specifically, here is the systemd configuration:

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity

Which lead to a very high ulimit (which MySQL 5.7 still doesn't play nicely with):

> docker run --rm mysql:5.7 /bin/bash -c 'ulimit -Hn && ulimit -Sn'
1073741816
1073741816

If you want this fixed for all containers, it's possible to modify the /usr/lib/systemd/system/docker.service directly (not sure why that didn't work for OP @hnioche, maybe cached container, EDIT: or maybe it should be set in /lib/systemd/system/containerd.service?). For me, I thought it better to modify /etc/docker/daemon.json config file (see doc):

{
        "default-ulimits": {
                "nofile": {
                        "Hard": 64000,
                        "Name": "nofile",
                        "Soft": 64000
                }
        }
}

Note the caveat about performance overhead of accounting. Anecdotally, I haven't observed any performance difference in my app, and am happy to be able to use mysql:5.7 without setting ulimits explicitly.

oeresundsgruppen · 2023-02-17T16:56:52Z

Maybe you can adjust mysql docker to allow easier override, e.g.t to innodb_buffer_pool_size, without having to copy in cnf-file. aka docker run --innodb_buffer_pool_size=50M

tianon · 2023-02-18T00:28:13Z

Good news: that's already a thing 😄

$ docker run -it --rm --env MYSQL_ROOT_PASSWORD=bad-example mysql:5.7 --innodb-buffer-pool-size=50M
Unable to find image 'mysql:5.7' locally
5.7: Pulling from library/mysql
e048d0a38742: Pull complete 
c7847c8a41cb: Pull complete 
351a550f260d: Pull complete 
8ce196d9d34f: Pull complete 
17febb6f2030: Pull complete 
d4e426841fb4: Pull complete 
fda41038b9f8: Pull complete 
f47aac56b41b: Pull complete 
a4a90c369737: Pull complete 
97091252395b: Pull complete 
84fac29d61e9: Pull complete 
Digest: sha256:8cf035b14977b26f4a47d98e85949a7dd35e641f88fc24aa4b466b36beecf9d6
Status: Downloaded newer image for mysql:5.7
2023-02-18 00:27:45+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.41-1.el7 started.
2023-02-18 00:27:45+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-02-18 00:27:45+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.41-1.el7 started.
2023-02-18 00:27:45+00:00 [Note] [Entrypoint]: Initializing database files
2023-02-18T00:27:45.430804Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-02-18T00:27:45.801071Z 0 [Warning] InnoDB: New log files created, LSN=45790
2023-02-18T00:27:45.873690Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2023-02-18T00:27:45.934705Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 10d33534-af23-11ed-b787-0242ac1b0006.
2023-02-18T00:27:45.941248Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2023-02-18T00:27:46.291463Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2023-02-18T00:27:46.291477Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2023-02-18T00:27:46.291842Z 0 [Warning] CA certificate ca.pem is self signed.
2023-02-18T00:27:46.520093Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2023-02-18 00:27:48+00:00 [Note] [Entrypoint]: Database files initialized
2023-02-18 00:27:48+00:00 [Note] [Entrypoint]: Starting temporary server
2023-02-18 00:27:48+00:00 [Note] [Entrypoint]: Waiting for server startup
2023-02-18T00:27:49.007503Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-02-18T00:27:49.008410Z 0 [Note] mysqld (mysqld 5.7.41) starting as process 124 ...
2023-02-18T00:27:49.010551Z 0 [Note] InnoDB: PUNCH HOLE support available
2023-02-18T00:27:49.010565Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-02-18T00:27:49.010568Z 0 [Note] InnoDB: Uses event mutexes
2023-02-18T00:27:49.010570Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2023-02-18T00:27:49.010572Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2023-02-18T00:27:49.010574Z 0 [Note] InnoDB: Using Linux native AIO
2023-02-18T00:27:49.010736Z 0 [Note] InnoDB: Number of pools: 1
2023-02-18T00:27:49.010802Z 0 [Note] InnoDB: Using CPU crc32 instructions
2023-02-18T00:27:49.011755Z 0 [Note] InnoDB: Initializing buffer pool, total size = 50M, instances = 1, chunk size = 50M
2023-02-18T00:27:49.013805Z 0 [Note] InnoDB: Completed initialization of buffer pool
2023-02-18T00:27:49.014459Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-02-18T00:27:49.025596Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2023-02-18T00:27:49.034442Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-02-18T00:27:49.034498Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-02-18T00:27:49.067157Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-02-18T00:27:49.067703Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2023-02-18T00:27:49.067713Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2023-02-18T00:27:49.068323Z 0 [Note] InnoDB: Waiting for purge to start
2023-02-18T00:27:49.118465Z 0 [Note] InnoDB: 5.7.41 started; log sequence number 2762314
2023-02-18T00:27:49.119004Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-02-18T00:27:49.119196Z 0 [Note] Plugin 'FEDERATED' is disabled.
2023-02-18T00:27:49.120444Z 0 [Note] InnoDB: Buffer pool(s) load completed at 230218  0:27:49
2023-02-18T00:27:49.123896Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2023-02-18T00:27:49.123907Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2023-02-18T00:27:49.123910Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2023-02-18T00:27:49.123912Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2023-02-18T00:27:49.124371Z 0 [Warning] CA certificate ca.pem is self signed.
2023-02-18T00:27:49.124400Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2023-02-18T00:27:49.130065Z 0 [Warning] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2023-02-18T00:27:49.135812Z 0 [Note] Event Scheduler: Loaded 0 events
2023-02-18T00:27:49.136003Z 0 [Note] mysqld: ready for connections.
Version: '5.7.41'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  MySQL Community Server (GPL)
2023-02-18 00:27:49+00:00 [Note] [Entrypoint]: Temporary server started.
'/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
2023-02-18T00:27:49.923017Z 3 [Note] InnoDB: Stopping purge
2023-02-18T00:27:49.930525Z 3 [Note] InnoDB: Resuming purge
2023-02-18T00:27:49.932378Z 3 [Note] InnoDB: Stopping purge
2023-02-18T00:27:49.935999Z 3 [Note] InnoDB: Resuming purge
2023-02-18T00:27:49.937960Z 3 [Note] InnoDB: Stopping purge
2023-02-18T00:27:49.941782Z 3 [Note] InnoDB: Resuming purge
2023-02-18T00:27:49.943575Z 3 [Note] InnoDB: Stopping purge
2023-02-18T00:27:49.947100Z 3 [Note] InnoDB: Resuming purge
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.

2023-02-18 00:27:51+00:00 [Note] [Entrypoint]: Stopping temporary server
2023-02-18T00:27:51.074246Z 0 [Note] Giving 0 client threads a chance to die gracefully
2023-02-18T00:27:51.074264Z 0 [Note] Shutting down slave threads
2023-02-18T00:27:51.074267Z 0 [Note] Forcefully disconnecting 0 remaining clients
2023-02-18T00:27:51.074271Z 0 [Note] Event Scheduler: Purging the queue. 0 events
2023-02-18T00:27:51.074328Z 0 [Note] Binlog end
2023-02-18T00:27:51.074694Z 0 [Note] Shutting down plugin 'ngram'
2023-02-18T00:27:51.074702Z 0 [Note] Shutting down plugin 'partition'
2023-02-18T00:27:51.074705Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2023-02-18T00:27:51.074708Z 0 [Note] Shutting down plugin 'ARCHIVE'
2023-02-18T00:27:51.074712Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2023-02-18T00:27:51.074734Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2023-02-18T00:27:51.074738Z 0 [Note] Shutting down plugin 'MyISAM'
2023-02-18T00:27:51.074745Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2023-02-18T00:27:51.074749Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2023-02-18T00:27:51.074753Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2023-02-18T00:27:51.074758Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2023-02-18T00:27:51.074761Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2023-02-18T00:27:51.074765Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2023-02-18T00:27:51.074768Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2023-02-18T00:27:51.074772Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2023-02-18T00:27:51.074776Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2023-02-18T00:27:51.074780Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2023-02-18T00:27:51.074784Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2023-02-18T00:27:51.074788Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2023-02-18T00:27:51.074791Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2023-02-18T00:27:51.074795Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2023-02-18T00:27:51.074799Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2023-02-18T00:27:51.074804Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2023-02-18T00:27:51.074808Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2023-02-18T00:27:51.074813Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2023-02-18T00:27:51.074817Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2023-02-18T00:27:51.074821Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2023-02-18T00:27:51.074826Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2023-02-18T00:27:51.074830Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2023-02-18T00:27:51.074835Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2023-02-18T00:27:51.074840Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2023-02-18T00:27:51.074844Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2023-02-18T00:27:51.074848Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2023-02-18T00:27:51.074852Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2023-02-18T00:27:51.074856Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2023-02-18T00:27:51.074859Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2023-02-18T00:27:51.074862Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2023-02-18T00:27:51.074865Z 0 [Note] Shutting down plugin 'InnoDB'
2023-02-18T00:27:51.074925Z 0 [Note] InnoDB: FTS optimize thread exiting.
2023-02-18T00:27:51.074967Z 0 [Note] InnoDB: Starting shutdown...
2023-02-18T00:27:51.175153Z 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2023-02-18T00:27:51.175373Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 230218  0:27:51
2023-02-18T00:27:53.098497Z 0 [Note] InnoDB: Shutdown completed; log sequence number 12184554
2023-02-18T00:27:53.104028Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2023-02-18T00:27:53.104079Z 0 [Note] Shutting down plugin 'MEMORY'
2023-02-18T00:27:53.104092Z 0 [Note] Shutting down plugin 'CSV'
2023-02-18T00:27:53.104102Z 0 [Note] Shutting down plugin 'sha256_password'
2023-02-18T00:27:53.104109Z 0 [Note] Shutting down plugin 'mysql_native_password'
2023-02-18T00:27:53.104464Z 0 [Note] Shutting down plugin 'binlog'
2023-02-18T00:27:53.106199Z 0 [Note] mysqld: Shutdown complete

2023-02-18 00:27:54+00:00 [Note] [Entrypoint]: Temporary server stopped

2023-02-18 00:27:54+00:00 [Note] [Entrypoint]: MySQL init process done. Ready for start up.

2023-02-18T00:27:54.227223Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-02-18T00:27:54.228131Z 0 [Note] mysqld (mysqld 5.7.41) starting as process 1 ...
2023-02-18T00:27:54.230307Z 0 [Note] InnoDB: PUNCH HOLE support available
2023-02-18T00:27:54.230322Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-02-18T00:27:54.230326Z 0 [Note] InnoDB: Uses event mutexes
2023-02-18T00:27:54.230329Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2023-02-18T00:27:54.230331Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2023-02-18T00:27:54.230335Z 0 [Note] InnoDB: Using Linux native AIO
2023-02-18T00:27:54.230503Z 0 [Note] InnoDB: Number of pools: 1
2023-02-18T00:27:54.230572Z 0 [Note] InnoDB: Using CPU crc32 instructions
2023-02-18T00:27:54.231526Z 0 [Note] InnoDB: Initializing buffer pool, total size = 50M, instances = 1, chunk size = 50M
2023-02-18T00:27:54.233748Z 0 [Note] InnoDB: Completed initialization of buffer pool
2023-02-18T00:27:54.234504Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-02-18T00:27:54.245696Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2023-02-18T00:27:54.254405Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-02-18T00:27:54.254451Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-02-18T00:27:54.287095Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-02-18T00:27:54.287624Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2023-02-18T00:27:54.287633Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2023-02-18T00:27:54.288040Z 0 [Note] InnoDB: Waiting for purge to start
2023-02-18T00:27:54.338296Z 0 [Note] InnoDB: 5.7.41 started; log sequence number 12184554
2023-02-18T00:27:54.338741Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-02-18T00:27:54.339880Z 0 [Note] Plugin 'FEDERATED' is disabled.
2023-02-18T00:27:54.345532Z 0 [Note] InnoDB: Buffer pool(s) load completed at 230218  0:27:54
2023-02-18T00:27:54.352910Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2023-02-18T00:27:54.352947Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2023-02-18T00:27:54.352965Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2023-02-18T00:27:54.352979Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2023-02-18T00:27:54.354446Z 0 [Warning] CA certificate ca.pem is self signed.
2023-02-18T00:27:54.354543Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2023-02-18T00:27:54.355203Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2023-02-18T00:27:54.355280Z 0 [Note] IPv6 is available.
2023-02-18T00:27:54.355313Z 0 [Note]   - '::' resolves to '::';
2023-02-18T00:27:54.355348Z 0 [Note] Server socket created on IP: '::'.
2023-02-18T00:27:54.361224Z 0 [Warning] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2023-02-18T00:27:54.379931Z 0 [Note] Event Scheduler: Loaded 0 events
2023-02-18T00:27:54.380314Z 0 [Note] mysqld: ready for connections.
Version: '5.7.41'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)

…ed distros refs: docker-library/mysql#579

The MySQL container is affected by a bug that causes it to gobble up excessive amounts of memory. Ref. docker-library/mysql#579

On some systems (for example ArchLinux) there is by default a large open file limit configured. MySQL 5.7 has a bug that triggers it to allocate a lot of RAM at startup (more than 16 GB!). This bug was fixed in later versions of MySQL. As a workaround the problem can be avoided by adding small ulimits in the docker-compose file which I have done in this commit. For more information see here: https://bugs.mysql.com/bug.php?id=96525 and docker-library/mysql#579.

MySQL コンテナを実行するとやたらメモリを食うので制限をかけた ref: docker-library/mysql#579 (comment)

wcgcoder · 2023-07-31T23:28:38Z

running the container with --ulimit nofile=262144:262144 solve the issue and mysql behaves normally.

Thanks so much @hnioche and @evolbug for helping find the issue. It's still a problem in 2023. My setup only uses kubernetes and containerd, and I'm leaving this here in case someone else has the issue.

Kubernetes has no cri interface for setting ulimit, so this has to be done on the host. If you're using systemd+containerd, you can set this with an additional file (suggestion found here).

#/etc/systemd/system/containerd.service.d/30-override.conf
[Service]
LimitNOFILE=1048576

The limit 1048576 matches my host's nofile limit.

xanhacks · 2023-10-27T10:32:49Z

Upgrading my docker image from mysql:5.7 to mysql:8 solve this issue.

tianon · 2023-12-20T22:21:09Z

Closing, given that 5.7 is now EOL 🙈 ❤️

igorrizhyi · 2024-01-27T12:16:51Z

ulimits are still a valid solution in case you don't want to migrate your mysql to 8
ulimits could also be set on your docker-compose

  mysql:
    image: mysql:5
    ports:
      - "3307:3306"
    ulimits:
      nofile:
        soft: "262144"
        hard: "262144"
    restart: on-failure

wglambert added the question Usability question, not directly related to an error with the image label Aug 9, 2019

zlangbert mentioned this issue Sep 30, 2020

MySQL 5.7 container OOM kill / changing the NOFILE limit bottlerocket-os/bottlerocket#1136

Closed

MattAlexMiracle mentioned this issue Dec 18, 2021

Really high memory usage due to mysqld wandb/server#48

Open

GerardoGR added a commit to katulu-io/fl-suite that referenced this issue Jul 5, 2022

chore(mysql): use mysql 8 to prevent high memory usage in environment…

0a0b825

…s with ulimit-nofile too high Ref docker-library/mysql#579)

GerardoGR added a commit to katulu-io/fl-suite that referenced this issue Jul 6, 2022

chore(mysql): use mysql 8 to prevent high memory usage in environment…

07ac94e

…s with ulimit-nofile too high Ref: docker-library/mysql#579

spelkey-ucd mentioned this issue Jul 28, 2022

update mysql docker compose UCDavisLibrary/main-wp-website-deployment#24

Closed

thaJeztah mentioned this issue Oct 20, 2022

Revert commit that changed LimitNOFILE to infinity to avoid regressions containerd/containerd#7566

Closed

ConstBur mentioned this issue Oct 24, 2022

added note about mysql container memory hog m5c/LobbyService#8

Merged

dodangquan added a commit to dodangquan/ent that referenced this issue Feb 18, 2023

entc/integration: fix high memory usage of mysql-docker on Redhat bas…

3baefe2

…ed distros refs: docker-library/mysql#579

yosifkit mentioned this issue Feb 23, 2023

Cannot build new mysql 5.7 container #950

Closed

whitehorsesoft mentioned this issue Mar 6, 2023

Unable to start MySQL 5.7 container #952

Closed

pfrenssen mentioned this issue Apr 3, 2023

Huge memory hogged by MYSQL lando/lando#3523

Open

pfrenssen added a commit to pfrenssen/herbalstore that referenced this issue Apr 5, 2023

Switch to MariaDB.

5e73936

The MySQL container is affected by a bug that causes it to gobble up excessive amounts of memory. Ref. docker-library/mysql#579

vidboda mentioned this issue May 2, 2023

Docker build fails openvar/rest_variantValidator#91

Open

patrickuhlmann mentioned this issue May 15, 2023

MySQL Bug triggers huge memory allocation with high open file limit hitobito/development#47

Open

mugijiru added a commit to mugijiru/ember-rails-todo-app that referenced this issue Jun 2, 2023

chore: Set ulimits for MySQL

2361b51

MySQL コンテナを実行するとやたらメモリを食うので制限をかけた ref: docker-library/mysql#579 (comment)

mugijiru mentioned this issue Jun 2, 2023

chore: Set ulimits for MySQL mugijiru/ember-rails-todo-app#296

Merged

iutx mentioned this issue Aug 3, 2023

Hang with Erlang Runtime in Kubernetes Pod erlang/otp#7523

Closed

asmigala mentioned this issue Sep 7, 2023

chore(it): Update mysql image smart-connectors/sco-connectors#6

Merged

carlobeltrame added a commit to hitobito/development that referenced this issue Sep 7, 2023

Work around high memory usage docker-library/mysql#579

f4d13f0

This comment was marked as duplicate.

Sign in to view

liudonghua123 mentioned this issue Nov 30, 2023

help request: docker-compose启动mysql登陆不上 eolinker/apinto-dashboard#21

Open

tianon closed this as completed Dec 20, 2023

sneuhaus mentioned this issue Jan 9, 2024

mySQL memory leak using docker compose on Rocky Linux 9.3 cBioPortal/cbioportal#10523

Open

Really high memory usage #579

Really high memory usage #579

Comments

hnioche commented Jul 19, 2019

wglambert commented Jul 19, 2019 • edited Loading

hnioche commented Jul 22, 2019

hnioche commented Jul 24, 2019

hschne commented Jul 29, 2019

hnioche commented Aug 2, 2019

hschne commented Aug 5, 2019

hnioche commented Aug 5, 2019

hschne commented Aug 5, 2019

evolbug commented Aug 5, 2019 • edited Loading

evolbug commented Aug 6, 2019

hnioche commented Aug 7, 2019

hnioche commented Aug 8, 2019 • edited Loading

hschne commented Aug 8, 2019

evolbug commented Aug 9, 2019

wglambert commented Aug 9, 2019

evolbug commented Aug 9, 2019 • edited Loading

wglambert commented Aug 9, 2019

evolbug commented Aug 9, 2019

hnioche commented Aug 9, 2019 • edited Loading

baj1210 commented Dec 1, 2019

yosifkit commented Dec 3, 2019

chiqui3d commented Apr 18, 2021 • edited Loading

chiqui3d commented Apr 18, 2021

Piemontez commented Jun 22, 2021 • edited Loading

fenKss commented Jul 27, 2021

giammin commented Dec 7, 2021

karakani commented Jan 21, 2022

mikehenrty commented Feb 16, 2023 • edited Loading

oeresundsgruppen commented Feb 17, 2023

tianon commented Feb 18, 2023

wcgcoder commented Jul 31, 2023 • edited Loading

xanhacks commented Oct 27, 2023

This comment was marked as duplicate.

This comment was marked as duplicate.

tianon commented Dec 20, 2023

igorrizhyi commented Jan 27, 2024

wglambert commented Jul 19, 2019 •

edited

Loading

evolbug commented Aug 5, 2019 •

edited

Loading

hnioche commented Aug 8, 2019 •

edited

Loading

evolbug commented Aug 9, 2019 •

edited

Loading

hnioche commented Aug 9, 2019 •

edited

Loading

chiqui3d commented Apr 18, 2021 •

edited

Loading

Piemontez commented Jun 22, 2021 •

edited

Loading

mikehenrty commented Feb 16, 2023 •

edited

Loading

wcgcoder commented Jul 31, 2023 •

edited

Loading