jbd2 with 100% Disk IO

[Xplouder]

---

BUG REPORT INFORMATION

Description
I wrote a small docker-compose (Nginx + PHP-FMP + MYSQL). Basically to reproduce my current development stack, however, when write operations occur (ex. mysql inserts) the entire host disk IO get overloaded ~99% every time.
The same kind of operations inside my VM stack do all the operations (the ones I have been benchmarking while finding a solution for this problem) runs in 2 - 3minutes and uses small amount of disk IO %. With docker it takes 10 - 12min.

Steps to reproduce the issue:

1. docker-compose up -d
2. docker-compose exec php bash
3. run a php script that executes multiple migrations from CLI and send data to mysql container using mysql client.
4. profit...

Describe the results you received:

Describe the results you expected:
Small/medium amount of disk IO % during the script runnings.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      17.10.0-ce
 API version:  1.33
 Go version:   go1.8.3
 Git commit:   f4ffd25
 Built:        Tue Oct 17 19:04:16 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.10.0-ce
 API version:  1.33 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   f4ffd25
 Built:        Tue Oct 17 19:02:56 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 3
 Running: 3
 Paused: 0
 Stopped: 0
Images: 15
Server Version: 17.10.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.10.0-37-generic
Operating System: Linux Mint 18.2
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.58GiB
Name: xxxx
ID: CTG3:YYYP:CEUA:UEEQ:QORQ:AOGU:HPRS:IKV7:QCQT:A3IK:UKY7:EXDZ
Docker Root Dir: /home/xplouder/.docker
Debug Mode (client): false
Debug Mode (server): false
Username: xxxx
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

docker-compose.yaml

version: "3.3"

services:
  mysql57:
    image: mysql:latest
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}

  php:
    image: xplouder/php-fpm-phalcon:v1
    user: ${USER_ID}:${GROUP_USER_ID}
    ports:
      - "9000:9000"
    volumes:
      - type: volume
        source: ./code
        target: /code
      - type: volume
        source: ./php_configs/log.conf
        target: /usr/local/etc/php-fpm.d/zz-log.conf

  web:
    image: nginx:1.13.5-alpine
    ports:
      - "8080:80"
    volumes:
      - type: volume
        source: ./code
        target: /code
      - type: volume
        source: ./nginx_conf.d
        target: /etc/nginx/conf.d
    links:
      - php
    depends_on:
      - php
      - mysql57

journalctl -u docker.service

level=info msg="libcontainerd: new containerd process, pid: 14805"
level=warning msg="failed to rename /home/xxxx/.docker/tmp for background deletion: rename /home/xplouder/.docker/tmp /home/xplouder/.docker/tmp-old: file exists. Deleting synchronously"
/home/xxxx/.docker/tmp /home/xxxx/.docker/tm
level=info msg="Graph migration to content-addressability took 0.00 seconds"
level=warning msg="Your kernel does not support swap memory limit"
level=warning msg="Your kernel does not support cgroup rt period"
level=warning msg="Your kernel does not support cgroup rt runtime"
level=info msg="Loading containers: start."
level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
level=info msg="Loading containers: done."
level=info msg="Docker daemon" commit=f4ffd25 graphdriver(s)=overlay2 version=17.10.0-ce
level=info msg="Daemon has completed initialization"

Note: I already tested AUFS, overlay and overlay2 storage drivers but without luck.

[thaJeztah]

Looks like this is not really an issue with docker, but jbd2 itself in combination with (e.g.) mysql; see https://serverfault.com/questions/363355/io-wait-causing-so-much-slowdown-ext4-jdb2-at-99-io-during-mysql-commit, https://www.linux.org/threads/jbd2-dm-0-8-usign-alot-of-i-o.11562/

[boaz0]

I am closing this as this isn't related directly to Docker/Moby.

[Xplouder]

@thaJeztah Thank you, apparently you are right. To solve this problem i had to create a XFS partition to hold every docker data (images, containers, etc) and add innodb_flush_log_at_trx_commit=0 configuration to Mysql server. Not perfect but now it performs away better.