container will start flawlessly on one volume but not the other (btrfs)

[ikcalB]

TL;DR:
one path works fine (root subvolume, / mount), the other one get's stuck at the libaio fallback (and does not recover):

seafile-mysql  | 2025-10-07 19:17:20+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
seafile-mysql  | 2025-10-07 19:17:20+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
seafile-mysql  | 2025-10-07 19:17:20+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.14+maria~ubu2204 started.
seafile-mysql  | 2025-10-07 19:17:20+00:00 [Note] [Entrypoint]: Initializing database files
seafile-mysql  | 2025-10-07 19:17:21 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the 
process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup).
seafile-mysql  | create_uring failed: falling back to libaio

mounts:

root@server: ~# mount
/dev/nvme1n1p1 on / type btrfs (rw,relatime,ssd,discard=async,space_cache,autodefrag,subvolid=267,subvol=/root@/current)
/dev/sdd on /srv/seafile-docker type btrfs (rw,relatime,space_cache,autodefrag,subvolid=2224,subvol=/seafile-docker@/current)

snippet of container config
(happens with container versions 10.11, 11, 12)

db:
...
  volumes:
    - "${PATH}:/var/lib/mysql"
 ...

two different options for PATH in .env: first works, second doesn't

PATH="/opt/seafile-docker/seafile-mysql/db"
PATH="/srv/seafile-docker/seafile-mysql/db"

After almost going crazy, I narrow down to what I believe is the filesystem/driver/subvolume.

Any pointers how to diagnose (or solve) that problem are appreciated!

root@server: ~# docker --version
Docker version 26.1.5+dfsg1, build a72d7cd
root@server: ~# docker-compose --version
Docker Compose version 2.26.1-4

[grooverdan]

1. start with a command: --innodb_use_native_aio=0 to see if that starts.

2. off chance it may have been fixed (I don't recall anything related but might be a chance) - pre-release images quay.io/mariadb-foundation/mariadb-devel:10.11

3. Create a new image:

FROM mariadb:10.11

RUN apt-get update && apt-get install -y strace gdb mariadb-server-core-dbgsym && apt-get clean

Then:

3.1 run this image on initialized datadir (copied from working path is ok) with:

• cap_add=CAP_SYS_PTRACE
• user: mysql
• command: strace /usr/sbin/mariadb

3.2 run this image normally and then when its stalled:

• do a process listing in the container using a docker/podman exec containername ps -elf
• gain a stacktrace - docker/podman exec containername gdb --batch --eval-command="set print frame-arguments all" --eval-command="thread apply all bt full" /usr/sbin/mariadbd 1 | tee /tmp/saved-gdb-back

4. on the off chance its a kernel bug, is this the latest distro version? (if not try it). What version is it?

5. this might be achievable outside container so a bug report would help.

How where the volumes created.

[ikcalB]

@grooverdan thanks for your fast reply.

1. adding the command: --innodb_use_native_aio=0 to the yml:
   a. still works for the btrfs-volume on /opt
   b. does seems to fix ths issue for btrfs-volume on /srv/seafile-docker
   now the up of the container hangs for ~1min after starting up so far:

root@server: ~# docker compose up db --no-deps 
[+] Running 0/0
 ⠋ Container seafile-mysql  Recreated                                                                                                                                    0.0s 
Attaching to seafile-mysql
seafile-mysql  | 2025-10-08 11:29:22+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.14+maria~ubu2204 started.
seafile-mysql  | 2025-10-08 11:29:23+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
seafile-mysql  | 2025-10-08 11:29:23+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
seafile-mysql  | 2025-10-08 11:29:23+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.14+maria~ubu2204 started.
seafile-mysql  | 2025-10-08 11:29:23+00:00 [Note] [Entrypoint]: Initializing database files

and then continues to initialize the db and start up successfully
(previously docker compose ran into a timeout, exited and reported the container "unhealthy" - approx. 3+min.
IDK whether it would magically have recovered with a longer timeout, although it seems unlikely to me)

2. after (1) seems to fix the issue, didn't test the dev version
3. see next unhidden comment for details
4. debian stable/trixie, fully up-to-date as of writing.
   relevant software versions:

root@server:~# cat /etc/debian_version 
13.1
root@server:~# uname -r
6.12.48+deb13-amd64
root@server:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 13 (trixie)
Release:        13
Codename:       trixie
root@server:~# docker --version
Docker version 26.1.5+dfsg1, build a72d7cd

5.: is there any benefit from me trying to diagnose the issue further, or does the solution in (1) already tell you, what is going on?

[ikcalB]

After some consideration: The ~1 minute delay does not feel right (should be instantaneous, is at least on the other btrfs-volume)

@grooverdan would an strace help, at that point or is there something more useful I can do?
(I'm myself not really capable of analyzing the strace in a meaningful way, at least for mysql I believe)

[ikcalB]

(3.2) gdb: when using your derived Dockerfile to create the image ('img-mysql-tmp') containing debug symbols:

I was not able to reproduce - I though maybe the additional packages somehow change the behavior of the container.

NOW, I'm having a hard time reproducing complete hang at all.
Could this be a race condition of some kind?
(Looks like the prefix /bin/bash /usr/local/bin/docker-entrypoint.sh is missing in case of failure - any race conditions/issues known in docker compose, regarding parameter interpolation maybe?)

ps -elf for an unhealthy container:

root@server:~# docker exec -it tmp-db ps -elf
F S UID          PID    PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S mysql          1       0  0  80   0 - 281896 -     15:48 ?        00:00:00 mariadb
4 R root         120       0  0  80   0 -  1871 -      15:50 pts/0    00:00:00 ps -elf

a healthy one:

root@server:~# docker exec -it tmp-db ps -elf
F S UID          PID    PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S mysql          1       0  0  80   0 -  1295 -      15:51 ?        00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
4 S mysql         58       1  0  80   0 -   723 -      15:51 ?        00:00:00 /bin/sh /usr/bin/mariadb-install-db --datadir=/var/lib/mysql/ --rpm --auth-root-authentication-
5 S mysql         72      58  0  80   0 -   723 -      15:51 ?        00:00:00 /bin/sh /usr/bin/mariadb-install-db --datadir=/var/lib/mysql/ --rpm --auth-root-authentication-
1 S mysql         73      58  0  80   0 -   723 -      15:51 ?        00:00:00 /bin/sh /usr/bin/mariadb-install-db --datadir=/var/lib/mysql/ --rpm --auth-root-authentication-
1 S mysql         74      58  0  80   0 -   723 -      15:51 ?        00:00:00 /bin/sh /usr/bin/mariadb-install-db --datadir=/var/lib/mysql/ --rpm --auth-root-authentication-
0 S mysql         75      73  0  80   0 -  1060 -      15:51 ?        00:00:00 sed -e /@current_hostname/d
0 S mysql         76      72  0  80   0 -   840 -      15:51 ?        00:00:00 cat /usr/share/mysql/mysql_system_tables.sql /usr/share/mysql/mysql_performance_tables.sql /usr
4 D mysql         77      74  1  80   0 - 339551 -     15:51 ?        00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mysql/english/.. --bootstrap --silent-startup -
4 R root         105       0  0  80   0 -  1871 -      15:51 pts/0    00:00:00 ps -elf

Remaining Issue:

What concerns me more although, is the delay after "... create_uring failed: falling back to libaio":
This is (until now) 100% reproducible:

• no delay (< 1s) for the working btrfs-volume on /opt (and subfolders)
• 50s after ".. libaio" + another 8s until "ready for connections" on the other btrfs-volume

How did I create the volumes:

1. mount the bare device on /mnt
2. cd volume_snapshot_folder && btrfs subvolume create volume_name

The only difference between the volumes:

• /opt is on / which is on an SSD
• I've only recently created the non-working volume, after a sys-upgrade to debian 13. This would inevitably mean a different version of btrfs-tools. (Hower this may or may not influence the volume behavior!?)

[grooverdan]

Last question before I start my own testing on this, there isn't dmesg showing failing hardware on this storage?

And fully updated is referring to the latest distro kernel actually running?

[ikcalB]

dmesg: (stripped of interface renames, mode changes, etc.):
(I don't think that this is relevant, just including for completeness)

[Oct 8 17:26] docker0: port 1(vethd15691a) entered blocking state                                                                                                             
[  +0.000008] docker0: port 1(vethd15691a) entered disabled state                                                                                                             
[  +0.000695] docker0: port 1(vethd15691a) entered blocking state                                                                                                             
[  +0.000009] docker0: port 1(vethd15691a) entered forwarding state                                                                                                           
[ +37.597730] docker0: port 1(vethd15691a) entered disabled state                                                                                                             
[  +0.060961] docker0: port 1(vethd15691a) entered disabled state                                                                                                             
[  +0.000004] docker0: port 1(vethd15691a) entered disabled state

kernel version is the latest for stable, although there would be a newer version in backports:
https://packages.debian.org/search?keywords=linux-image-amd64

root@server:~# uname -r
6.12.48+deb13-amd64

Do you suggest I try with the more recent trixie-backports kernel version (6.16.3)?

What else I've tried:

... and has failed:

mysql:

• MYSQL_INITDB_SKIP_TZINFO=1
• skip-name-resolve is already set in the container's /etc/mysql/mariadb.conf.d/05-skipcache.cnf
• command: --innodb_flush_method=fsync --innodb_buffer_pool_size=4G --innodb_log_file_size=1G
  (and checked the values were actually used with i.e. MariaDB [(none)]> show variables like 'innodb_flush%';)

docker related:

• ... :delegate and ... :cached the volume
• cfq scheduler (/sys/block/sda/queue/iosched/slice_idle and .../group_idle) has been superseded by blk-mq. I've not found issues or ways how to meaningfully tune the latter, so I couldn't test influence of IO scheduler settings

More details to my setup:

On this exact btrfs pool there are ~10 services running, using 6 PostgreSQL DBs. None of these services has ever experienced related or any perfomance issues. (They are running natively, though - so no containerization/virtualization involved)

storage pools

• are btrfs raid1
• root volume layout:
  (one folder per service to store the snapshots, and a symlink to 'current' from which the volume will be mounted. This technique is the same for the working and non-working storage pools)

root@server:/mnt# tree -L2                 
.                                          
├── service1 -> service1@/current/           
├── service1@                               
│   ├── buster-last                        
│   ├── current    
│   ├── post-pg15                          
│   └── v1.1.12-first
├── service2 -> service2@/current/           
└── service2@                                                                          
       ├── buster-last                        
       ├── current     
       ├── forgejo-first                      
       ├── gitea-1.22.1-last                  
       └── v12.04-first

The only difference between working and non-working (repeated from above):

• working is mounted on / (root filesystem)
• working uses SSDs

mount:

root@server: ~# mount
/dev/nvme1n1p1 on / type btrfs (rw,relatime,ssd,discard=async,space_cache,autodefrag,subvolid=267,subvol=/root@/current)
...
/dev/sdd on /srv/seafile-docker type btrfs (rw,relatime,space_cache,autodefrag,subvolid=2224,subvol=/seafile-docker@/current)

thanks for taking your time. Please let me know, if I can provide any more details or assistance in debugging.

[grooverdan]

Do you suggest I try with the more recent trixie-backports kernel version (6.16.3)?

No, seems too much a guess to be something useful. Distro and upstream is doing packaging elsewhere without having to go for very edge kernels, at this stage at least :-).

Thanks for additional details. I'll see what I can reproduce/conclude.

While I think of it, are you (and anyone reading this) willing to do a 5-10minute MariaDB survey this month (October 2025): https://mariadb.typeform.com/survey-2025?utm_source=doiissue