Wings issue with cgroups on Debian Bullseye

[Marfjeh]

Background:

• Panel or Daemon: Daemon
• Version of Panel/Daemon: 1.2.1
• Server's OS: Debian 11 (Bullseye) 5.10.0-1-amd64 Migrate ability to reset passwords #1 SMP Debian 5.10.4-1 (2020-12-31) x86_64 GNU/Linux
• Your Computer's OS & Browser: Manjaro Linux, Firefox

Describe the bug

Kernel Info

Linux Meneer-Aart 5.10.0-1-amd64 #1 SMP Debian 5.10.4-1 (2020-12-31) x86_64 GNU/Linux

php -v

PHP 7.3.26-1+0~20210112.74+debian10~1.gbpd78724 (cli) (built: Jan 12 2021 13:59:47) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.26, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.3.26-1+0~20210112.74+debian10~1.gbpd78724, Copyright (c) 1999-2018, by Zend Technologies

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
Server:

Containers: 5

Running: 2

Paused: 0

Stopped: 3

Images: 4

Server Version: 20.10.2

Storage Driver: overlay2

Backing Filesystem: extfs

Supports d_type: true

Native Overlay Diff: true

Logging Driver: json-file

Cgroup Driver: systemd

Cgroup Version: 2

Plugins:

Volume: local

Network: bridge host ipvlan macvlan null overlay

Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog

Swarm: inactive

Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc

Default Runtime: runc

Init Binary: docker-init

containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b

runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff

init version: de40ad0

Security Options:

apparmor

seccomp

Profile: default

cgroupns

Kernel Version: 5.10.0-1-amd64

Operating System: Debian GNU/Linux bullseye/sid

OSType: linux

Architecture: x86_64

CPUs: 8

Total Memory: 62.61GiB

Name: Meneer-Aart

ID: 3YVF:LTPV:PRKV:ZY27:OS3X:ZH5Q:LKJS:EPSI:VSJQ:FUSX:G467:CX43

Docker Root Dir: /var/lib/docker

Debug Mode: false

Registry: https://index.docker.io/v1/

Labels:

Experimental: false

Insecure Registries:

127.0.0.0/8

Live Restore Enabled: false
WARNING: No kernel memory TCP limit support

WARNING: No oom kill disable support

WARNING: Support for cgroup v2 is experimental

I'm having issues with wings. as it will log this into the console on web:

Console output

Pterodactyl Daemon]: Updating process configuration files...
[Pterodactyl Daemon]: Ensuring file permissions are set correctly, this could take a few seconds...
container@pterodactyl~ Server marked as starting...
[Pterodactyl Daemon]: Pulling Docker container image, this could take a few minutes to complete...
Pulling from pterodactyl/core 
Digest: sha256:cf8006eea562c6f3c42167b175e994412f20cad85bfa1e05184edef427829042 
Status: Image is up to date for quay.io/pterodactyl/core:java 
[Pterodactyl Daemon]: Finished pulling Docker container image
container@pterodactyl~ Server marked as offline...
container@pterodactyl~ Error Event [fd31f4d1-eefe-439e-989b-a9c6c573d75d]: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "1" to "/sys/fs/cgroup/system.slice/docker-deea5264800f722381301f7869c8b4372f28bf08c87534f13220bbf7cedecc70.scope/io.bfq.weight": open /sys/fs/cgroup/system.slice/docker-deea5264800f722381301f7869c8b4372f28bf08c87534f13220bbf7cedecc70.scope/io.bfq.weight: permission denied: unknown
[Pterodactyl Daemon]: ---------- Detected server process in a crashed state! ----------
[Pterodactyl Daemon]: Exit code: 126
[Pterodactyl Daemon]: Out of memory: false
[Pterodactyl Daemon]: Aborting automatic restart, last crash occurred less than 60 seconds ago.

wings.log output

 INFO: [Jan 18 12:50:51.270] syncing server state with remote source before executing installation process server=e40b3c22-c2b9-40dc-a0ae-cc46dee290c1
 INFO: [Jan 18 12:50:51.296] beginning installation process for server server=e40b3c22-c2b9-40dc-a0ae-cc46dee290c1
 INFO: [Jan 18 12:50:52.561] creating install container for server process install_script=/tmp/pterodactyl/e40b3c22-c2b9-40dc-a0ae-cc46dee290c1/install.sh server=e40b3c22-c2b9-40dc-a0ae-cc46dee290c1
 INFO: [Jan 18 12:50:52.601] running installation script for server in container container_id=90f1d7b5d90d9647eb06e69f6bee797964db006515fdf291aad9d25da69b3696 server=e40b3c22-c2b9-40dc-a0ae-cc46dee290c1
 INFO: [Jan 18 12:51:02.460] completed installation process for server server=e40b3c22-c2b9-40dc-a0ae-cc46dee290c1
 INFO: [Jan 18 12:51:03.861] syncing server configuration with panel server=e40b3c22-c2b9-40dc-a0ae-cc46dee290c1
 INFO: [Jan 18 12:51:03.880] performing server limit modification on-the-fly server=e40b3c22-c2b9-40dc-a0ae-cc46dee290c1
ERROR: [Jan 18 12:51:05.659] error processing websocket event "set state" error=Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "1" to "/sys/fs/cgroup/system.slice/docker-deea5264800f722381301f7869c8b4372f28bf08c87534f13220bbf7cedecc70.scope/io.bfq.weight": open /sys/fs/cgroup/system.slice/docker-deea5264800f722381301f7869c8b4372f28bf08c87534f13220bbf7cedecc70.scope/io.bfq.weight: permission denied: unknown error_identifier=fd31f4d1-eefe-439e-989b-a9c6c573d75d event=set state server=e40b3c22-c2b9-40dc-a0ae-cc46dee290c1
Stacktrace:

Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "1" to "/sys/fs/cgroup/system.slice/docker-deea5264800f722381301f7869c8b4372f28bf08c87534f13220bbf7cedecc70.scope/io.bfq.weight": open /sys/fs/cgroup/system.slice/docker-deea5264800f722381301f7869c8b4372f28bf08c87534f13220bbf7cedecc70.scope/io.bfq.weight: permission denied: unknown
INFO: [Jan 18 12:51:05.666] detected server as entering a crashed state; running crash handler server=e40b3c22-c2b9-40dc-a0ae-cc46dee290c1

INFO: [Jan 18 12:51:05.669] did not restart server after crash; occurred too soon after the last server=e40b3c22-c2b9-40dc-a0ae-cc46dee290c1

There is a permission problem as you can see, the wierd thing is that, i have other docker containers running like mariadb and those folders are made: /sys/fs/cgroup/system.slice/docker-xxxxxxxxxx.scope and those run fine.

I'm not sure if this is a wings issue or a docker issue. I've reinstalled docker, reinstalled wings, but that did not have a effect. even did docker system prune -a that also fixed things in the past but nope.

I also remove the ram limits and storage limits on a test server on the panel but that did not work either.

Things that catched my attention: in docker info (see above)
it says:

WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support
WARNING: Support for cgroup v2 is experimental

Has this anything to do with the issue i'm having?
The system is running Bare metal. Not running on a VM or any virtualization.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'Any server'
2. Click on 'start'
3. Scroll down to 'n/a'
4. See error in console

Expected behavior
That the server will boot up.

[Marfjeh]

Thats weird, its running on bare metal.

[parkervcp]

Set the memory limit to 0 and not -1

Please note I'm on my phone and can't read the whole error properly yet.

[Marfjeh]

No worries, I did not use memory limit to -1its set to 0

Even setting a limit, it still throws the same exception from wings.

[schrej]

First of all, thanks for that excellent bug report, those are rare unfortunately.

This doesn't seem to be caused by memory limit, but by storage limits. I'd assume in this case it's the Block IO Proportion.

BFQ, which is used by docker for io scheduling, seems to be disabled in the latest Debian kernels (mailing list). Not sure what the best approach to solving this would be.

[Marfjeh]

Thank you, I know the fustrations of people not giving enough data, and then you need figure it out somehow... :P

Anyways that out of the way, Yeah that is something we also realized, my friend did some experimentations, he executed the docker container manually

docker run -u 999 -it --rm -v $PWD:/home/container quay.io/pterodactyl/core:java java -jar server.jar

This booted up the docker image fine without any problems. but the wings service has no idea / control over it.

He also found out that,
/sys/fs/cgroup/system.slice/docker-xxxxxx.scope/io.bfq.weight

No longer exists. he tried to echo 10000 to it which failed. But what did exist was:

io.max
io.pressure
io.stat
io.weight. <----

But i'm not sure if that is the same for Block IO Proportion.

[Marfjeh]

More experiments:

my friend executed this,

docker run --blkio-weight=1000 -u 999 -it --rm -v $PWD:/home/container quay.io/pterodactyl/core:java java -jar server.jar

This fails as well. the output is:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "10000" to "/sys/fs/cgroup/system.slice/docker-cc1ddbd9492deabcd0410d452008a71e2b1531ac1ed891d5e8c50b6eea4de5ab.scope/io.bfq.weight": open /sys/fs/cgroup/system.slice/docker-cc1ddbd9492deabcd0410d452008a71e2b1531ac1ed891d5e8c50b6eea4de5ab.scope/io.bfq.weight: permission denied: unknown

[DaneEveritt]

Going to close this because it isn't specifically an issue with Pterodactyl, but if you find a solution and let us know here we can look into adding it to the documentation to help people out in the future. Absolute worst case we can just disable block IO on Debian 11 I guess.

[Marfjeh]

For now we compiled a custom version where we removed BlkioWeight in wings.
https://github.com/pterodactyl/wings/blob/develop/environment/docker/container.go#L121

That is working fine for us. However, This is probably not the best way to do this, for us this is fine because we're only hosting a minecraft server for friends. and for hosting companies that probably do make use of Block IO Proportion is a bad idea.

I think maybe adding a flag to wings as a workaround? or that it does automatically.

[parkervcp]

This sounds specific to deb 11 and them disabling a feature that worked before.

[X-Coder]

For a proper fix pterodactyl/wings requires support for cgroups v2. I guess cgroups v1 will be removed in the long term.

Until pterodactyl has support for v2 you can use this workaround:

You have to downgrade your os to cgroup v1 by adding these to /etc/default/grub parameter GRUB_CMDLINE_LINUX:
systemd.unified_cgroup_hierarchy=false systemd.legacy_systemd_cgroup_controller=false

And don't forget update-grub and reboot afterwards editing the file.

Source: https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#openstack-cgroups