ZFS File System - Fatal Error: Steamcmd needs 250MB of free disk space to update.

[diablotf]

My first ever issue reported on Github.

Using Container Station on QNAP with a ZFS file system I get the error below.
A second Container Station on QNAP works perfectly with the exact same config file. Literally copy and pasting it.
Only difference between the two is the filesystem.

/config and /opt/valheim are created without issues. Even the /tmp/dump files are created. There are no space restrictions on the container.

Error Messages:

Mar 27 00:30:01 /supervisord: valheim-updater [----] Verifying installation...
Mar 27 00:30:01 /supervisord: valheim-updater [ 0%] Downloading update...
Mar 27 00:30:01 /supervisord: valheim-updater statvfs64: ret -1 errno 75
Mar 27 00:30:01 /supervisord: valheim-updater f_bsize : 0
Mar 27 00:30:01 /supervisord: valheim-updater f_frsize: 0
Mar 27 00:30:01 /supervisord: valheim-updater f_blocks: 0
Mar 27 00:30:01 /supervisord: valheim-updater f_bfree : 0
Mar 27 00:30:01 /supervisord: valheim-updater f_bavail: 0
Mar 27 00:30:01 /supervisord: valheim-updater src/tier1/fileio.cpp (3785) : Assertion Failed: statvfs64 failed for ., errno 75
Mar 27 00:30:01 /supervisord: valheim-updater Execute: /root/.steam/root/ubuntu12_64/disk-free . >/tmp/disk-free.nfiLRn
Mar 27 00:30:02 crash_20210327003001_4.dmp[243]: Finished uploading minidump (out-of-process): success = yes
Mar 27 00:30:02 crash_20210327003001_4.dmp[243]: response: Discarded=1
Mar 27 00:30:02 crash_20210327003001_4.dmp[243]: file ''/tmp/dumps/crash_20210327003001_4.dmp'', upload yes: ''Discarded=1''
2021-03-27 00:30:02,876 INFO reaped unknown pid 243 (exit status 0)
Mar 27 00:30:02 /supervisord: valheim-updater src/tier1/fileio.cpp (3815) : Assertion Failed: disk-free command failed
Mar 27 00:30:02 /supervisord: valheim-updater
Mar 27 00:30:02 /supervisord: valheim-updater [ 0%] !!! Fatal Error: Steamcmd needs 250MB of free disk space to update.
Mar 27 00:30:02 /supervisord: valheim-updater src/tier0/threadtools.cpp (3553) : Assertion Failed: Illegal termination of worker thread 'Thread(0x0x58a1d8f0/0x0xf7780b'
Mar 27 00:30:02 /supervisord: valheim-updater ERROR - Failed to download Valheim server from Steam - retrying later - check your networking and volume access permissions

Config File

version: '3'

services:
valheim:
image: lloesche/valheim-server
cap_add:

• sys_nice
  volumes:
• valheim_config:/config
• valheim_opt:/opt/valheim
  ports:
• "2456-2457:2456-2457/udp"
• "9001:9001/tcp"
  environment:
• SERVER_NAME=Redacted
• SERVER_PORT=2456
• WORLD_NAME=Redacted
• SERVER_PASS=Redacted
• SERVER_PUBLIC=true
• SUPERVISOR_HTTP=true
• SUPERVISOR_HTTP_USER=Redacted
• SUPERVISOR_HTTP_PASS=Redacted
  restart: always
  stop_grace_period: 2m
  volumes:
  valheim_config:
  valheim_opt:

Been searching around but not come across any fixes short of someone installing Docker in another fashion on another device.

Thanks for any help.

[lloesche]

Been searching around but not come across any fixes short of someone installing Docker in another fashion on another device.

If the error is being thrown by Steam and you're sure it is because of the filesystem there's literally nothing we can do then.
Although one of the first results on Google says that setting a quota on the fs might solve the issue. But that's something you'll have to do on your system, not something we can do inside the container.

I just tested on a local system and had no issues so doesn't seem to be a general ZFS incompatibility. Was hoping I could reproduce your error to test the quota workaround.

[root@blackhole ~]# zfs create data/valheim
[root@blackhole ~]# mkdir /data/valheim/data /data/valheim/config
[root@blackhole ~]# docker run --rm -it --name valheim-server -v /data/valheim/data:/opt/valheim -v /data/valheim/config:/config lloesche/valheim-server
...
...
Mar 27 09:53:32 /supervisord: valheim-updater Connecting anonymously to Steam Public...
Mar 27 09:53:34 /supervisord: valheim-updater Logged in OK
Mar 27 09:53:34 /supervisord: valheim-updater Waiting for user info...
Mar 27 09:53:44 /supervisord: valheim-updater OK
Mar 27 09:53:44 /supervisord: valheim-updater  Update state (0x3) reconfiguring, progress: 0.00 (0 / 0)
Mar 27 09:53:44 /supervisord: valheim-updater  Update state (0x5) verifying install, progress: 5.85 (61667003 / 1053298784)
Mar 27 09:53:44 /supervisord: valheim-updater  Update state (0x5) verifying install, progress: 39.93 (420604111 / 1053298784)
Mar 27 09:53:44 /supervisord: valheim-updater  Update state (0x5) verifying install, progress: 74.24 (781993129 / 1053298784)
Mar 27 09:53:44 /supervisord: valheim-updater Success! App '896660' fully installed.
...
...
Mar 27 09:54:04 /supervisord: valheim-server DEBUG - [117] - Server is now listening on UDP port 2456

And it sees the data available to the FS inside the container.

[lukas@blackhole ~]$ docker exec -it 39af7b85b814 df -h | grep config
data/valheim    7.6T  2.0G  7.6T   1% /config

But this is on a Fedora Server with ZFS manually installed, not a QNAP appliance.
Can you check what a df inside the container returns on your system?

[diablotf]

This is what it looks like within the container. (Yep that's 56TB available)

root@6f88325b9567:/# df -h
Filesystem Size Used Avail Use% Mounted on
zpool1/zfs530/zfs5300002/151d05a616f1bc268705886a609218568371b172e1cd9e897d0ac27b0f1abeb1 56T 424M 56T 1% /
tmpfs 64M 0 64M 0% /dev
tmpfs 63G 0 63G 0% /sys/fs/cgroup
shm 64M 0 64M 0% /dev/shm
zpool1/zfs530/zfs5300002 56T 1.9M 56T 1% /config
tmpfs 63G 0 63G 0% /proc/asound
tmpfs 63G 0 63G 0% /proc/acpi
tmpfs 63G 0 63G 0% /proc/scsi
tmpfs 63G 0 63G 0% /sys/firmware

I have one more QNAP server that doesn't use ZFS and I am going to try the exact same config to get a second confirmation one way or the other. The path within the NAS looks very different to the ones seen by applications running on it and what's seen within container station.

Container station shows the volumes as :

On File manager it tells me the two volumes are located here.

I'll get back to you by tomorrow with a test on my final QNAP device. Thanks for your help.

[diablotf]

I found some time today and did the test. Worked first time with same config file on a Non-ZFS based QNAP device.
The path on the QNAP File Manager was identical to the ZFS edition. However you can see the mounts are different from Container station and the output from within the container. in the screenshots below.

[lloesche]

Were you able to test what that Google search suggested? Create a valheim ZFS and put like a 10G quota on it.
I don't know how it works on QNAP but on standard ZFS its e.g.

# zfs create zpool1/valheim
# zfs set quota=10G zpool1/valheim

So I guess if your zfs is called zpool1/zfs530 then zfs set quota=10G zpool1/zfs530.

In addition you could also try zfs set reservation=10G zpool1/zfs530 which ensures that 10GB of disk space are reserved for that pool.
Again that is assuming that zpool1 is the name of your zpool and zfs530 the name of the zfs. You can get the actual names of all ZFSs using zfs list.

[diablotf]

I'll give it a go. There isn't an obvious

[pmow]

I use ZFS, using a privileged LXC container with host mounts so it's nice and exposed by the time it gets to docker.

Can you do a touch to the files? Just to make sure you're getting the right permissions there?

[senpo-h]

I can't speak English, so I'll use a translator.
I have a Synology NAS and was in the same situation as you.
Rebooting the NAS seems to have fixed it.
It's hard to see what is under your nose.

[lloesche]

@diablotf were you able to find a solution?

[diablotf]

Unfortunately not.

I've rebooted, and reinstalled container station on the QNAP.
I've attached a 10GB ZFS quota and mounted the container volumes on that.

No joy. Works immediately on the non-ZFS one.

I can't mess around with too much more as it's a production device. Appreciate all the help.

[lloesche]

@diablotf I see. I am extending the README with the information you were able to find. Could you do one last test? What happens if you don't mount /opt/valheim at all? Like, allow the container to be created on a ZFS volume but only add the /config volume.
The disadvantage would be that whenever you update or re-create the container it will have to download the 1 GB Valheim dedicated server from Steam. But I wonder if the issue is just with the /opt/valheim mount being ZFS or if it also happens if the container itself runs on ZFS.

[lloesche]

Added the workaround to the README. Closing this issue as there's nothing more we can do from the container side. If anyone finds new information please open a new issue so we can help others when they encounter this problem.