No Space Left on device

[SSS475]

General description

When attempting to touch in one directory of a mergerfs volume I get No space left on device while other directories work fine.

Expected behavior

All directories on device allow touching a new file

Actual behavior

/warehouse/kids_television$ touch testing                                                                                                                                                                                                                                            
/warehouse/kids_television$ cd ../movies-jr
/warehouse/movies-jr$ touch testing                                                                                                                                                                                                                                                  
touch: cannot touch 'testing': No space left on device

Precise steps to reproduce the behavior

See script above

Explicitly list all steps to reproduce. Preferably create a minimal example of the problem using standard command line tools. The more variables (apps, settings, etc.) that are involved the more difficult it is to debug. Also, please be sure to have read all of the README. It contains a lot of information regarding known system and user issues.

System information

Please provide as much of the following information as possible:

• mergerfs version: mergerfs -V

/warehouse/movies-jr$ mergerfs -V                                                                                                                                                                                                                                                    
mergerfs version: 2.25.1
FUSE library version: 2.9.8-mergerfs
fusermount version: 2.9.8
using FUSE kernel interface version 7.19

• mergerfs settings: cat /etc/fstab or the command line arguments
  Only relevant lines

/dev/disk/by-id/ata-ST4000VN008-2DR166_ZDH34G3S-part1	/mnt/pool/disk1		ext4	defaults	0	2
/dev/disk/by-id/ata-ST4000VN008-2DR166_ZGY20FEZ-part1	/mnt/pool/disk2		ext4	defaults	0	2
/dev/disk/by-id/ata-WDC_WD6002FFWX-68TZ4N0_K1GMG7UD-part1 /mnt/pool/parity ext4 defaults 0 2
/dev/disk/by-id/ata-WDC_WD6002FFWX-68TZ4N0_NCH160MV-part1 /mnt/pool/disk3 ext4 defaults 0 2
/dev/disk/by-id/ata-WDC_WD4003FZEX-00Z4SA0_WD-WMC130D2XTHD-part1 /mnt/pool/disk4 ext4 defaults 0 2
/dev/disk/by-id/ata-WDC_WD30EZRX-00MMMB0_WD-WMAWZ0073154-part1 /mnt/pool/disk5 ext4 defaults 0 2

/mnt/pool/disk* /warehouse fuse.mergerfs category.create=eplfs,moveonenospc=true,defaults,allow_other,minfreespace=20G,fsname=mergerfsPool 0 0

• Linux version: uname -a

Linux bignoisy 4.19.4-arch1-1-ARCH #1 SMP PREEMPT Fri Nov 23 09:06:58 UTC 2018 x86_64 GNU/Linux

• Versions of any additional software being used
  None
• List of drives, filesystems, & sizes: df -h
df -h

dev              32G     0   32G   0% /dev
run              32G  1.3M   32G   1% /run
/dev/sda1       196G  6.3G  180G   4% /
tmpfs            32G     0   32G   0% /dev/shm
tmpfs            32G     0   32G   0% /sys/fs/cgroup
mergerfsPool     20T   14T  5.6T  71% /warehouse
tmpfs            32G     0   32G   0% /tmp
/dev/sda2      1022M   56M  967M   6% /boot
/dev/sdf1       5.5T  5.4T   20G 100% /mnt/pool/disk3
/dev/sdg1       5.5T  5.4T  125G  98% /mnt/pool/parity
/dev/sde1       3.7T  3.6T   17G 100% /mnt/pool/disk4
/dev/sdh1       3.7T  1.3T  2.3T  36% /mnt/pool/disk1
/dev/sdc1       3.7T  1.3T  2.3T  37% /mnt/pool/disk2
/dev/sdd1       2.8T  1.8T  950G  66% /mnt/pool/disk5
tmpfs           6.3G     0  6.3G   0% /run/user/1000

df -i

Filesystem         Inodes    IUsed      IFree IUse% Mounted on
dev               8242220      527    8241693    1% /dev
run               8244151      950    8243201    1% /run
/dev/sda1        13107200    92784   13014416    1% /
tmpfs             8244151        1    8244150    1% /dev/shm
tmpfs             8244151       18    8244133    1% /sys/fs/cgroup
mergerfsPool      5007904   888891    4119013   18% /warehouse
tmpfs             8244151        6    8244145    1% /tmp
/dev/sda2               0        0          0     - /boot
/dev/sdf1         1430816    49275    1381541    4% /mnt/pool/disk3
/dev/sdg1         1430816       12    1430804    1% /mnt/pool/parity
/dev/sde1          953888     6449     947439    1% /mnt/pool/disk4
/dev/sdh1          953888   526284     427604   56% /mnt/pool/disk1
/dev/sdc1          953888    39887     914001    5% /mnt/pool/disk2
/dev/sdd1          715424   266996     448428   38% /mnt/pool/disk5
tmpfs             8244151       13    8244138    1% /run/user/1000

• strace of application having problem: strace -f -o /tmp/app.strace.txt <cmd> or strace -f -p <appPID> -o /tmp/app.strace.txt
• strace of mergerfs while app tried to do it's thing: strace -f -p <mergerfsPID> -o /tmp/mergerfs.strace.txt

685   read(3,  <unfinished ...>
683   read(3,  <unfinished ...>
682   read(3,  <unfinished ...>
674   read(3,  <unfinished ...>
681   read(3,  <unfinished ...>
680   read(3,  <unfinished ...>
678   read(3,  <unfinished ...>
677   read(3,  <unfinished ...>
676   read(3,  <unfinished ...>
675   read(3,  <unfinished ...>
673   read(3,  <unfinished ...>
669   read(3,  <unfinished ...>
672   read(3,  <unfinished ...>
671   read(3,  <unfinished ...>
670   read(3,  <unfinished ...>
668   read(3,  <unfinished ...>
667   read(3,  <unfinished ...>
662   read(3,  <unfinished ...>
661   read(3,  <unfinished ...>
666   read(3,  <unfinished ...>
665   read(3,  <unfinished ...>
664   read(3,  <unfinished ...>
663   read(3,  <unfinished ...>
660   read(3,  <unfinished ...>
657   read(3,  <unfinished ...>
656   read(3,  <unfinished ...>
659   read(3,  <unfinished ...>
658   read(3,  <unfinished ...>
655   read(3,  <unfinished ...>
654   read(3,  <unfinished ...>
653   read(3,  <unfinished ...>
652   futex(0x7ffd2d5f52f0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
684   <... read resumed> "2\0\0\0\1\0\0\0\227\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 50
684   lstat("/mnt/pool/disk1/movies-jr", 0x7fa9c5ed2130) = -1 ENOENT (No such file or directory)
684   lstat("/mnt/pool/disk2/movies-jr", 0x7fa9c5ed2130) = -1 ENOENT (No such file or directory)
684   lstat("/mnt/pool/disk3/movies-jr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
684   lstat("/mnt/pool/disk3/movies-jr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
684   writev(3, [{iov_base="\220\0\0\0\0\0\0\0\227\244\273\0\0\0\0\0", iov_len=16}, {iov_base="\372\371\5\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., iov_len=128}], 2) = 144
684   read(3, "0\0\0\0\"\0\0\0\230\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 48
684   lstat("/mnt/pool/disk1/movies-jr", 0x7fa9c5ed2250) = -1 ENOENT (No such file or directory)
684   lstat("/mnt/pool/disk2/movies-jr", 0x7fa9c5ed2250) = -1 ENOENT (No such file or directory)
684   lstat("/mnt/pool/disk3/movies-jr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
684   faccessat(AT_FDCWD, "/mnt/pool/disk3/movies-jr", X_OK) = 0
684   writev(3, [{iov_base="\20\0\0\0\0\0\0\0\230\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
684   read(3,  <unfinished ...>
682   <... read resumed> "-\0\0\0\1\0\0\0\231\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
682   lstat("/mnt/pool/disk1/movies-jr/.git", 0x7fa9c6f18130) = -1 ENOENT (No such file or directory)
682   lstat("/mnt/pool/disk2/movies-jr/.git", 0x7fa9c6f18130) = -1 ENOENT (No such file or directory)
682   lstat("/mnt/pool/disk3/movies-jr/.git", 0x7fa9c6f18130) = -1 ENOENT (No such file or directory)
682   lstat("/mnt/pool/disk4/movies-jr/.git", 0x7fa9c6f18130) = -1 ENOENT (No such file or directory)
682   lstat("/mnt/pool/disk5/movies-jr/.git", 0x7fa9c6f18130) = -1 ENOENT (No such file or directory)
682   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\231\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
682   read(3,  <unfinished ...>
683   <... read resumed> "-\0\0\0\1\0\0\0\232\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
683   lstat("/mnt/pool/disk1/movies-jr/.git", 0x7fa9c66f5130) = -1 ENOENT (No such file or directory)
683   lstat("/mnt/pool/disk2/movies-jr/.git", 0x7fa9c66f5130) = -1 ENOENT (No such file or directory)
683   lstat("/mnt/pool/disk3/movies-jr/.git", 0x7fa9c66f5130) = -1 ENOENT (No such file or directory)
683   lstat("/mnt/pool/disk4/movies-jr/.git", 0x7fa9c66f5130) = -1 ENOENT (No such file or directory)
683   lstat("/mnt/pool/disk5/movies-jr/.git", 0x7fa9c66f5130) = -1 ENOENT (No such file or directory)
683   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\232\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
683   read(3,  <unfinished ...>
674   <... read resumed> "-\0\0\0\1\0\0\0\233\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
674   lstat("/mnt/pool/disk1/movies-jr/HEAD", 0x7fa9ca7eb130) = -1 ENOENT (No such file or directory)
674   lstat("/mnt/pool/disk2/movies-jr/HEAD", 0x7fa9ca7eb130) = -1 ENOENT (No such file or directory)
674   lstat("/mnt/pool/disk3/movies-jr/HEAD", 0x7fa9ca7eb130) = -1 ENOENT (No such file or directory)
674   lstat("/mnt/pool/disk4/movies-jr/HEAD", 0x7fa9ca7eb130) = -1 ENOENT (No such file or directory)
674   lstat("/mnt/pool/disk5/movies-jr/HEAD", 0x7fa9ca7eb130) = -1 ENOENT (No such file or directory)
674   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\233\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
674   read(3,  <unfinished ...>
681   <... read resumed> "8\0\0\0\3\0\0\0\234\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
681   lstat("/mnt/pool/disk1/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
681   lstat("/mnt/pool/disk1/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
681   writev(3, [{iov_base="x\0\0\0\0\0\0\0\234\244\273\0\0\0\0\0", iov_len=16}, {iov_base="\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0"..., iov_len=104}], 2) = 120
681   read(3,  <unfinished ...>
677   <... read resumed> "-\0\0\0\1\0\0\0\235\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
677   lstat("/mnt/pool/disk1/.git", 0x7fa9c8fa4130) = -1 ENOENT (No such file or directory)
677   lstat("/mnt/pool/disk2/.git", 0x7fa9c8fa4130) = -1 ENOENT (No such file or directory)
677   lstat("/mnt/pool/disk3/.git", 0x7fa9c8fa4130) = -1 ENOENT (No such file or directory)
677   lstat("/mnt/pool/disk4/.git", 0x7fa9c8fa4130) = -1 ENOENT (No such file or directory)
677   lstat("/mnt/pool/disk5/.git", 0x7fa9c8fa4130) = -1 ENOENT (No such file or directory)
677   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\235\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
677   read(3,  <unfinished ...>
680   <... read resumed> "-\0\0\0\1\0\0\0\236\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
680   lstat("/mnt/pool/disk1/.git", 0x7fa9c7f5e130) = -1 ENOENT (No such file or directory)
680   lstat("/mnt/pool/disk2/.git", 0x7fa9c7f5e130) = -1 ENOENT (No such file or directory)
680   lstat("/mnt/pool/disk3/.git", 0x7fa9c7f5e130) = -1 ENOENT (No such file or directory)
680   lstat("/mnt/pool/disk4/.git", 0x7fa9c7f5e130) = -1 ENOENT (No such file or directory)
680   lstat("/mnt/pool/disk5/.git", 0x7fa9c7f5e130) = -1 ENOENT (No such file or directory)
680   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\236\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
680   read(3,  <unfinished ...>
678   <... read resumed> "-\0\0\0\1\0\0\0\237\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
678   lstat("/mnt/pool/disk1/HEAD", 0x7fa9c8781130) = -1 ENOENT (No such file or directory)
678   lstat("/mnt/pool/disk2/HEAD", 0x7fa9c8781130) = -1 ENOENT (No such file or directory)
678   lstat("/mnt/pool/disk3/HEAD", 0x7fa9c8781130) = -1 ENOENT (No such file or directory)
678   lstat("/mnt/pool/disk4/HEAD", 0x7fa9c8781130) = -1 ENOENT (No such file or directory)
678   lstat("/mnt/pool/disk5/HEAD", 0x7fa9c8781130) = -1 ENOENT (No such file or directory)
678   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\237\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
678   read(3,  <unfinished ...>
676   <... read resumed> "-\0\0\0\1\0\0\0\240\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
676   lstat("/mnt/pool/disk1/movies-jr/.git", 0x7fa9c97c7130) = -1 ENOENT (No such file or directory)
676   lstat("/mnt/pool/disk2/movies-jr/.git", 0x7fa9c97c7130) = -1 ENOENT (No such file or directory)
676   lstat("/mnt/pool/disk3/movies-jr/.git", 0x7fa9c97c7130) = -1 ENOENT (No such file or directory)
676   lstat("/mnt/pool/disk4/movies-jr/.git", 0x7fa9c97c7130) = -1 ENOENT (No such file or directory)
676   lstat("/mnt/pool/disk5/movies-jr/.git", 0x7fa9c97c7130) = -1 ENOENT (No such file or directory)
676   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\240\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
676   read(3,  <unfinished ...>
675   <... read resumed> "-\0\0\0\1\0\0\0\241\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
675   lstat("/mnt/pool/disk1/movies-jr/.git", 0x7fa9c9fea130) = -1 ENOENT (No such file or directory)
675   lstat("/mnt/pool/disk2/movies-jr/.git", 0x7fa9c9fea130) = -1 ENOENT (No such file or directory)
675   lstat("/mnt/pool/disk3/movies-jr/.git", 0x7fa9c9fea130) = -1 ENOENT (No such file or directory)
675   lstat("/mnt/pool/disk4/movies-jr/.git", 0x7fa9c9fea130) = -1 ENOENT (No such file or directory)
675   lstat("/mnt/pool/disk5/movies-jr/.git", 0x7fa9c9fea130) = -1 ENOENT (No such file or directory)
675   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\241\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
675   read(3, "-\0\0\0\1\0\0\0\242\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
675   lstat("/mnt/pool/disk1/movies-jr/HEAD", 0x7fa9c9fea130) = -1 ENOENT (No such file or directory)
675   lstat("/mnt/pool/disk2/movies-jr/HEAD", 0x7fa9c9fea130) = -1 ENOENT (No such file or directory)
675   lstat("/mnt/pool/disk3/movies-jr/HEAD", 0x7fa9c9fea130) = -1 ENOENT (No such file or directory)
675   lstat("/mnt/pool/disk4/movies-jr/HEAD", 0x7fa9c9fea130) = -1 ENOENT (No such file or directory)
675   lstat("/mnt/pool/disk5/movies-jr/HEAD", 0x7fa9c9fea130) = -1 ENOENT (No such file or directory)
675   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\242\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
675   read(3,  <unfinished ...>
669   <... read resumed> "-\0\0\0\1\0\0\0\243\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
669   lstat("/mnt/pool/disk1/.git", 0x7fa9ccff0130) = -1 ENOENT (No such file or directory)
669   lstat("/mnt/pool/disk2/.git", 0x7fa9ccff0130) = -1 ENOENT (No such file or directory)
669   lstat("/mnt/pool/disk3/.git", 0x7fa9ccff0130) = -1 ENOENT (No such file or directory)
669   lstat("/mnt/pool/disk4/.git", 0x7fa9ccff0130) = -1 ENOENT (No such file or directory)
669   lstat("/mnt/pool/disk5/.git", 0x7fa9ccff0130) = -1 ENOENT (No such file or directory)
669   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\243\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
669   read(3,  <unfinished ...>
672   <... read resumed> "-\0\0\0\1\0\0\0\244\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
672   lstat("/mnt/pool/disk1/.git", 0x7fa9cb7ed130) = -1 ENOENT (No such file or directory)
672   lstat("/mnt/pool/disk2/.git", 0x7fa9cb7ed130) = -1 ENOENT (No such file or directory)
672   lstat("/mnt/pool/disk3/.git", 0x7fa9cb7ed130) = -1 ENOENT (No such file or directory)
672   lstat("/mnt/pool/disk4/.git", 0x7fa9cb7ed130) = -1 ENOENT (No such file or directory)
672   lstat("/mnt/pool/disk5/.git", 0x7fa9cb7ed130) = -1 ENOENT (No such file or directory)
672   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\244\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
672   read(3,  <unfinished ...>
671   <... read resumed> "-\0\0\0\1\0\0\0\245\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
671   lstat("/mnt/pool/disk1/HEAD", 0x7fa9cbfee130) = -1 ENOENT (No such file or directory)
671   lstat("/mnt/pool/disk2/HEAD", 0x7fa9cbfee130) = -1 ENOENT (No such file or directory)
671   lstat("/mnt/pool/disk3/HEAD", 0x7fa9cbfee130) = -1 ENOENT (No such file or directory)
671   lstat("/mnt/pool/disk4/HEAD", 0x7fa9cbfee130) = -1 ENOENT (No such file or directory)
671   lstat("/mnt/pool/disk5/HEAD", 0x7fa9cbfee130) = -1 ENOENT (No such file or directory)
671   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\245\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
671   read(3,  <unfinished ...>
670   <... read resumed> "0\0\0\0\1\0\0\0\246\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 48
670   lstat("/mnt/pool/disk1/movies-jr/testing", 0x7fa9cc7ef130) = -1 ENOENT (No such file or directory)
670   lstat("/mnt/pool/disk2/movies-jr/testing", 0x7fa9cc7ef130) = -1 ENOENT (No such file or directory)
670   lstat("/mnt/pool/disk3/movies-jr/testing", 0x7fa9cc7ef130) = -1 ENOENT (No such file or directory)
670   lstat("/mnt/pool/disk4/movies-jr/testing", 0x7fa9cc7ef130) = -1 ENOENT (No such file or directory)
670   lstat("/mnt/pool/disk5/movies-jr/testing", 0x7fa9cc7ef130) = -1 ENOENT (No such file or directory)
670   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\246\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
670   read(3, "@\0\0\0#\0\0\0\247\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 64
670   lstat("/mnt/pool/disk1/movies-jr", 0x7fa9cc7ef060) = -1 ENOENT (No such file or directory)
670   lstat("/mnt/pool/disk2/movies-jr", 0x7fa9cc7ef060) = -1 ENOENT (No such file or directory)
670   lstat("/mnt/pool/disk3/movies-jr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
670   lstat("/mnt/pool/disk1/movies-jr", 0x7fa9cc7ef0f0) = -1 ENOENT (No such file or directory)
670   lstat("/mnt/pool/disk2/movies-jr", 0x7fa9cc7ef0f0) = -1 ENOENT (No such file or directory)
670   lstat("/mnt/pool/disk3/movies-jr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
670   statfs("/mnt/pool/disk3", {f_type=EXT2_SUPER_MAGIC, f_bsize=4096, f_blocks=1464673844, f_bfree=34501008, f_bavail=5194308, f_files=1430816, f_ffree=1381541, f_fsid={val=[3622832335, 78668374]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
670   lstat("/mnt/pool/disk4/movies-jr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
670   statfs("/mnt/pool/disk4", {f_type=EXT2_SUPER_MAGIC, f_bsize=4096, f_blocks=976362522, f_bfree=23830202, f_bavail=4291023, f_files=953888, f_ffree=947439, f_fsid={val=[132250216, 1037572905]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
670   lstat("/mnt/pool/disk5/movies-jr", 0x7fa9cc7ef0f0) = -1 ENOENT (No such file or directory)
670   writev(3, [{iov_base="\20\0\0\0\344\377\377\377\247\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
670   read(3,  <unfinished ...>
667   <... read resumed> "0\0\0\0\1\0\0\0\250\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 48
667   lstat("/mnt/pool/disk1/movies-jr/testing", 0x7fa9cdff2130) = -1 ENOENT (No such file or directory)
667   lstat("/mnt/pool/disk2/movies-jr/testing", 0x7fa9cdff2130) = -1 ENOENT (No such file or directory)
667   lstat("/mnt/pool/disk3/movies-jr/testing", 0x7fa9cdff2130) = -1 ENOENT (No such file or directory)
667   lstat("/mnt/pool/disk4/movies-jr/testing", 0x7fa9cdff2130) = -1 ENOENT (No such file or directory)
667   lstat("/mnt/pool/disk5/movies-jr/testing", 0x7fa9cdff2130) = -1 ENOENT (No such file or directory)
667   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\250\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
667   read(3,  <unfinished ...>
662   <... read resumed> "2\0\0\0\1\0\0\0\251\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 50
662   lstat("/mnt/pool/disk1/movies-jr", 0x7fa9d49b5130) = -1 ENOENT (No such file or directory)
662   lstat("/mnt/pool/disk2/movies-jr", 0x7fa9d49b5130) = -1 ENOENT (No such file or directory)
662   lstat("/mnt/pool/disk3/movies-jr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
662   lstat("/mnt/pool/disk3/movies-jr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
662   writev(3, [{iov_base="\220\0\0\0\0\0\0\0\251\244\273\0\0\0\0\0", iov_len=16}, {iov_base="\372\371\5\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., iov_len=128}], 2) = 144
662   read(3,  <unfinished ...>
661   <... read resumed> "-\0\0\0\1\0\0\0\252\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
661   lstat("/mnt/pool/disk1/movies-jr/.git", 0x7fa9d51d8130) = -1 ENOENT (No such file or directory)
661   lstat("/mnt/pool/disk2/movies-jr/.git", 0x7fa9d51d8130) = -1 ENOENT (No such file or directory)
661   lstat("/mnt/pool/disk3/movies-jr/.git", 0x7fa9d51d8130) = -1 ENOENT (No such file or directory)
661   lstat("/mnt/pool/disk4/movies-jr/.git", 0x7fa9d51d8130) = -1 ENOENT (No such file or directory)
661   lstat("/mnt/pool/disk5/movies-jr/.git", 0x7fa9d51d8130) = -1 ENOENT (No such file or directory)
661   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\252\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
661   read(3,  <unfinished ...>
666   <... read resumed> "-\0\0\0\1\0\0\0\253\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
666   lstat("/mnt/pool/disk1/movies-jr/.git", 0x7fa9ce7f3130) = -1 ENOENT (No such file or directory)
666   lstat("/mnt/pool/disk2/movies-jr/.git", 0x7fa9ce7f3130) = -1 ENOENT (No such file or directory)
666   lstat("/mnt/pool/disk3/movies-jr/.git", 0x7fa9ce7f3130) = -1 ENOENT (No such file or directory)
666   lstat("/mnt/pool/disk4/movies-jr/.git", 0x7fa9ce7f3130) = -1 ENOENT (No such file or directory)
666   lstat("/mnt/pool/disk5/movies-jr/.git", 0x7fa9ce7f3130) = -1 ENOENT (No such file or directory)
666   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\253\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
666   read(3,  <unfinished ...>
665   <... read resumed> "-\0\0\0\1\0\0\0\254\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
665   lstat("/mnt/pool/disk1/movies-jr/HEAD", 0x7fa9ceff4130) = -1 ENOENT (No such file or directory)
665   lstat("/mnt/pool/disk2/movies-jr/HEAD", 0x7fa9ceff4130) = -1 ENOENT (No such file or directory)
665   lstat("/mnt/pool/disk3/movies-jr/HEAD", 0x7fa9ceff4130) = -1 ENOENT (No such file or directory)
665   lstat("/mnt/pool/disk4/movies-jr/HEAD", 0x7fa9ceff4130) = -1 ENOENT (No such file or directory)
665   lstat("/mnt/pool/disk5/movies-jr/HEAD", 0x7fa9ceff4130) = -1 ENOENT (No such file or directory)
665   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\254\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
665   read(3,  <unfinished ...>
664   <... read resumed> "8\0\0\0\3\0\0\0\255\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
664   lstat("/mnt/pool/disk1/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
664   lstat("/mnt/pool/disk1/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
664   writev(3, [{iov_base="x\0\0\0\0\0\0\0\255\244\273\0\0\0\0\0", iov_len=16}, {iov_base="\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0"..., iov_len=104}], 2) = 120
664   read(3,  <unfinished ...>
663   <... read resumed> "-\0\0\0\1\0\0\0\256\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
663   setregid(-1, 1000)                = 0
663   setgroups(3, [1000, 978, 1001])   = 0
663   setreuid(-1, 1000)                = 0
663   lstat("/mnt/pool/disk1/.git", 0x7fa9cfff6130) = -1 ENOENT (No such file or directory)
663   lstat("/mnt/pool/disk2/.git", 0x7fa9cfff6130) = -1 ENOENT (No such file or directory)
663   lstat("/mnt/pool/disk3/.git", 0x7fa9cfff6130) = -1 ENOENT (No such file or directory)
663   lstat("/mnt/pool/disk4/.git", 0x7fa9cfff6130) = -1 ENOENT (No such file or directory)
663   lstat("/mnt/pool/disk5/.git", 0x7fa9cfff6130) = -1 ENOENT (No such file or directory)
663   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\256\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
663   read(3,  <unfinished ...>
660   <... read resumed> "-\0\0\0\1\0\0\0\257\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
660   lstat("/mnt/pool/disk1/.git", 0x7fa9d59fb130) = -1 ENOENT (No such file or directory)
660   lstat("/mnt/pool/disk2/.git", 0x7fa9d59fb130) = -1 ENOENT (No such file or directory)
660   lstat("/mnt/pool/disk3/.git", 0x7fa9d59fb130) = -1 ENOENT (No such file or directory)
660   lstat("/mnt/pool/disk4/.git", 0x7fa9d59fb130) = -1 ENOENT (No such file or directory)
660   lstat("/mnt/pool/disk5/.git", 0x7fa9d59fb130) = -1 ENOENT (No such file or directory)
660   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\257\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
660   read(3,  <unfinished ...>
657   <... read resumed> "-\0\0\0\1\0\0\0\260\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
657   lstat("/mnt/pool/disk1/HEAD", 0x7fa9d7264130) = -1 ENOENT (No such file or directory)
657   lstat("/mnt/pool/disk2/HEAD", 0x7fa9d7264130) = -1 ENOENT (No such file or directory)
657   lstat("/mnt/pool/disk3/HEAD", 0x7fa9d7264130) = -1 ENOENT (No such file or directory)
657   lstat("/mnt/pool/disk4/HEAD", 0x7fa9d7264130) = -1 ENOENT (No such file or directory)
657   lstat("/mnt/pool/disk5/HEAD", 0x7fa9d7264130) = -1 ENOENT (No such file or directory)
657   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\260\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
657   read(3,  <unfinished ...>
656   <... read resumed> "-\0\0\0\1\0\0\0\261\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
656   lstat("/mnt/pool/disk1/movies-jr/.git", 0x7fa9d7a87130) = -1 ENOENT (No such file or directory)
656   lstat("/mnt/pool/disk2/movies-jr/.git", 0x7fa9d7a87130) = -1 ENOENT (No such file or directory)
656   lstat("/mnt/pool/disk3/movies-jr/.git", 0x7fa9d7a87130) = -1 ENOENT (No such file or directory)
656   lstat("/mnt/pool/disk4/movies-jr/.git", 0x7fa9d7a87130) = -1 ENOENT (No such file or directory)
656   lstat("/mnt/pool/disk5/movies-jr/.git", 0x7fa9d7a87130) = -1 ENOENT (No such file or directory)
656   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\261\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
656   read(3,  <unfinished ...>
659   <... read resumed> "-\0\0\0\1\0\0\0\262\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
659   lstat("/mnt/pool/disk1/movies-jr/.git", 0x7fa9d621e130) = -1 ENOENT (No such file or directory)
659   lstat("/mnt/pool/disk2/movies-jr/.git", 0x7fa9d621e130) = -1 ENOENT (No such file or directory)
659   lstat("/mnt/pool/disk3/movies-jr/.git", 0x7fa9d621e130) = -1 ENOENT (No such file or directory)
659   lstat("/mnt/pool/disk4/movies-jr/.git", 0x7fa9d621e130) = -1 ENOENT (No such file or directory)
659   lstat("/mnt/pool/disk5/movies-jr/.git", 0x7fa9d621e130) = -1 ENOENT (No such file or directory)
659   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\262\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
659   read(3,  <unfinished ...>
655   <... read resumed> "-\0\0\0\1\0\0\0\263\244\273\0\0\0\0\0\372\371\5\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
655   lstat("/mnt/pool/disk1/movies-jr/HEAD", 0x7fa9d82aa130) = -1 ENOENT (No such file or directory)
655   lstat("/mnt/pool/disk2/movies-jr/HEAD", 0x7fa9d82aa130) = -1 ENOENT (No such file or directory)
655   lstat("/mnt/pool/disk3/movies-jr/HEAD", 0x7fa9d82aa130) = -1 ENOENT (No such file or directory)
655   lstat("/mnt/pool/disk4/movies-jr/HEAD", 0x7fa9d82aa130) = -1 ENOENT (No such file or directory)
655   lstat("/mnt/pool/disk5/movies-jr/HEAD", 0x7fa9d82aa130) = -1 ENOENT (No such file or directory)
655   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\263\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
655   read(3,  <unfinished ...>
658   <... read resumed> "-\0\0\0\1\0\0\0\264\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
658   lstat("/mnt/pool/disk1/.git", 0x7fa9d6a41130) = -1 ENOENT (No such file or directory)
658   lstat("/mnt/pool/disk2/.git", 0x7fa9d6a41130) = -1 ENOENT (No such file or directory)
658   lstat("/mnt/pool/disk3/.git", 0x7fa9d6a41130) = -1 ENOENT (No such file or directory)
658   lstat("/mnt/pool/disk4/.git", 0x7fa9d6a41130) = -1 ENOENT (No such file or directory)
658   lstat("/mnt/pool/disk5/.git", 0x7fa9d6a41130) = -1 ENOENT (No such file or directory)
658   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\264\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
658   read(3,  <unfinished ...>
654   <... read resumed> "-\0\0\0\1\0\0\0\265\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
654   lstat("/mnt/pool/disk1/.git", 0x7fa9d8acd130) = -1 ENOENT (No such file or directory)
654   lstat("/mnt/pool/disk2/.git", 0x7fa9d8acd130) = -1 ENOENT (No such file or directory)
654   lstat("/mnt/pool/disk3/.git", 0x7fa9d8acd130) = -1 ENOENT (No such file or directory)
654   lstat("/mnt/pool/disk4/.git", 0x7fa9d8acd130) = -1 ENOENT (No such file or directory)
654   lstat("/mnt/pool/disk5/.git", 0x7fa9d8acd130) = -1 ENOENT (No such file or directory)
654   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\265\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
654   read(3,  <unfinished ...>
653   <... read resumed> "-\0\0\0\1\0\0\0\266\244\273\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
653   lstat("/mnt/pool/disk1/HEAD", 0x7fa9d92f0130) = -1 ENOENT (No such file or directory)
653   lstat("/mnt/pool/disk2/HEAD", 0x7fa9d92f0130) = -1 ENOENT (No such file or directory)
653   lstat("/mnt/pool/disk3/HEAD", 0x7fa9d92f0130) = -1 ENOENT (No such file or directory)
653   lstat("/mnt/pool/disk4/HEAD", 0x7fa9d92f0130) = -1 ENOENT (No such file or directory)
653   lstat("/mnt/pool/disk5/HEAD", 0x7fa9d92f0130) = -1 ENOENT (No such file or directory)
653   writev(3, [{iov_base="\20\0\0\0\376\377\377\377\266\244\273\0\0\0\0\0", iov_len=16}], 1) = 16
653   read(3,  <detached ...>

[trapexit]

Does that directory exist on a device with sufficient space? You have minfreespace set to 20GB. Two of your drives are below that. Is the dir on one of those?

[SSS475]

It exists on disk3 and disk4

/mnt/pool/disk3/movies-jr:
...<redacted>
/mnt/pool/disk4/movies-jr:
...<redacted>

[SSS475]

I had not been seeing this issue, until after updating to 2.25.1

[trapexit]

Yeah, both are below your threshold.

As the notes for 2.25.0 mention the silent fallback of policies was changed. It didn't work as users expected. You're using the path preservation policy. Only the drives with those directories are considered. Then you put a size limit. So it returns no space.

What behavior would you expect with those settings? If you want all drives considered lfs policy would be the right policy.

[Eetsi123]

I'm experiencing a little bit similar issue (No space left on device error). That is not possible because I am using empty disks. Maybe the problem has something to with arch? I have this issue in a VM and on my old pc both and they both run arch. @SSS475 seems to be also using arch.

[trapexit]

@Eetsi123 If you don't describe your literal setup there is no way for me to assess the situation.

Each policy was made more strict as described in the notes and docs. There are a number of conditions which can cause a branch to be ignored. Only a single error can be returned so they are prioritized. Without knowing exactly your setup I can't tell you which drive triggered which filter.

As I mentioned prior according to his setup and the drive capacities the policy appears to be working as described. Pick the drive with that path and at least minfreespace. Of the 2 paths available nether has enough space. Hence the enospc error.

[trapexit]

If you don't care about which drive gets the file you should not use a path preserving policy as it is by it's nature requiring you to manually manage some of the tree across drives. If you want it to use drives with the path usually but not error when everything fails the tests (as it does now) I'd need to add a new feature to not do path preserving in some cases.

[BigBopper66]

I'm not sure what wasn't working as expected. It worked perfectly fine for me.

[SSS475]

It does indeed work as described in the readme. Just different than the how it previously operated. But, I will tune my settings and get it to a place that works for me. Thanks.