Incorrect link count for files

[terribleplan]

Describe the bug
Under some conditions an incorrect number of (hard) links is stored and returned for files. I was able to reproduce this issue by deleting the original file on a different mount from the one that made the link, then after understanding what was going on using a single machine and remounting the fs.

System Setup

start commands:

weed master -defaultReplication 010 -ip monolith -ip.bind <monolith-ip> -mdir /path/to/seaweedfs/master/metadata -port 9333 -volumePreallocate -volumeSizeLimitMB 64000

weed filer -collection common-filer -dataCenter dc1 -defaultReplicaPlacement 010 -encryptVolumeData -filerGroup common-filer -ip monolith -ip.bind <monolith-ip> -master monolith:9333 -port 8888

weed volume -dataCenter dc1 -dir /mnt/disk1/volume -disk hdd -index leveldb -ip volserv1 -ip.bind <volserv1-ip> -max 0 -minFreeSpace 100GiB -mserver monolith:9333 -port 8081 -rack volserv1 -readMode redirect
(many commands similar to the above volume command are run on additional volume servers)

mount is through fstab entry:

fuse /mnt/weed fuse.weed filer=monolith:8888,collection=common-filer 0 0

• OS version:

master/filer: Ubuntu 16.04.7 LTS

volume: Ubuntu 20.04.4 LTS

mount: Ubuntu 20.04.4 LTS

• output of weed version:

master/filer/mount: version 8000GB 3.14 3c79c770562ef4f7c0d4e57a88f616eb3671b9cd linux amd64

volume: version 8000GB 3.14 3c79c770562ef4f7c0d4e57a88f616eb3671b9cd linux arm64

• if using filer, show the content of filer.toml:

[filer.options]
recursive_delete = false
[leveldb2]
enabled = true
dir = "/path/to/filer/leveldb2"
# todo: consider sqlite or cockroach

Expected behavior
When deleting a file that is a hard link the number of links is correctly reflected in the remaining file that shared an inode.

Screenshots

Not screenshots, but here is a simplified "log" of steps to reproduce the issue:

# set up linked files
user@box1:/mnt/weed/tmp$ touch test1
user@box1:/mnt/weed/tmp$ ln test1 test2
user@box1:/mnt/weed/tmp$ stat test1 test2
  File: test1
Device: 35h/53d Inode: 12351762350232632967  Links: 2
  File: test2
Device: 35h/53d Inode: 12351762350232632967  Links: 2
# delete one of the linked files, doesn't matter which
user@box1:/mnt/weed/tmp$ rm test2
# things _appear_ ok initially
user@box1:/mnt/weed/tmp$ stat test1
  File: test1
Device: 35h/53d Inode: 12351762350232632967  Links: 1
# but once you restart the mount
user@box1:/mnt# umount /mnt/weed && mount -a
# you can see the bad result
user@box1:/mnt/weed/tmp$ stat test1
  File: test1
  Size: 0               Blocks: 0          IO Block: 512    regular empty file
Device: 43h/67d Inode: 12351762350232632967  Links: 2
# and there are no other files with that inode on the whole filer
user@box1:/mnt/weed# find /mnt/weed -type f -inum 12351762350232632967
/mnt/weed/tmp/test1

Additional context

I know hard links are not used super widely, but they are very useful for some of the operations I am doing that would result in the unnecessary duplication of a large file many times.

I originally discovered the issue while using multiple mounts, which displays similar behavior to the simpler reproduction steps above (but is slightly more confusing until you realize you need to remount to see the true error):

# create a file, and create a hard link of it on one box
user@box1:/mnt/weed/tmp$ touch tmp1
user@box1:/mnt/weed/tmp$ ln tmp1 tmp2
user@box1:/mnt/weed/tmp$ stat tmp1 tmp2
  File: tmp1
Device: 35h/53d Inode: 14069293879345266307  Links: 2
  File: tmp2
Device: 35h/53d Inode: 14069293879345266307  Links: 2

# we delete one of the files on the second box
user@box2:/mnt/weed/tmp$ stat tmp1 tmp2
  File: tmp1
Device: 4fh/79d Inode: 14069293879345266307  Links: 2
  File: tmp2
Device: 4fh/79d Inode: 14069293879345266307  Links: 2
user@box2:/mnt/weed/tmp$ rm tmp1
user@box2:/mnt/weed/tmp$ stat tmp2
  File: tmp2
Device: 4fh/79d Inode: 14069293879345266307  Links: 1

# back on the original box things are very wrong
user@box1:/mnt/weed/tmp$ stat tmp2
  File: tmp2
Device: 35h/53d Inode: 14069293879345266307  Links: 2

# doesn't change on the first box after re-mount
user@box1:/mnt# umount /mnt/weed && mount -a
user@box1:/mnt/weed/tmp$ stat tmp2
  File: tmp2
Device: c0h/192d        Inode: 14069293879345266307  Links: 2

# and things are actually more broken on the second box after remount
user@edh5:/mnt# umount /mnt/weed && mount -a
user@edh5:/mnt/weed/tmp# stat tmp2
  File: tmp2
Device: 35h/53d Inode: 14069293879345266307  Links: 2

[chrislusf]

what is the problem you are facing if the link count is not correct?

[terribleplan]

One of the simpler ways I use the link count is I have one directory of source files that are haphazardly named (which came that way from the source, and renaming them would cause other problems), and I maintain a "mapping" of those files into a directory with standardized names for consumption by other tools that expect their input to be organized a certain way. Sometimes a higher quality version of files are added to the source directory which will take priority next time the mapping is generated (and the lower quality version is deleted). Sometimes mappings are added manually from source files not in the managed source file directory. When the mapping is run and an output mapped file is not the expected one (based on inode) the link count is used to determine whether the output file is dangling due to an older source being deleted (and can safely be deleted), or whether manual intervention is needed to either revise the mapping (because the better source is in some unmanaged location) or to delete the linked output file (because the better source is in the input directory).

[chrislusf]

thanks for the explanation! I added a fix. Please use the dev version to test.

[terribleplan]

Awesome, I'll test in the next day or so when I get a chance. It looks like this won't address files that already exist in a bad state (of which I have several), any advice on how to do that without recreating the files? (Or am I missing something in how the code works?)

[chrislusf]

hard to fix existing data unless write a program specifically to fix this.

[terribleplan]

I figured as much, just thought I'd ask. Looks like the fix worked, I look forward to this making it into the next release. Also, congrats on the move to seaweedfs/seaweedfs.

root@testserver:/mnt/testweed/tmp$ touch test1
root@testserver:/mnt/testweed/tmp$ ln test1 test2
root@testserver:/mnt/testweed/tmp$ stat test1 test2
  File: test1
Device: 4fh/79d Inode: 12351762350239143302  Links: 2
  File: test2
Device: 4fh/79d Inode: 12351762350239143302  Links: 2
root@testserver:/mnt/testweed/tmp$ rm test1
root@testserver:/mnt/testweed/tmp$ stat test2
  File: test2
Device: 4fh/79d Inode: 12351762350239143302  Links: 1
root@testserver:/mnt$ umount /mnt/testweed
root@testserver:/mnt$ mount -a
root@testserver:/mnt/testweed/tmp$ stat test2
  File: test2
Device: 4eh/78d Inode: 12351762350239143302  Links: 1