Weed mount not showing changes happening on another node

[yoobles]

Describe the bug

I am trying to use seaweedfs as a distributed filesystem. Essentially the seaweedfs mounted folder should more or less be the same on all hosts. Change a file on one server, the changes should be visible on the other server in a a few seconds.

In my testing setup it seems like file creation and deletion sync properly but changes to a file don't. The master server sees changes but the other mounts don't see the changes.

System Setup

• List the command line to start "weed master", "weed volume", "weed filer", "weed s3", "weed mount".

Create the master, filer, and volume server as specified in wiki: https://github.com/chrislusf/seaweedfs/wiki/Directories-and-Files

weed server -filer=true

Create a mount in server1:

mkdir -p /mnt/seaweed
weed mount -filer=localhost:8888 -dir=/mnt/seaweed/ -filer.path=/

Create a mount in server2:

mkdir -p /mnt/seaweed
weed mount -filer=server1ip:8888 -dir=/mnt/seaweed/ -filer.path=/

Create a file on server1:

echo "testing" > /mnt/seaweed/testing.txt

Edit file on server2:

nano /mnt/seaweed/testing.txt
insert changes

See differences in files on both servers:

server2:

# cat testing.txt
testing

testing2

server1:

# cat testing.txt
testing

But if I dig around in the actual master server the changes do appear to be there.

curl http://server1:8080/5,3f6f9515ff

testing

testing2

• OS version

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:        18.04
Codename:       bionic

• output of weed version

root@server1:~# weed version
version 30GB 2.42 84312e6 linux amd64

• if using filer, show the content of filer.toml

[leveldb2]
# local on disk, mostly for simple single-machine setup, fairly scalable
# faster than previous leveldb, recommended.
enabled = true
dir = "./filerldb2"                                     # directory to store level db files

Expected behavior

The files should be synced eventually. Even if I wait a few minutes, the files never seem to sync up. The only thing that syncs them is unmount and then remounting. Or deleting and creating a new file.

It seems like seaweedfs should support this, but let me know if I misunderstood something.

[chrislusf]

I can not reproduce this.

[yoobles]

Is there any more info I can give you that might be useful?

Perhaps this is a kernel issue caching files too heavily? I noticed there were some other issues referencing similar things.

[chrislusf]

You can use https://github.com/chrislusf/seaweedfs/blob/master/docker/Makefile#L24 as an example to create reproducible docker setup.

[yoobles]

This didn't work in local-mount-compose.yml either unfortunately.

Setting it up and then doing

docker exec -it seaweedfs_mount_1_1 sh
docker exec -it seaweedfs_mount_2_1 sh

Added vim to both containers and then created a test.txt file on mount_1 and editted it on mount_2:

I had to make a minor change to get it to run on my Mac though:

diff --git a/docker/compose/local-mount-compose.yml b/docker/compose/local-mount-compose.yml
index b1c579cd..8c432905 100644
--- a/docker/compose/local-mount-compose.yml
+++ b/docker/compose/local-mount-compose.yml
@@ -38,7 +38,7 @@ services:
   mount_2:
     image: chrislusf/seaweedfs:local
     privileged: true
-    entrypoint: '/bin/sh -c "mkdir -p t2 && mkdir -p cache/t2 && weed -v=4  mount -filer=filer:8888 -cacheDir=./cache/t2 -dir=./t2 -filer.path=/c1 -volumeServerAcess=publicUrl"'
+    entrypoint: '/bin/sh -c "mkdir -p t2 && mkdir -p cache/t2 && weed -v=4  mount -filer=filer:8888 -cacheDir=./cache/t2 -dir=./t2 -filer.path=/c1"'
     depends_on:
       - master
       - volume

[chrislusf]

Added a fix. Please help to verify the fix.

[yoobles]

Hi there, unfortunately I don't think that worked. Same setup with docker-compose:

Created the file in t1 first, then editted it in t2, then viewed it in t1 again.

[yoobles]

I think I was able to fix this by downgrading to 9672f9e but I need to test further. Happy to test any other fixes you put up though.

[chrislusf]

can not reproduce. Please check the version.

[yoobles]

This still didn't work for me on Docker. I ran make clean on in the repo root directory as well as in the docker folder. I also deleted all containers, and images.

I'll test with one of the binaries that are built on dev now.

[yoobles]

Okay perhaps the Docker image is caching something? This works properly with https://github.com/chrislusf/seaweedfs/releases/download/dev/weed-2021-05-01-05-52-linux-amd64.tar.gz running on actual hardware. Thanks for the quick fix!