Trying to use Git on seaweedfs FUSE encounters arbitrary errors

[segevfiner]

Describe the bug

Trying to do the following using the default docker-compose setup causes Git to throw the following arbitrary errors due to unexpected filesystem syscalls failures, such as ENOENT on getcwd. When some of those happen, even the shell calling Git often becomes confused about what its current pwd is, ending up showing it as (unknown)

# After starting the docker-compose setup
docker run --rm -it --network seaweedfs_default --device /dev/fuse --cap-add SYS_ADMIN --entrypoint /bin/sh chrislusf/seaweedfs
apk add fuse git
mkdir /mnt/seaweedfs
weed mount -filer filer:8888 -dir /mnt/seaweedfs &
cd /mnt/seaweedfs
git config --global user.name test
git config --global user.email test@test.test
git init test
echo Hello, World! > hello.txt
git add hello.txt
git commit
cd ..
touch foo
git init test2

Errors:

fatal: not a git repository (or any parent up to mount point /mnt)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

fatal: unable to get current working directory: No such file or directory
sh: getcwd: No such file or directory

System Setup

• The default docker-compose setup from here: docker/seaweedfs-compose.yml
• The default from the docker-compose setup: Alpine 3.10.2, on top of Docker for Mac 2.1.0.5 on macOS 10.14.6.
• output of weed version: version 30GB 1.50 linux amd64
• if using filer, show the content of filer.toml

Expected behavior
Git should work correctly like it does on a local disk.

[chrislusf]

For the first error, export GIT_DISCOVERY_ACROSS_FILESYSTEM=1

[segevfiner]

For the first error, export GIT_DISCOVERY_ACROSS_FILESYSTEM=1

Thanks for looking into this!

I haven't got around to testing yet, but AFAIK this shouldn't be necessary and won't fix the problem. The Git repository is inside the seaweedfs mount, this isn't the case of a Git repository that has a mount inside of the repository and running Git from inside that mount.

What's happening is that for some reason Git fails to detect the repository, something in it either got corrupted (Inside the .git directory) or it has hit some filesystem syscall error that caused it to skip it. It then proceeds to the parent directory, eventually hitting /mnt, which is outside the seaweedfs mount, and giving up. Setting GIT_DISCOVERY_ACROSS_FILESYSTEM will just make it continue searching /mnt and / which won't help here as it has already skipped the Git repository it should have detected earlier.

I think strace and perhaps debug logging of the FUSE driver should reveal what is failing.

The code in Git that does the discovery is here: git/git/setup.c:957, and this function is used to check if candidate/suspect is a Git directory: git/git/setup.c:311.

[chrislusf]

Thanks for the details!

I got

fatal: unable to get current working directory: No such file or directory
sh: getcwd: No such file or directory

The error "unable to get current working directory" happens here: https://github.com/git/git/blob/53f9a3e157dbbc901a02ac2c73346d375e24978c/abspath.c#L105
Could be related to this?
docker/for-mac#1509 It seems there are some way out of this.

[segevfiner]

I haven't used Docker for Mac bind mounts when testing this, and this happened on top of the seaweedfs mount anyhow, which shouldn't be going through such bind mounts. In the seaweedfs case, this happens quite deterministicly for me while I haven't had any such issue with bind mounts yet.

[chrislusf]

Found the problem. There is a rename operation and the SeaweedFS mount is not clearing the old item cache.

[chrislusf]

hmm, still

fatal: not a git repository (or any of the parent directories): .git

[chrislusf]

works on docker. seems still some problem on Mac

[segevfiner]

Strange. With my instructions it's supposed to be just a normal mount inside a Linux VM not some Docker for Mac magical bind mount. Since it is a Linux VM it has its own kernel running so maybe the kernel version and/or FUSE version plays a part here.

[chrislusf]

When git use mmap to update a file, the file content is not flushed in time. see https://github.com/git/git/blob/master/config.c#L2722 and osxfuse/osxfuse#202

This caused an additional file .git/config.lock and a minimum .git/config file.

[segevfiner]

Now I understand that you were talking about running on macOS directly with OSXFUSE when you meant there is still an issue on Mac.

[segevfiner]

Using the same setup with version 1.51 the instructions in the PR description seem to work. I then tried: git clone https://github.com/moby/moby.git to test a clone of a moderately large repo and got:

error: inflate: data stream error (unknown compression method)
fatal: serious inflate inconsistency
fatal: index-pack failed

[chrislusf]

able to git clone https://github.com/chrislusf/teeproxy.git now. but have error on git clone https://github.com/moby/moby.git

[chrislusf]

Mostly working now. For small~medium repo, everything works. For large ones, often I got "File exists" error.

From the logging, it seems a race condition. Usually there are 3 file existence checking followed by a file creation. But sometimes the file creation happens first and causing this. git checkout -f HEAD usually will fix the clone problems.

You can use 1.52 to try it out.

git clone https://github.com/moby/moby.git /mnt/seaweedfs/moby
Cloning into '/mnt/seaweedfs/moby'...
remote: Enumerating objects: 282589, done.
remote: Total 282589 (delta 0), reused 0 (delta 0), pack-reused 282589
Receiving objects: 100% (282589/282589), 146.46 MiB | 9.43 MiB/s, done.
Resolving deltas: 100% (192251/192251), done.
error: unable to create file vendor/github.com/docker/distribution/manifest/ocischema/builder.go: File exists
error: unable to create file vendor/github.com/opencontainers/runc/libcontainer/configs/namespaces_syscall.go: File exists
error: unable to create file vendor/golang.org/x/sys/unix/zerrors_linux_arm64.go: File exists
Checking out files: 100% (5777/5777), done.
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

[chrislusf]

After commit 892e726 , the git clone for large repo works fine now.

This is because reusing the gRPC context object for too many operations caused the object to be cancelled for unknown reason.