gcsfuse crashes with "panic: inode 8 is *inode.explicitDirInode, wanted *inode.FileInode"

[steliosrammos]

Setup

I am mounting a Cloud Storage bucket on a Compute Engine VM with the following command:

gcsfuse --uid 1002 --gid 1003 --implicit-dirs --foreground --debug_invariants --debug_gcs --debug_fs --debug_fuse --limit-ops-per-sec -1 --only-dir bitcoin/blocks bitcoin-blockchain-data /home/satoshi/.bitcoin/blocks

I use the Cloud Storage to store the Bitcoin blockchain data which is accessed by the bitcoind program. I use the same user (1002) for mounting and for running the program. The service account has the following permissions: Compute Storage Admin, Storage Admin, Storage Object Admin, Storage Object Creator.

Problem

GCSFuse crashes with "panic: inode 8 is *inode.explicitDirInode, wanted *inode.FileInode" and I can't find the issue. The debug flags have not been very helpful but maybe I am not reading them properly.

Here's a sample from the logs (with final error):

debug_fs: 2021/11/19 12:07:41.493380 LookUpInode(8, "000003.log"): no such file or directory
fuse_debug: 2021/11/19 12:07:41.493433 Op 0x000000b2        connection.go:500] -> Error: "no such file or directory"
fuse_debug: 2021/11/19 12:07:41.493598 Op 0x000000b4        connection.go:416] <- CreateFile (parent 8, name "000003.log", PID 114892)
gcs: 2021/11/19 12:07:41.493797 Req             0x3f: <- CreateObject("bitcoin/blocks/index/000003.log")
gcs: 2021/11/19 12:07:41.602730 Req             0x3f: -> CreateObject("bitcoin/blocks/index/000003.log") (108.951104ms): OK
debug_fs: 2021/11/19 12:07:41.602834 CreateFile(8, "000003.log"): <nil>
fuse_debug: 2021/11/19 12:07:41.602921 Op 0x000000b4        connection.go:498] -> OK (inode 14)
fuse_debug: 2021/11/19 12:07:41.603118 Op 0x000000b6        connection.go:416] <- LookUpInode (parent 1, name "index", PID 114892)
gcs: 2021/11/19 12:07:41.603266 Req             0x40: <- ListObjects("bitcoin/blocks/index/")
gcs: 2021/11/19 12:07:41.624950 Req             0x40: -> ListObjects("bitcoin/blocks/index/") (21.671157ms): OK
debug_fs: 2021/11/19 12:07:41.625071 LookUpInode(1, "index"): <nil>
fuse_debug: 2021/11/19 12:07:41.625130 Op 0x000000b6        connection.go:498] -> OK (inode 8)
fuse_debug: 2021/11/19 12:07:41.625264 Op 0x000000b8        connection.go:416] <- LookUpInode (parent 8, name "MANIFEST-000002", PID 114892)
gcs: 2021/11/19 12:07:41.625451 Req             0x41: <- ListObjects("bitcoin/blocks/index/MANIFEST-000002/")
gcs: 2021/11/19 12:07:41.625525 Req             0x42: <- StatObject("bitcoin/blocks/index/MANIFEST-000002")
gcs: 2021/11/19 12:07:41.625594 Req             0x43: <- StatObject("bitcoin/blocks/index/MANIFEST-000002/")
2021/11/19 12:07:41.642176 Not retrying StatObject("bitcoin/blocks/index/MANIFEST-000002") after error of type *gcs.NotFoundError ("gcs.NotFoundError: googleapi: Error 404: No such object: bitcoin-blockchain-data/bi
tcoin/blocks/index/MANIFEST-000002, notFound"): &gcs.NotFoundError{Err:(*googleapi.Error)(0xc0002812c0)}
gcs: 2021/11/19 12:07:41.642401 Req             0x42: -> StatObject("bitcoin/blocks/index/MANIFEST-000002") (16.867663ms): gcs.NotFoundError: googleapi: Error 404: No such object: bitcoin-blockchain-data/bitcoin/blo
cks/index/MANIFEST-000002, notFound
gcs: 2021/11/19 12:07:41.643307 Req             0x41: -> ListObjects("bitcoin/blocks/index/MANIFEST-000002/") (17.847765ms): OK
2021/11/19 12:07:41.666337 Not retrying StatObject("bitcoin/blocks/index/MANIFEST-000002/") after error of type *gcs.NotFoundError ("gcs.NotFoundError: googleapi: Error 404: No such object: bitcoin-blockchain-data/b
itcoin/blocks/index/MANIFEST-000002/, notFound"): &gcs.NotFoundError{Err:(*googleapi.Error)(0xc000281320)}
gcs: 2021/11/19 12:07:41.666569 Req             0x43: -> StatObject("bitcoin/blocks/index/MANIFEST-000002/") (40.965097ms): gcs.NotFoundError: googleapi: Error 404: No such object: bitcoin-blockchain-data/bitcoin/bl
ocks/index/MANIFEST-000002/, notFound
debug_fs: 2021/11/19 12:07:41.666723 LookUpInode(8, "MANIFEST-000002"): no such file or directory
fuse_debug: 2021/11/19 12:07:41.666869 Op 0x000000b8        connection.go:500] -> Error: "no such file or directory"
fuse_debug: 2021/11/19 12:07:41.667053 Op 0x000000ba        connection.go:416] <- CreateFile (parent 8, name "MANIFEST-000002", PID 114892)
gcs: 2021/11/19 12:07:41.667245 Req             0x44: <- CreateObject("bitcoin/blocks/index/MANIFEST-000002")
gcs: 2021/11/19 12:07:41.852335 Req             0x44: -> CreateObject("bitcoin/blocks/index/MANIFEST-000002") (185.107493ms): OK
debug_fs: 2021/11/19 12:07:41.852461 CreateFile(8, "MANIFEST-000002"): <nil>
fuse_debug: 2021/11/19 12:07:41.852508 Op 0x000000ba        connection.go:498] -> OK (inode 15)
fuse_debug: 2021/11/19 12:07:41.852788 Op 0x000000bc        connection.go:416] <- GetXattr (inode 15, name "security.capability", PID 114892, name security.capability)
debug_fs: 2021/11/19 12:07:41.852869 GetXattr(15, security.capability): no data available
fuse_debug: 2021/11/19 12:07:41.852893 Op 0x000000bc        connection.go:500] -> Error: "no data available"
fuse_debug: 2021/11/19 12:07:41.852995 Op 0x000000be        connection.go:416] <- GetXattr (inode 15, name "security.capability", PID 114892, name security.capability)
debug_fs: 2021/11/19 12:07:41.853080 GetXattr(15, security.capability): no data available
fuse_debug: 2021/11/19 12:07:41.853109 Op 0x000000be        connection.go:500] -> Error: "no data available"
fuse_debug: 2021/11/19 12:07:41.853186 Op 0x000000c0        connection.go:416] <- LookUpInode (parent 1, name "index", PID 114892)
gcs: 2021/11/19 12:07:41.853347 Req             0x45: <- ListObjects("bitcoin/blocks/index/")
gcs: 2021/11/19 12:07:41.877126 Req             0x45: -> ListObjects("bitcoin/blocks/index/") (23.770409ms): OK
debug_fs: 2021/11/19 12:07:41.877252 LookUpInode(1, "index"): <nil>
fuse_debug: 2021/11/19 12:07:41.877326 Op 0x000000c0        connection.go:498] -> OK (inode 8)
fuse_debug: 2021/11/19 12:07:41.877430 Op 0x000000c2        connection.go:416] <- OpenDir (inode 8, PID 114892)
debug_fs: 2021/11/19 12:07:41.877539 OpenDir(8): <nil>
fuse_debug: 2021/11/19 12:07:41.877564 Op 0x000000c2        connection.go:498] -> OK ()
fuse_debug: 2021/11/19 12:07:41.877812 Op 0x000000c4        connection.go:416] <- SyncFile (inode 8, PID 114892)
panic: inode 8 is *inode.explicitDirInode, wanted *inode.FileInode
goroutine 349 [running]:
github.com/googlecloudplatform/gcsfuse/internal/fs.(*fileSystem).fileInodeOrDie(0xc000400f18, 0x0)
        /tmp/build_gcsfuse_gopath3443182161/src/github.com/googlecloudplatform/gcsfuse/internal/fs/fs.go:953 +0xd9
github.com/googlecloudplatform/gcsfuse/internal/fs.(*fileSystem).SyncFile(0xc000400ea0, {0xcf26f8, 0xc000510ed0}, 0xc0003e9320)
        /tmp/build_gcsfuse_gopath3443182161/src/github.com/googlecloudplatform/gcsfuse/internal/fs/fs.go:1752 +0x71
github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*debugLogging).SyncFile(0xc0003be7f8, {0xcf26f8, 0xc000510ed0}, 0xc0003e9320)
        /tmp/build_gcsfuse_gopath3443182161/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/debug_logging.go:199 +0x45
github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*errorMapping).SyncFile(0xc0003be810, {0xcf26f8, 0xc000510ed0}, 0xcf2650)
        /tmp/build_gcsfuse_gopath3443182161/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/error_mapping.go:238 +0x36
github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*monitoring).SyncFile(0xc0000f5120, {0xcf26f8, 0xc000510ed0}, 0xc00042f1a0)
        /tmp/build_gcsfuse_gopath3443182161/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/monitoring.go:287 +0x64
github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0xc00049bed8, 0xc0000a1380, {0xcf26f8, 0xc000510ed0}, {0xa9c4c0, 0xc0003e9320})
        /tmp/build_gcsfuse_gopath3443182161/src/github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil/file_system.go:197 +0x8c2
created by github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps
        /tmp/build_gcsfuse_gopath3443182161/src/github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil/file_system.go:122 +0x205

Environment

The VM is running an Ubuntu 20.04 instance.