gcsfuse hangs when called from Linux' autofs

[riccardomurri]

Hello,

I'm trying to get gcsfuse running under Linux' autofs autmounter,
so I can access GCS volumes transparently as /net/gcs/VOLUME from a
Google Cloud VM.

Everything works fine if I run the mount command from the shell::

$ sudo mount -t gcsfuse -o allow_other,nosuid,nodev gcp-public-data-landsat /mnt
Calling gcsfuse with arguments: -o rw -o nosuid -o nodev -o allow_other gcp-public-data-landsat /mnt
Using mount point: /mnt
Opening GCS connection...
Opening bucket...
Mounting file system...
File system has been successfully mounted.

$ sudo ls -1 /mnt
index.csv.gz
index.csv.gz-0QQ39Y4953
index.csv.gz-2S6K16H3ZD
...

If I try to make the same from the automounter, gcsfuse just hangs
after outputting the Mounting file system... line. Here's the debug
log of the automounter process::

automount[15928]: attempting to mount entry /net/gcs/gcp-public-data-landsat
automount[15928]: lookup_mount: lookup(program): gcp-public-data-landsat -> -fstype=fuse,allow_other,nosuid,nodev gcsfuse#gcp-public-data-landsat
automount[15928]: lookup_mount: lookup(program): looking up gcp-public-data-landsat
automount[15928]: lookup_mount: lookup(program): gcp-public-data-landsat -> -fstype=gcsfuse,allow_other,implicit_dirs,nosuid,nodev gcp-public-data-landsat
automount[15928]: parse_mount: parse(sun): expanded entry: -fstype=gcsfuse,allow_other,implicit_dirs,nosuid,nodev gcp-public-data-landsat
automount[15928]: parse_mount: parse(sun): gathered options: fstype=gcsfuse,allow_other,implicit_dirs,nosuid,nodev
automount[15928]: parse_mount: parse(sun): dequote("gcp-public-data-landsat") -> gcp-public-data-landsat
automount[15928]: parse_mount: parse(sun): core of entry: options=fstype=gcsfuse,allow_other,implicit_dirs,nosuid,nodev, loc=gcp-public-data-landsat
automount[15928]: sun_mount: parse(sun): mounting root /net/gcs, mountpoint gcp-public-data-landsat, what gcp-public-data-landsat, fstype gcsfuse, options allow_other,implicit_dirs,nosuid,nodev
automount[15928]: do_mount: gcp-public-data-landsat /net/gcs/gcp-public-data-landsat type gcsfuse options allow_other,implicit_dirs,nosuid,nodev using module generic
automount[15928]: mount_mount: mount(generic): calling mkdir_path /net/gcs/gcp-public-data-landsat
automount[15928]: mount_mount: mount(generic): calling mount -t gcsfuse -o allow_other,implicit_dirs,nosuid,nodev gcp-public-data-landsat /net/gcs/gcp-public-data-landsat
automount[15928]: >> Calling gcsfuse with arguments: -o rw -o nosuid -o nodev -o allow_other --implicit-dirs gcp-public-data-landsat /net/gcs/gcp-public-data-landsat
automount[15928]: >> Using mount point: /net/gcs/gcp-public-data-landsat
automount[15928]: >> Opening GCS connection...
automount[15928]: >> Opening bucket...
automount[15928]: >> Mounting file system...

Any process trying to access the automounted directory hangs. Options
allow_other, nosuid, etc. seem to play no role: I get the same
behavior whatever combination of options I use.

Here's the relevant part of the process table::

oot     15928  0.0  0.2 497904  4672 ?        Ssl  00:37   0:00 /usr/sbin/automount --debug --pid-file /var/run/autofs.pid
root     15959  0.0  0.0  22096  1296 ?        S    00:39   0:00  \_ /bin/mount -t gcsfuse -o allow_other,implicit_dirs,nosuid,nodev gcp-public-data-landsat /net/gcs/gcp-public-data-landsat
root     15960  0.0  0.0   3400  1612 ?        Sl   00:39   0:00      \_ /sbin/mount.gcsfuse gcp-public-data-landsat /net/gcs/gcp-public-data-landsat -o rw,nosuid,nodev,allow_other,implicit_dirs
root     15963  0.0  0.3 114612  6388 ?        Sl   00:39   0:00          \_ /usr/bin/gcsfuse -o rw -o nosuid -o nodev -o allow_other --implicit-dirs gcp-public-data-landsat /net/gcs/gcp-public-data-landsat
root     15968  3.8  0.7 202556 13356 ?        Ssl  00:39   0:00              \_ /usr/bin/gcsfuse --foreground -o rw -o nosuid -o nodev -o allow_other --implicit-dirs gcp-public-data-landsat /net/gcs/gcp-public-data-landsat

What am I doing wrong? Any suggestions?

[riccardomurri]

For the record, strace of the gcsfuse --foreground process shows that it's waiting in epoll():

$ sudo strace -p 16082
strace: Process 16082 attached
epoll_wait(5, ^Cstrace: Process 16082 detached
 <detached ...>

$ sudo lsof -p 16082
COMMAND   PID USER   FD      TYPE DEVICE SIZE/OFF  NODE NAME
gcsfuse 16082 root  cwd       DIR    8,1     4096     2 /
gcsfuse 16082 root  rtd       DIR    8,1     4096     2 /
gcsfuse 16082 root  txt       REG    8,1 10644970 65089 /usr/bin/gcsfuse
gcsfuse 16082 root  mem       REG    8,1    47600  2016 /lib/x86_64-linux-gnu/libnss_files-2.23.so
gcsfuse 16082 root  mem       REG    8,1    47648  2023 /lib/x86_64-linux-gnu/libnss_nis-2.23.so
gcsfuse 16082 root  mem       REG    8,1    93128  2000 /lib/x86_64-linux-gnu/libnsl-2.23.so
gcsfuse 16082 root  mem       REG    8,1    35688  2011 /lib/x86_64-linux-gnu/libnss_compat-2.23.so
gcsfuse 16082 root  mem       REG    8,1  1868984  2003 /lib/x86_64-linux-gnu/libc-2.23.so
gcsfuse 16082 root  mem       REG    8,1   138696  2002 /lib/x86_64-linux-gnu/libpthread-2.23.so
gcsfuse 16082 root  mem       REG    8,1   162632  2001 /lib/x86_64-linux-gnu/ld-2.23.so
gcsfuse 16082 root    0r      CHR    1,3      0t0     6 /dev/null
gcsfuse 16082 root    1w      CHR    1,3      0t0     6 /dev/null
gcsfuse 16082 root    2w      CHR    1,3      0t0     6 /dev/null
gcsfuse 16082 root    3w     FIFO   0,12      0t0 62979 pipe
gcsfuse 16082 root    5u  a_inode   0,13        0  9584 [eventpoll]

[riccardomurri]

Might be the same issue as jacobsa/fuse#22

[riccardomurri]

I think the root cause of this is explained here: e2cd3dc#r18826869

[hradec]

Is there a fix or workaround for this? I'm running in the same problem.
Tried using a mount.gcsfuse wrapper script, running gcsfuse in foreground with nohup, but no luck.

[avidullu]

Duplicate of #358

[avidullu]

Closing this one in preference to #358 which is essentially the same issue.