Multiple cvmfs processes after concurrent mounts

[vvolkl]

@traylenator @vaneldik As reported by CERN batch computing admins, attempting to mount the same repository multiple times simultaneously can result in multiple cvmfs processes and a stuck mount helper process:

#  ps -efwww|grep mount
cvmfs      23863       1  0 Jul05 ?        00:04:35 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch
cvmfs     582755       1  0 08:26 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 atlas.cern.ch /cvmfs/atlas.cern.ch
cvmfs     582758       1  0 08:26 ?        00:00:26 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 atlas.cern.ch /cvmfs/atlas.cern.ch
cvmfs     583011       1  0 08:26 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 sft.cern.ch /cvmfs/sft.cern.ch
cvmfs     583015       1  0 08:26 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 sft.cern.ch /cvmfs/sft.cern.ch
cvmfs     654340       1  0 08:38 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 atlas-condb.cern.ch /cvmfs/atlas-condb.cern.ch
cvmfs     654344       1  0 08:38 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 atlas-condb.cern.ch /cvmfs/atlas-condb.cern.ch
root      896660       1  0 Aug21 ?        00:01:46 /usr/sbin/automount --systemd-service --dont-check-daemon
cvmfs     906651       1  0 Aug21 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch
root     1347748 1347408  0 10:52 pts/0    00:00:00 grep --color=auto mount
cvmfs    1821240       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 singularity.opensciencegrid.org /cvmfs/singularity.opensciencegrid.org
cvmfs    1821243       1  0 Sep09 ?        00:20:04 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 singularity.opensciencegrid.org /cvmfs/singularity.opensciencegrid.org
cvmfs    1821318       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 oasis.opensciencegrid.org /cvmfs/oasis.opensciencegrid.org
cvmfs    1821322       1  0 Sep09 ?        00:01:46 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 oasis.opensciencegrid.org /cvmfs/oasis.opensciencegrid.org
cvmfs    1825353       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 dune.opensciencegrid.org /cvmfs/dune.opensciencegrid.org
cvmfs    1825357       1  0 Sep09 ?        00:00:50 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 dune.opensciencegrid.org /cvmfs/dune.opensciencegrid.org
root     1964144  896660  0 Sep09 ?        00:00:00 /usr/bin/mount -t cvmfs larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
root     1964145  896660  0 Sep09 ?        00:00:00 /usr/bin/mount -t cvmfs larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
root     1964147 1964144  2 Sep09 ?        01:04:52 /sbin/mount.cvmfs larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org -o rw
root     1964148 1964145  2 Sep09 ?        01:04:51 /sbin/mount.cvmfs larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org -o rw
root     1964153  896660  0 Sep09 ?        00:00:00 /usr/bin/mount -t cvmfs larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
root     1964155 1964153  2 Sep09 ?        01:04:49 /sbin/mount.cvmfs larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org -o rw
cvmfs    1964322 1964155  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964323 1964147  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964329 1964148  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964566       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964575       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964583       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964586       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964592       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964596       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964601       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964608       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964612       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964630       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964791       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1964951       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    1965847       1  0 Sep09 ?        00:03:12 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
cvmfs    2248826       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 grid.cern.ch /cvmfs/grid.cern.ch
cvmfs    2248831       1  0 Sep09 ?        00:00:07 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 grid.cern.ch /cvmfs/grid.cern.ch
cvmfs    2259602       1  0 Sep09 ?        00:00:00 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 cms.cern.ch /cvmfs/cms.cern.ch
cvmfs    2259606       1  7 Sep09 ?        03:21:33 /usr/bin/cvmfs2 -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=990,gid=989 cms.cern.ch /cvmfs/cms.cern.ch

The issue seems to be that CheckConcurrentMount checks for the repository socket, which is only created later. When using fuse2, the subsequent mount helpers would later fail with:

fuse: mountpoint is not empty
fuse: if you are sure this is safe, use the 'nonempty' mount option
failed to create Fuse channel
(monitor) monitor stopped    [09-13-2023 11:36:28 CEST]

but fuse3 apparently changed this behavior and lets the other mount attempts succeed as well, so this problem only becomes apparent when cvmfs-fuse3 is installed. In any case, this can be properly handled by using a lockfile in CheckConcurrentMount.

EDIT: the connection to fuse3 is not so clear, this has been observed with only fuse2 as well