On multiple lookups, 2 symlinks seen on distrep instead of one (was BZ 1600923)

[csabahenk]

This issue is imported from Red Hat Bugzilla. It has no customer impact, so I'm reinstantiating it in community scope.

See https://bugzilla.redhat.com/1600923.

Here we quote the relevant comments from the discussion. (Some are marked private over there, so they are not visible without sufficient privileges; the comments included here don't carry sensitive data, so they are fine to be published here.)

TL;DR busily performed namespace changing operations from two distinct mounts of the same volume demonstrates insufficient synchronization around namespace operations.

---

Comment 0 (description), Rochelle, 2018-07-13 11:55:59 UTC

Description of problem:

Ambiguous behavior seen while renaming a symlink and having a constant lookup on the second mount point (the volume was mounted on) , MKDIR of the original symlink was not seen on the second mount point.

The directory is seen on the second mount point only after about 20-30 min

Version-Release number of selected component (if applicable):

[root@dhcp42-18 scripts]# rpm -qa | grep gluster
glusterfs-rdma-3.12.2-13.el7rhgs.x86_64
glusterfs-server-3.12.2-13.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-client-xlators-3.12.2-13.el7rhgs.x86_64
glusterfs-cli-3.12.2-13.el7rhgs.x86_64
python2-gluster-3.12.2-13.el7rhgs.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.6.x86_64
glusterfs-3.12.2-13.el7rhgs.x86_64
glusterfs-api-3.12.2-13.el7rhgs.x86_64
gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64
glusterfs-libs-3.12.2-13.el7rhgs.x86_64
vdsm-gluster-4.19.43-2.3.el7rhgs.noarch
glusterfs-fuse-3.12.2-13.el7rhgs.x86_64
glusterfs-geo-replication-3.12.2-13.el7rhgs.x86_64
glusterfs-events-3.12.2-13.el7rhgs.x86_64
[root@dhcp42-18 scripts]#

How reproducible:

Always with constant lookups

Steps to Reproduce:

1. Create and start a 3x3 dist-rep volume

2. Mount the volume on 2 different mount points

3. On one mntpt1 : touch file1 (Create file) --> on mntpt2, do a few lookups

4. On mntpt1: ln -s file1 400 (Create symlink file) --> on mntpt2, do a few lookups

5. On mntpt1: mv 400 sym_400 (Rename symlink file) --> on mntpt2, keep doing loopkups

6. On mntpt1 : mkdir 400 (Create directory with same name as symlink file )--> on mntpt2, do continuous look ups

There were two symlinks on the slave instead of one.

Actual results:

There were 2 symlinks on mntpt2 instead of one
The directory that was created is seen on the slave only after 20-30 min

Expected results:

There should be only 1 symlink on mntpt2 (the renamed symlink) and the directory created in the name of the original symlink (along with the file that was touched)

Comment 3, Poornima G, 2018-07-20 06:11:23 UTC

The issue is the after rename, the old dentry is present in the inode-table and hence the name of new directory resolved to a wrong inode.

This is not md-cache bug, the component should be changed to core. This is more likely bug in fuse/libglusterfs.

Need to analyse this further.

Comment 6, Raghavendra G, 2018-07-23 11:27:32 UTC

Are the two names seen after mv, 400 and sym_400?

Comment 7, Rochelle, 2018-07-25 06:42:06 UTC

Yes, Du. It is 400 and sym_400

Comment 19, Raghavendra G, 2018-11-06 06:06:36 UTC

A likely hypothesis for this bug is stale dentries (which can be confirmed once fusedumps are available) in inode table. If the hypothesis turns out to be true, its not a trivial fix to avoid stale dentries. Solutions I can think of are:

• strict serialization of entry operations (lookup, rename etc). This approach is not feasible due to performance reasons.
• Once ctime xlator is available, fuse can rely on ctime of parent to detect changes to parent directory and discard results of an older operation. For eg., it can identify lookup was issued before rename and hence not link stale dentry after rename. However, ctime xlator is not available by default. Without this trusting on mtime/ctime of stat is not correct as cluster.consistent-metadata won't be turned on always and hence stats could be obtained from different replica pairs.

[csabahenk]

A reproduction session was fuse dumped on the course of the investigation. The attached tarball includes the fuse dumps taken on the two mounts.

bz1600923-fusedumps.tar.gz

[stale]

Thank you for your contributions.
Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity.
It will be closed in 2 weeks if no one responds with a comment here.

[stale]

Closing this issue as there was no update since my last update on issue. If this is an issue which is still valid, feel free to open it.