Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test case: rollback_003_pos #6143

Open
behlendorf opened this issue May 19, 2017 · 3 comments · Fixed by #6632
Open

Test case: rollback_003_pos #6143

behlendorf opened this issue May 19, 2017 · 3 comments · Fixed by #6632
Labels
Bot: Not Stale Override for the stale bot Component: Test Suite Indicates an issue with the test framework or a test case

Comments

@behlendorf
Copy link
Contributor

System information

Type Version/Name
Distribution Name all
Distribution Version all
Linux Kernel all
Architecture all
ZFS Version 0.7.0-rc4
SPL Version 0.7.0-rc4

Describe the problem you're observing

Occasional failure of test case rollback_003_pos.

Describe how to reproduce the problem

Reproducible occasionally by the automated testing and locally.

Include any warning/errors/backtraces from the system logs

http://build.zfsonlinux.org/builders/CentOS%207%20x86_64%20Mainline%20%28TEST%29/builds/2466/steps/shell_9/logs/log

Test: /usr/share/zfs/zfs-tests/tests/functional/snapshot/rollback_003_pos (run as root) [00:00] [FAIL]
03:33:11.37 ASSERTION: Verify rollback succeeds when there are nested file systems.
03:33:11.39 SUCCESS: zfs snapshot testpool@testsnap
03:33:11.43 SUCCESS: zfs rollback testpool@testsnap
03:33:11.44 SUCCESS: zfs snapshot testpool@testsnap exited 1
03:33:11.45 SUCCESS: touch /testpool/testfile-snapshot
03:33:11.48 SUCCESS: zfs rollback testpool@testsnap
03:33:11.52 SUCCESS: zfs create testpool/testfile-snapshot
03:33:11.55 SUCCESS: zfs rollback testpool@testsnap
03:33:11.55 NOTE: Verify rollback of multiple nested file systems succeeds.
03:33:11.58 SUCCESS: zfs snapshot testpool/testfile-snapshot@testsnap
03:33:11.60 SUCCESS: zfs snapshot testpool@testsnap.1
03:33:11.64 umount: /testpool/testfile-snapshot: mountpoint not found cannot unmount '/testpool/testfile-snapshot': umount failed
03:33:11.64 ERROR: zfs unmount -a exited 1
@behlendorf behlendorf added the Component: Test Suite Indicates an issue with the test framework or a test case label May 19, 2017
behlendorf pushed a commit that referenced this issue Sep 25, 2017
* Add 'zfs bookmark' coverage (zfs_bookmark_cliargs)

 * Add OpenZFS 8166 coverage (zpool_scrub_offline_device)

 * Fix "busy" zfs_mount_remount failures

 * Fix bootfs_003_pos, bootfs_004_neg, zdb_005_pos local cleanup

 * Update usage of $KEEP variable, add get_all_pools() function

 * Enable history_008_pos and rsend_019_pos (non-32bit builders)

 * Enable zfs_copies_005_neg, update local cleanup

 * Fix zfs_send_007_pos (large_dnode + OpenZFS 8199)

 * Fix rollback_003_pos (use dataset name, not mountpoint, to unmount)

 * Update default_raidz_setup() to work properly with more than 3 disks

 * Use $TEST_BASE_DIR instead of hardcoded (/var)/tmp for file VDEVs

 * Update usage of /dev/random to /dev/urandom

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Issue #6086 
Closes #5658 
Closes #6143 
Closes #6421 
Closes #6627 
Closes #6632
behlendorf pushed a commit that referenced this issue Oct 3, 2018
When using "zfs destroy" on a dataset that is using "sharenfs=on" and
has been automatically exported (by libzfs), the dataset will not be
automatically unexported as it should be. This workflow appears to have
been broken by this commit: 3fd3e56

In that change, the "zfs_unmount" function was modified to use the
"mnt.mnt_special" field when determining the mount point that is being
unmounted, rather than "mnt.mnt_mountp".

As a result, when "mntpt" is passed into "zfs_unshare_proto", it's value
is now the dataset name rather than the mountpoint. Thus, when this
value is used with the "is_shared" function (via "zfs_unshare_proto") it
will not find a match (since that function assumes it'll be passed the
mountpoint) and incorrectly reports that the dataset is not shared.

This can be easily reproduced with the following commands:

    $ sudo zpool create tank xvdb
    $ sudo zfs create -o sharenfs=on tank/fish
    $ sudo zfs destroy tank/fish

    $ sudo zfs list -r tank
    NAME   USED  AVAIL  REFER  MOUNTPOINT
    tank  97.5K  7.27G    24K  /tank

    $ sudo exportfs
    /tank/fish      <world>
    $ sudo cat /etc/dfs/sharetab
    /tank/fish      -       nfs     rw,crossmnt

At this point, the "tank/fish" filesystem doesn't exist, but it's still
listed as exported when looking at "exportfs" and "/etc/dfs/sharetab".

Also note, this change brings us back in-sync with the illumos code, as
it pertains to this one line; on illumos, "mnt.mnt_mountp" is used.

Reviewed by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: George Wilson <george.wilson@delphix.com>
Signed-off-by: Prakash Surya <prakash.surya@delphix.com>
Issue #6143
Closes #7941
@tcaputi
Copy link
Contributor

tcaputi commented Oct 26, 2018

Some additional info on this bug. The problem is actually not the umount, as the test log would indicate. The root cause of the problem can be recreated more simplistically:

# setup
echo 4294966270 > /sys/module/zfs/parameters/zfs_flags
zpool destroy -f testpool &> /dev/null
zpool create -f testpool sdb

# basic test
zfs snapshot testpool@testsnap
zfs create testpool/testfile-snapshot
zfs rollback testpool@testsnap

zfs get mounted,mountpoint
NAME                        PROPERTY    VALUE                        SOURCE
testpool                    mounted     yes                          -
testpool                    mountpoint  /testpool                    default
testpool@testsnap           mounted     -                            -
testpool@testsnap           mountpoint  -                            -
testpool/testfile-snapshot  mounted     yes                          -
testpool/testfile-snapshot  mountpoint  /testpool/testfile-snapshot  default

ls /testpool/testfile-snapshot
ls: cannot access '/testpool/testfile-snapshot': No such file or directory

zfs get mounted,mountpoint
NAME                        PROPERTY    VALUE                        SOURCE
testpool                    mounted     yes                          -
testpool                    mountpoint  /testpool                    default
testpool@testsnap           mounted     -                            -
testpool@testsnap           mountpoint  -                            -
testpool/testfile-snapshot  mounted     no                           -
testpool/testfile-snapshot  mountpoint  /testpool/testfile-snapshot  default

It seems the call to zfs rollback isnt properly reinstating the child mountpoints. What is a bit interesting is that zfs get mounted, mount, and cat /proc/mounts all believe that the child is still mounted until the user attempts to look at that mountpoint. Once that happens the Linux directory cache seems to realize this is wrong and update everything.

We could definitely solve this problem in userspace with changelists or some other mechanism, but I would rather see if we can fix the kernel module so that this can all happen atomically.

Now that we know what's actually wrong, I'm going to reopen this issue so we can get a proper fix together.

@tcaputi tcaputi reopened this Oct 26, 2018
GregorKopka pushed a commit to GregorKopka/zfs that referenced this issue Jan 7, 2019
When using "zfs destroy" on a dataset that is using "sharenfs=on" and
has been automatically exported (by libzfs), the dataset will not be
automatically unexported as it should be. This workflow appears to have
been broken by this commit: 3fd3e56

In that change, the "zfs_unmount" function was modified to use the
"mnt.mnt_special" field when determining the mount point that is being
unmounted, rather than "mnt.mnt_mountp".

As a result, when "mntpt" is passed into "zfs_unshare_proto", it's value
is now the dataset name rather than the mountpoint. Thus, when this
value is used with the "is_shared" function (via "zfs_unshare_proto") it
will not find a match (since that function assumes it'll be passed the
mountpoint) and incorrectly reports that the dataset is not shared.

This can be easily reproduced with the following commands:

    $ sudo zpool create tank xvdb
    $ sudo zfs create -o sharenfs=on tank/fish
    $ sudo zfs destroy tank/fish

    $ sudo zfs list -r tank
    NAME   USED  AVAIL  REFER  MOUNTPOINT
    tank  97.5K  7.27G    24K  /tank

    $ sudo exportfs
    /tank/fish      <world>
    $ sudo cat /etc/dfs/sharetab
    /tank/fish      -       nfs     rw,crossmnt

At this point, the "tank/fish" filesystem doesn't exist, but it's still
listed as exported when looking at "exportfs" and "/etc/dfs/sharetab".

Also note, this change brings us back in-sync with the illumos code, as
it pertains to this one line; on illumos, "mnt.mnt_mountp" is used.

Reviewed by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: George Wilson <george.wilson@delphix.com>
Signed-off-by: Prakash Surya <prakash.surya@delphix.com>
Issue openzfs#6143
Closes openzfs#7941
FransUrbo pushed a commit to FransUrbo/zfs that referenced this issue Apr 28, 2019
* Add 'zfs bookmark' coverage (zfs_bookmark_cliargs)

 * Add OpenZFS 8166 coverage (zpool_scrub_offline_device)

 * Fix "busy" zfs_mount_remount failures

 * Fix bootfs_003_pos, bootfs_004_neg, zdb_005_pos local cleanup

 * Update usage of $KEEP variable, add get_all_pools() function

 * Enable history_008_pos and rsend_019_pos (non-32bit builders)

 * Enable zfs_copies_005_neg, update local cleanup

 * Fix zfs_send_007_pos (large_dnode + OpenZFS 8199)

 * Fix rollback_003_pos (use dataset name, not mountpoint, to unmount)

 * Update default_raidz_setup() to work properly with more than 3 disks

 * Use $TEST_BASE_DIR instead of hardcoded (/var)/tmp for file VDEVs

 * Update usage of /dev/random to /dev/urandom

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Issue openzfs#6086 
Closes openzfs#5658 
Closes openzfs#6143 
Closes openzfs#6421 
Closes openzfs#6627 
Closes openzfs#6632
@stale
Copy link

stale bot commented Aug 25, 2020

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the Status: Stale No recent activity for issue label Aug 25, 2020
@stale stale bot closed this as completed Nov 23, 2020
behlendorf added a commit to behlendorf/zfs that referenced this issue Dec 22, 2021
Under Linux when rolling back a mounted filesystem negative dentries
may not be dropped from the cache.  This can result in an ENOENT
being incorrectly returned on first access.  Issuing a `df` before
the unmount results in the negative dentries being invalidated and
side steps the issue.

This is solely a workaround for the test case on Linux and not
correct behavior.  The core issue of invalidating negative dentries
needs to be handled with a kernel side change.  This is being
tracked as issue openzfs#6143.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#6143
@behlendorf
Copy link
Contributor Author

This remains an issue, see #12898 Reopening.

@behlendorf behlendorf reopened this Dec 22, 2021
@stale stale bot removed the Status: Stale No recent activity for issue label Dec 22, 2021
@behlendorf behlendorf added the Bot: Not Stale Override for the stale bot label Dec 22, 2021
behlendorf added a commit to behlendorf/zfs that referenced this issue Dec 22, 2021
Under Linux when rolling back a mounted filesystem negative dentries
may not be dropped from the cache.  This can result in an ENOENT
being incorrectly returned on first access.  Issuing a `df` before
the unmount results in the negative dentries being invalidated and
side steps the issue.

This is solely a workaround for the test case on Linux and not
correct behavior.  The core issue of invalidating negative dentries
needs to be handled with a kernel side change.  This is being
tracked as issue openzfs#6143.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#6143
behlendorf added a commit that referenced this issue Dec 22, 2021
Under Linux when rolling back a mounted filesystem negative dentries
may not be dropped from the cache.  This can result in an ENOENT
being incorrectly returned on first access.  Issuing a `df` before
the unmount results in the negative dentries being invalidated and
side steps the issue.

This is solely a workaround for the test case on Linux and not
correct behavior.  The core issue of invalidating negative dentries
needs to be handled with a kernel side change.  This is being
tracked as issue #6143.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #12898 
Issue #6143
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Feb 10, 2022
Under Linux when rolling back a mounted filesystem negative dentries
may not be dropped from the cache.  This can result in an ENOENT
being incorrectly returned on first access.  Issuing a `df` before
the unmount results in the negative dentries being invalidated and
side steps the issue.

This is solely a workaround for the test case on Linux and not
correct behavior.  The core issue of invalidating negative dentries
needs to be handled with a kernel side change.  This is being
tracked as issue openzfs#6143.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#12898 
Issue openzfs#6143
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Feb 14, 2022
Under Linux when rolling back a mounted filesystem negative dentries
may not be dropped from the cache.  This can result in an ENOENT
being incorrectly returned on first access.  Issuing a `df` before
the unmount results in the negative dentries being invalidated and
side steps the issue.

This is solely a workaround for the test case on Linux and not
correct behavior.  The core issue of invalidating negative dentries
needs to be handled with a kernel side change.  This is being
tracked as issue openzfs#6143.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#12898 
Issue openzfs#6143
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Feb 16, 2022
Under Linux when rolling back a mounted filesystem negative dentries
may not be dropped from the cache.  This can result in an ENOENT
being incorrectly returned on first access.  Issuing a `df` before
the unmount results in the negative dentries being invalidated and
side steps the issue.

This is solely a workaround for the test case on Linux and not
correct behavior.  The core issue of invalidating negative dentries
needs to be handled with a kernel side change.  This is being
tracked as issue openzfs#6143.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#12898 
Issue openzfs#6143
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Feb 17, 2022
Under Linux when rolling back a mounted filesystem negative dentries
may not be dropped from the cache.  This can result in an ENOENT
being incorrectly returned on first access.  Issuing a `df` before
the unmount results in the negative dentries being invalidated and
side steps the issue.

This is solely a workaround for the test case on Linux and not
correct behavior.  The core issue of invalidating negative dentries
needs to be handled with a kernel side change.  This is being
tracked as issue openzfs#6143.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#12898 
Issue openzfs#6143
nicman23 pushed a commit to nicman23/zfs that referenced this issue Aug 22, 2022
Under Linux when rolling back a mounted filesystem negative dentries
may not be dropped from the cache.  This can result in an ENOENT
being incorrectly returned on first access.  Issuing a `df` before
the unmount results in the negative dentries being invalidated and
side steps the issue.

This is solely a workaround for the test case on Linux and not
correct behavior.  The core issue of invalidating negative dentries
needs to be handled with a kernel side change.  This is being
tracked as issue openzfs#6143.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#12898 
Issue openzfs#6143
nicman23 pushed a commit to nicman23/zfs that referenced this issue Aug 22, 2022
Under Linux when rolling back a mounted filesystem negative dentries
may not be dropped from the cache.  This can result in an ENOENT
being incorrectly returned on first access.  Issuing a `df` before
the unmount results in the negative dentries being invalidated and
side steps the issue.

This is solely a workaround for the test case on Linux and not
correct behavior.  The core issue of invalidating negative dentries
needs to be handled with a kernel side change.  This is being
tracked as issue openzfs#6143.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#12898 
Issue openzfs#6143
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bot: Not Stale Override for the stale bot Component: Test Suite Indicates an issue with the test framework or a test case
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants