djwong / xfstests

This test fails on btrfs due to the presence of delayed processing
of file deletes if the file is smaller than 32mb. Initially commit
97575ac tried to fix a similar
failure by bumping the size of the filesystem. However that change
had a knock-on effect in that the scratch filesystem created is
larger than 100mb and thus not created in mixed mode. This in turn
causes the fs to have only 20mb for file data (rest is taken by DUP
metadata). Naturally, this leads to file freeing taking up to
"transaction commit interval" (default 30 s) time to properly account
the freed space.

Not standards define when unlink operations should be accounted so
btrfs is well within its right to be implemented in that way. So
to avoid this edge case just issue a sync before taking the 2nd
free space reading.

Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: Eryu Guan <guaneryu@gmail.com>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

Typically, when following absolute redirect, if an opauqe dentry is
found, lookup in further lower directories is stopped. But if a child
dentry has another absolute redirect, then lookup in further lower
layers should continue.

Say, following is example setup.
upper:  /redirect (redirect=/a/b/c)
lower1: /a/[b]/c       ([b] is opaque) (c has absolute redirect=/a/b/d/)
lower0: /a/b/d/foo

"redirect" directory in upper should merge with lower1:/a/b/c/ and
lower0:/a/b/d/, despite lower1:/a/b/ being opaque.

This example and kernel fix has come from Amir Goldstein. I am just
putting a test for it to make sure its not broken down the line.

Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

The regression is introduced to btrfs in linux v4.4 and it refuses to
create new files after log replay by returning -EEXIST.

Although the problem is on btrfs only, there is no btrfs stuff in terms
of test, so this makes it generic.

The kernel fix is
  Btrfs: fix unexpected -EEXIST when creating new inode

[Eryu: add _require_metadata_journaling rule and 'log' 'metadata' group]

Reviewed-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

In the case of compression, each 128K input data chunk will be
compressed to 4K (because of the characters written are duplicate).
Therefore we have to write (128K * 16) to make sure every stripe can be
hit.

Signed-off-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
Reviewed-by: Eryu Guan <guaneryu@gmail.com>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

… value 64K

Because of commit e76e13c ("fsstress: implement the
clonerange/deduperange ioctls"), dedupe makes the number of references
to the same extent item increase so much that the default 4K buffer of
logical-resolve is no longer sufficient.

Signed-off-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
Reviewed-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

Test that if we have a file with two hard links in the same parent
directory, then remove of the links, create a new file in the same
parent directory and with the name of the link removed, fsync the new
file and have a power loss, mounting the filesystem succeeds.

This test is motivated by a bug found in btrfs, which is fixed by
the linux kernel patch titled:

  "Btrfs: fix log replay failure after unlink and link combination"

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Eryu Guan <guaneryu@gmail.com>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

Test that when a fsync journal/log exists, if we rename a special file
(fifo, symbolic link or device), create a hard link for it with its old
name and then commit the journal/log, if a power loss happens the
filesystem will not fail to replay the journal/log when it is mounted
the next time.

This test is motivated by a bug found in btrfs, which is fixed by the
following patch for the linux kernel:

  "Btrfs: fix log replay failure after linking special file and fsync"

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Eryu Guan <guaneryu@gmail.com>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

Tests that use _overlay_scratch_mount_dirs instead of _scratch_mount
should use _require_scratch_nocheck instead of _require_scratch
because these tests are either mounting with multiple lower dirs or
mounting with non-default lower/upper/work dir, so
_check_overlay_scratch_fs won't handle these cases correctly. So we
introduce _overlay_check_scratch_dirs helper and should call this
helper with the correct dir arguments for these non-default cases.

This patch modify these tests to optionally call
_overlay_check_scratch_dirs at the end of the test or after
_scratch_umount to mount base filesystem only and run the checker.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

No post-test check of the overlay dirs is required if case leaves
corrupt filesystem after test. We shoud use _require_scratch_nocheck()
instead of _require_scratch() in these cases.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

_check_overlay_scratch_fs() will check lowerdir of overlay filesystem,
this case remove this directory after test will lead to check failure,
and it is not really necessary to remove this directory, so keep this
directory.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

Hook filesystem check helper to _check_test_fs and _check_scratch_fs
for checking consistency of underlying dirs of overlay filesystem.
These helpers works only if fsck.overlay exists.

This patch introduce OVERLAY_FSCK_OPTIONS use for check overlayfs like
OVERLAY_MOUNT_OPTIONS, and also introduce a mount point check helper in
common/rc to detect a dir is a mount point or not.

[ _check_test_fs/_check_scratch_fs part picked from Amir's patch, thanks ]

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Eryu Guan <guaneryu@gmail.com>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

There is a problem of missing fstype check in _is_mounted() helper,
it will return the mountpoint if only the device arguments matches.

For example:
  Base mounted filesystem:
    /dev/sda2 on /boot type ext4 (rw,relatime,data=ordered)

  FSTYPE=xfs
  mountpoint=`_is_mounted /dev/sda1`
  echo "$mountpoint"

  Output: /boot

This patch rename _is_mounted to _is_dev_mounted because it check
the given device only (not mount dir), and add an optional "fstype"
parameter, let user specify file system type instead of default
FSTYPE. Finally, use findmnt instead of mount to avoid complex
processing of mount info and fix this problem simply.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Eryu Guan <guaneryu@gmail.com>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

Evidently ceph will report a 4M blocksize, which trips clonerange_f's
clumsy attempt to avoid reflinking an extent on top of itself.  The
original code assumed that "pick a random destination up to 1MB past the
end of the file" would suffice, but that clearly won't with a 4M
blocksize.  Instead, we'll change it to 1024*blksize.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Tested-by: Luis Henriques <lhenriques@suse.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

generic/472 is for changes that are not upstream and seem dead in
the water at the moment, so remove this test from the auto and quick
groups until it's been resolved upstream and the changes merged.

generic/473 really doesn't seme useful. FIEMAP is a debugging
interface, xfs_io is a debugging tool and so trying to make every
filesytem report exactly the same thing for a ranged query just
strikes me as the wrong thing to be doing. This fails on XFS, and
there's no apparent resolution to that in sight, so remove the test
from the auto and quick, too.

Signed-Off-By: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

According to the bash man page:

	OPTIND is initialized to 1 each time the shell or a shell
	script is invoked.

This doesn't appear to be true - in tests scripts with no other
getopts calls, I'm seeing the getopts loop in _xfs_check to fail to
parse input parameters correctly. Tracing shows the parameters are
being passed to _xfs_check correctly, but on occassion getopts
simply doesn't see them.

Hence when running tests with both external log and real time
devices, tests are failing at random because xfs_check is
mis-parsing the parameters passed to it and not configuring the
external log correctly:

_check_xfs_filesystem: filesystem on /dev/sdg is inconsistent (c)
*** xfs_check output ***
aborting - no external log specified for FS with an external log
*** end xfs_check output

Fix this by ensuring OPTIND is correctly initialised before using
getopts. Do it for all places that call getopts that don't already
set OPTIND=1 before starting their parsing loop.

Signed-Off-By: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

For 64k block size, the require code fails because the block range
[4k, 12k] would cause the fcollapse syscall to return -EINVAL. Hence
the tests using them are not executed.

This commit fixes the issue by calculating file offset ranges based
on the block size of the underlying filesystem.

Signed-off-by: Chandan Rajendra <chandan@linux.vnet.ibm.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

When -P is used, report the correct *.fsxgood filename in
report_failure and fix the default *.fsxops filename.

Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

Previously _scratch_mount didn't check the mount status and most
tests continue to run even if the mount failed (unless test checks
for the mount status explicitly). This would result in running tests
on the underlying filesystem (usually rootfs) and implicit test
failures, and such failures can be annoying and are usually hard to
debug.

Now _fail test by default if _scratch_mount failed and introduce
_try_scratch_mount for tests that need to check mount results
themselves.

Suggested-by: Andreas Gruenbacher <agruenba@redhat.com>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Andreas Gruenbacher <agruenba@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

On reflink+rmapbt XFs filesystems there isn't enough free space to
run this test on the 64MB filesystem image created. It notruns with
a curious error message - needs at least 0GB free:

generic/250             [10:01:57] [10:01:58] [not run]
        generic/250 -- This test requires at least 0GB free on /mnt/scratch to run

Fix this by increasing the size of the base filesystem image.

Signed-Off-By: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

btrfs/150 uses RAID1 profile and make SCRATCH_DEV fail for test.
However, if SCRATCH_DEV_POOL consists more than two devices,
SCRATCH_DEV may not be used for RAID1 pair and the tests may not run
as expected.

Fix this by add _scratch_dev_pool_get/put like other tests (141, 143
etc.) do.

Signed-off-by: Tomohiro Misono <misono.tomohiro@jp.fujitsu.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

Kernel commit f59cf5c29919 ("xfs: remove "no-allocation"
reservations for file creations") is known to cause earlier ENOSPC
conditions, and xfs/015 is affected in the way that no new file/dir
can be created in a newly created 16M XFS with both reflink and
rmapbt enabled, thus xfs/015 fails due to the missing test dirs.

So enlarge the initial fs size to 32M that we're able to create new
dir/file before growing the filesystem size.

Note that we doubled the fs initial size, we need to double the
required space too.

Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

Test OFD locks. Use fcntl F_OFD_SETLK/F_OFD_GETLK, to verify we are
being given correct advices through getlk by kernel.

The basic idea is one setlk routine setting locks via fcntl *_SETLK,
followed by operations like clone, dup then close fd; another
routine getlk getting locks via fcntl *_GETLK.

Firstly in setlk routine process P0, place a lock L0 on an opened
testfile, then do clone or dup and close relative fd.

In getlk process P2, do fcntl *_GETLK with lock L1 after get
notified by setlk routine.

In the end, getlk routine check the returned struct flock.l_type to
see if the lock mechanism works fine.

Test combainations of:
- shared or exclusive lock
- these locks are conflicting or not
- one OFD lock and one POSIX lock
- that open testfile RDONLY or RDWR
- clone with CLONE_FILES or not
- dup and close newfd

[eguan: made some minor non-functional changes]

Signed-off-by: Xiong Zhou <xzhou@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

The XFS rmapbt extent swap mechanism performs an extent by extent
swap to ensure the rmapbt is rectified with the appropriate extent
owner information after the operation. This implementation suffers
from a corner case that requires extra reservation if the swap
operation results in bouncing one of the associated inodes between
extent and btree formats. When this corner case occurs, it results
in a transaction block reservation overrun and possible corruption
of the free space accounting.

This regression test provides coverage for this corner case. It
creates two files with a large enough extent count to require btree
format, regardless of inode size, and performs a sequence of extent
swaps between them with a decreasing extent count until all extents
are removed from the file(s). This ensures that one of the swaps
covers the btree <-> extent fork format boundary case.

This test reproduces fs corruption on rmapbt enabled filesystems
running on kernels without the associated extent swap fix.

Signed-off-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

Add three tests to look for quota bugs in xfs reflink. The first
test looks for problems when we have speculative cow reservations in
memory, we chown the file, but the reservations don't move to the
new owner.  The second test checks that we remembered to dqattach
the inodes before performing reflink operations. The third is a
stress test for reflink quota handling near enospc and helped us to
find a directio cow write corruption bug when free space is
fragmented.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

A directory corrupted into a symlink is caught by the local format
ifork verifiers, and previously this test failed to catch such
kernel bug, so fix the test.

Note that the local format ifork verifiers were introduced by commit
0795e004fd4f ("xfs: create structure verifier function for short
form symlinks"), and didn't get wired up until commit 9cfb9b47479e
("xfs: provide a centralized method for verifying inline fork
data").

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

Remove the -y parameter from scrub runs since we're removing that
option from xfs_scrub.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

Tests were merged with high seq numbers to avoid conflicts with
other tests.  Now renumber them to contiguous numbers, as all other
tests have been merged correctly. This is easier to do than
assigning the final seq numbers at commit time.

Signed-off-by: Eryu Guan <eguan@redhat.com>

Add fsck.overlay test case to test it how to deal with impure xattr
in underlying directories of overlayfs.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

Add fsck.overlay test case to test it how to deal with invalid/valid/
duplicate redirect xattr in underlying directories of overlayfs.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

Add fsck.overlay test case to test it how to deal with orphan/valid
whiteouts in underlying directories of overlayfs.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

Define common overlay xattr keys from Linux kernel for ovl tests use.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

Add filesystem check helper for the upcoming fsck.overlay utility.
This helper invoke fsck.overlay to check and optionally repair
underlying directories of overlay filesystem.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

We don't want to fill up the scratch mount point if the scratch
mount fails.

Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

Exit with status 1 if there were any failures: using the number of
failed tests as the exit status makes no sense and can easily
overwrap.

In addition, clean up a few minor things.

Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>

…direct

Overlayfs with nfs_export enabled, indexes all directories on copy
up.  Directory index is requires for decoding lower directory file
handles in case ancestors have been renamed.

When enabling nfs_export on an overlay that has merge dirs with
lower layer redirects, the possibility of ancestor rename requires
special handling when encoding lower directory file handles from
layer > 1.

- Check decode of lower dir with parent under lower redirect
- Check decode of lower dir with grandparent under lower redirect
- Check decode of lower dir after rename of lower redirected parent
- Check decode of lower dir after rename of lower redirected grandparent

This test requires and enables overlayfs NFS export support and
merge dir rename support (redirect_dir).
NFS export support depends on and requires overlayfs index feature.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>