Enable additional test cases

Enable additional test cases, in most cases this required a few
minor modifications to the test scripts.  In a few cases a real
bug was uncovered and fixed.  And in a handful of cases where pools
are layered on pools the test case will be skipped until this is
supported.  Details below for each test case.

* zpool_add_004_pos - Skip test on Linux until adding zvols to pools
  is fully supported and deadlock free.

* zpool_add_005_pos.ksh - Skip dumpadm portion of the test which isn't
  relevant for Linux.  The find_vfstab_dev, find_mnttab_dev, and
  save_dump_dev functions were updated accordingly for Linux.

* zpool_add_006_pos - Update test case such that it doesn't depend
  on nested pools.  Switch to truncate from mkfile to reduce space
  requirements and speed up the test case.

* zpool_clear_001_pos - Speed up test case by filling filesystem to
  25% capacity.

* zpool_create_002_pos, zpool_create_004_pos - Use sparse files for
  file vdevs in order to avoid increasing the partition size.

* zpool_create_006_pos - 6ba1ce9 allows raidz+mirror configs with
  similar redundancy.  Updating the valid_args and forced_args cases.

* zpool_create_008_pos - Disable overlapping partition portion.

* zpool_create_011_neg - Fix to correctly create the extra partition.
  Modified zpool_vdev.c to use fstat64_blk() wrapper which includes
  the st_size even for block devices.

* zpool_create_012_neg - Updated to properly find swap devices.

* zpool_create_014_neg, zpool_create_015_neg - Updated to use
  swap_setup() and swap_cleanup() wrappers which do the right thing
  on Linux and Illumos.  Removed '-n' option which successed under
  Linux due to differences in the inuse checks.

* zpool_create_016_pos.ksh - Skipped test case isn't useful.

* zpool_create_020_pos - Added missing / to cleanup() function.
  Remove cache file prior to test to ensure a clean environment
  and avoid false positives.

* zpool_destroy_001_pos - Removed test case which creates a pool on
  a zvol.  This is more likely to deadlock under Linux and has never
  been completely supported on any platform.

* zpool_destroy_002_pos - 'zpool destroy -f' is unsupported on Linux.
  Mount point must not be busy in order to unmount them.

* zfs_destroy_001_pos - Handle EBUSY error which can occur with
  volumes when racing with udev.

* zpool_expand_001_pos, zpool_expand_003_neg - Skip test on Linux
  until adding zvols to pools is fully supported and deadlock free.
  The test could be modified to use loopback devices but it would
  be preerable to use the test case as is for improved coverage.

* zpool_export_004_pos - Updated test case to such that it doesn't
  depend on nested pools.  Normal file vdev under /var/tmp are fine.

* zpool_import_all_001_pos - Updated to skip partition 1, which is
  known as slice 2, on Illumos.  This prevents overwritting the
  default TESTPOOL which was causing the failure.

* zpool_import_002_pos, zpool_import_012_pos - No changes needed.

* zpool_remove_003_pos - No changes needed

* zpool_upgrade_002_pos, zpool_upgrade_004_pos - Root cause addressed
  by upstream OpenZFS commit 3b7f360.

* zpool_upgrade_007_pos - Disabled in test case due to known failure.
  Opened issue openzfs#6112

* zvol_misc_002_pos - Updated to to use ext2.

* zvol_misc_001_neg, zvol_misc_003_neg, zvol_misc_004_pos,
  zvol_misc_005_neg, zvol_misc_006_pos - Moved to skip list, thesei
  test case could be updated to use Linux's crash dump facility.

* zvol_swap_* - Updated to use swap_setup/swap_cleanup helpers.
  File creation switched from /tmp to /var/tmp.  Enabled minimal
  useful tests for Linux, skip test cases which aren't applicable.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#3484
Issue openzfs#5634
Issue openzfs#2437
Issue openzfs#5202
Issue openzfs#4034
Issue openzfs#6095

cmd/zpool/zpool_vdev.c

@@ -919,7 +919,7 @@ get_replication(nvlist_t *nvroot, boolean_t fatal)
 				 * this device altogether.
 				 */
 				if ((fd = open(path, O_RDONLY)) >= 0) {
-					err = fstat64(fd, &statbuf);
+					err = fstat64_blk(fd, &statbuf);
 					(void) close(fd);
 				} else {
 					err = stat64(path, &statbuf);

scripts/zfs-tests.sh

@@ -223,6 +223,7 @@ constrain_path() {
 
 	# Exceptions
 	ln -fs "$STF_PATH/awk" "$STF_PATH/nawk"
+	ln -fs /sbin/fsck.ext2 "$STF_PATH/fsck"
 	ln -fs /sbin/mkfs.ext2 "$STF_PATH/newfs"
 	ln -fs "$STF_PATH/gzip" "$STF_PATH/compress"
 	ln -fs "$STF_PATH/gunzip" "$STF_PATH/uncompress"

tests/runfiles/linux.run

@@ -219,70 +219,49 @@ tests = ['zfs_upgrade_001_pos', 'zfs_upgrade_002_pos', 'zfs_upgrade_003_pos',
 [tests/functional/cli_root/zpool]
 tests = ['zpool_001_neg', 'zpool_002_pos', 'zpool_003_pos']
 
-# DISABLED:
-# zpool_add_004_pos - https://github.com/zfsonlinux/zfs/issues/3484
-# zpool_add_005_pos - no 'dumpadm' command.
-# zpool_add_006_pos - https://github.com/zfsonlinux/zfs/issues/3484
 [tests/functional/cli_root/zpool_add]
 tests = ['zpool_add_001_pos', 'zpool_add_002_pos', 'zpool_add_003_pos',
+    'zpool_add_004_pos', 'zpool_add_005_pos', 'zpool_add_006_pos',
     'zpool_add_007_neg', 'zpool_add_008_neg', 'zpool_add_009_neg',
     'add-o_ashift', 'add_prop_ashift']
 
 [tests/functional/cli_root/zpool_attach]
 tests = ['zpool_attach_001_neg', 'attach-o_ashift']
 
-# DISABLED:
-# zpool_clear_001_pos - https://github.com/zfsonlinux/zfs/issues/5634
 [tests/functional/cli_root/zpool_clear]
-tests = ['zpool_clear_002_neg', 'zpool_clear_003_neg']
+tests = ['zpool_clear_001_pos', 'zpool_clear_002_neg', 'zpool_clear_003_neg']
 
-# DISABLED:
-# zpool_create_001_pos - needs investigation
-# zpool_create_002_pos - needs investigation
-# zpool_create_004_pos - needs investigation
-# zpool_create_006_pos - https://github.com/zfsonlinux/zfs/issues/3484
-# zpool_create_008_pos - uses VTOC labels (?) and 'overlapping slices'
-# zpool_create_011_neg - tries to access /etc/vfstab etc
-# zpool_create_012_neg - swap devices
-# zpool_create_014_neg - swap devices
-# zpool_create_015_neg - swap devices
-# zpool_create_016_pos - no dumadm command.
-# zpool_create_020_pos - needs investigation
 [tests/functional/cli_root/zpool_create]
-tests = [
-    'zpool_create_003_pos', 'zpool_create_005_pos', 'zpool_create_007_neg',
-    'zpool_create_009_neg', 'zpool_create_010_neg', 'zpool_create_017_neg',
-    'zpool_create_018_pos', 'zpool_create_019_pos',
+tests = ['zpool_create_001_pos', 'zpool_create_002_pos',
+    'zpool_create_003_pos', 'zpool_create_004_pos', 'zpool_create_005_pos',
+    'zpool_create_006_pos', 'zpool_create_007_neg', 'zpool_create_008_pos',
+    'zpool_create_009_neg', 'zpool_create_010_neg', 'zpool_create_011_neg',
+    'zpool_create_012_neg', 'zpool_create_014_neg',
+    'zpool_create_015_neg', 'zpool_create_016_pos', 'zpool_create_017_neg',
+    'zpool_create_018_pos', 'zpool_create_019_pos', 'zpool_create_020_pos',
     'zpool_create_021_pos', 'zpool_create_022_pos', 'zpool_create_023_neg',
     'zpool_create_024_pos',
     'zpool_create_features_001_pos', 'zpool_create_features_002_pos',
     'zpool_create_features_003_pos', 'zpool_create_features_004_neg',
     'zpool_create_features_005_pos',
     'create-o_ashift']
 
-# DISABLED:
-# zpool_destroy_001_pos - needs investigation
-# zpool_destroy_002_pos - busy mountpoint behavior
 [tests/functional/cli_root/zpool_destroy]
-tests = [
+tests = ['zpool_destroy_001_pos', 'zpool_destroy_002_pos',
     'zpool_destroy_003_neg']
 pre =
 post =
 
 [tests/functional/cli_root/zpool_detach]
 tests = ['zpool_detach_001_neg']
 
-# DISABLED: Requires full FMA support in ZED
-# zpool_expand_001_pos - https://github.com/zfsonlinux/zfs/issues/2437
-# zpool_expand_003_pos - https://github.com/zfsonlinux/zfs/issues/2437
 [tests/functional/cli_root/zpool_expand]
-tests = ['zpool_expand_002_pos', 'zpool_expand_004_pos']
+tests = ['zpool_expand_001_pos', 'zpool_expand_002_pos',
+    'zpool_expand_003_neg', 'zpool_expand_004_pos']
 
-# DISABLED:
-# zpool_export_004_pos - https://github.com/zfsonlinux/zfs/issues/3484
 [tests/functional/cli_root/zpool_export]
 tests = ['zpool_export_001_pos', 'zpool_export_002_pos',
-    'zpool_export_003_neg']
+    'zpool_export_003_neg', 'zpool_export_004_pos']
 
 [tests/functional/cli_root/zpool_get]
 tests = ['zpool_get_001_pos', 'zpool_get_002_pos', 'zpool_get_003_pos',
@@ -291,20 +270,16 @@ tests = ['zpool_get_001_pos', 'zpool_get_002_pos', 'zpool_get_003_pos',
 [tests/functional/cli_root/zpool_history]
 tests = ['zpool_history_001_neg', 'zpool_history_002_pos']
 
-# DISABLED:
-# zpool_import_002_pos - https://github.com/zfsonlinux/zfs/issues/5202
-# zpool_import_012_pos - sharenfs issue
-# zpool_import_all_001_pos - partition issue
 [tests/functional/cli_root/zpool_import]
-tests = ['zpool_import_001_pos',
+tests = ['zpool_import_001_pos', 'zpool_import_002_pos',
     'zpool_import_003_pos', 'zpool_import_004_pos', 'zpool_import_005_pos',
     'zpool_import_006_pos', 'zpool_import_007_pos', 'zpool_import_008_pos',
     'zpool_import_009_neg', 'zpool_import_010_pos', 'zpool_import_011_neg',
-    'zpool_import_013_neg', 'zpool_import_014_pos',
+    'zpool_import_012_pos', 'zpool_import_013_neg', 'zpool_import_014_pos',
     'zpool_import_features_001_pos', 'zpool_import_features_002_neg',
     'zpool_import_features_003_pos','zpool_import_missing_001_pos',
     'zpool_import_missing_002_pos', 'zpool_import_missing_003_pos',
-    'zpool_import_rename_001_pos']
+    'zpool_import_rename_001_pos', 'zpool_import_all_001_pos']
 
 [tests/functional/cli_root/zpool_labelclear]
 tests = ['zpool_labelclear_active', 'zpool_labelclear_exported']
@@ -317,10 +292,9 @@ tests = ['zpool_offline_001_pos', 'zpool_offline_002_neg']
 [tests/functional/cli_root/zpool_online]
 tests = ['zpool_online_001_pos', 'zpool_online_002_neg']
 
-# DISABLED:
-# zpool_remove_003_pos - needs investigation
 [tests/functional/cli_root/zpool_remove]
-tests = ['zpool_remove_001_neg', 'zpool_remove_002_pos']
+tests = ['zpool_remove_001_neg', 'zpool_remove_002_pos',
+    'zpool_remove_003_pos']
 
 [tests/functional/cli_root/zpool_replace]
 tests = ['zpool_replace_001_neg', 'replace-o_ashift', 'replace_prop_ashift']
@@ -338,14 +312,11 @@ post =
 tests = ['zpool_status_001_pos', 'zpool_status_002_pos','zpool_status_003_pos']
 user =
 
-# DISABLED:
-# zpool_upgrade_002_pos - https://github.com/zfsonlinux/zfs/issues/4034
-# zpool_upgrade_004_pos - https://github.com/zfsonlinux/zfs/issues/4034
-# zpool_upgrade_007_pos - needs investigation
 [tests/functional/cli_root/zpool_upgrade]
-tests = ['zpool_upgrade_001_pos',
-    'zpool_upgrade_003_pos', 'zpool_upgrade_005_neg',
-    'zpool_upgrade_006_neg', 'zpool_upgrade_008_pos',
+tests = ['zpool_upgrade_001_pos', 'zpool_upgrade_002_pos',
+    'zpool_upgrade_003_pos', 'zpool_upgrade_004_pos',
+    'zpool_upgrade_005_neg', 'zpool_upgrade_006_neg',
+    'zpool_upgrade_007_pos', 'zpool_upgrade_008_pos',
     'zpool_upgrade_009_neg']
 
 # DISABLED:
@@ -667,15 +638,13 @@ tests = ['zvol_ENOSPC_001_pos']
 [tests/functional/zvol/zvol_cli]
 tests = ['zvol_cli_001_pos', 'zvol_cli_002_pos', 'zvol_cli_003_neg']
 
-# DISABLED: requires dumpadm
-#[tests/functional/zvol/zvol_misc]
-#tests = ['zvol_misc_001_neg', 'zvol_misc_002_pos', 'zvol_misc_003_neg',
-#    'zvol_misc_004_pos', 'zvol_misc_005_neg', 'zvol_misc_006_pos']
+[tests/functional/zvol/zvol_misc]
+tests = ['zvol_misc_001_neg', 'zvol_misc_002_pos', 'zvol_misc_003_neg',
+    'zvol_misc_004_pos', 'zvol_misc_005_neg', 'zvol_misc_006_pos']
 
-# DISABLED: requires updated for Linux
-#[tests/functional/zvol/zvol_swap]
-#tests = ['zvol_swap_001_pos', 'zvol_swap_002_pos', 'zvol_swap_003_pos',
-#    'zvol_swap_004_pos', 'zvol_swap_005_pos', 'zvol_swap_006_pos']
+[tests/functional/zvol/zvol_swap]
+tests = ['zvol_swap_001_pos', 'zvol_swap_002_pos', 'zvol_swap_003_pos',
+    'zvol_swap_004_pos', 'zvol_swap_005_pos', 'zvol_swap_006_pos']
 
 [tests/functional/libzfs]
 tests = ['many_fds']

tests/zfs-tests/include/commands.cfg

@@ -65,6 +65,7 @@ export SYSTEM_FILES='arp
     md5sum
     mkdir
     mknod
+    mkswap
     mktemp
     modprobe
     mount
@@ -100,6 +101,7 @@ export SYSTEM_FILES='arp
     su
     sudo
     sum
+    swapoff
     swapon
     sync
     tail

tests/zfs-tests/include/libtest.shlib

@@ -3302,3 +3302,59 @@ function zed_stop
 		log_must rmdir $ZEDLET_DIR
 	fi
 }
+
+#
+# Check is provided device is being active used as a swap device.
+#
+function is_swap_inuse
+{
+	typeset device=$1
+
+	if [[ -z $device ]] ; then
+		log_note "No device specified."
+		return 1
+	fi
+
+	if is_linux; then
+		swapon -s | grep -w $(readlink -f $device) > /dev/null 2>&1
+	else
+		swap -l | grep -w $device > /dev/null 2>&1
+	fi
+
+	return $?
+}
+
+#
+# Setup a swap device using the provided device.
+#
+function swap_setup
+{
+	typeset swapdev=$1
+
+	if is_linux; then
+		log_must mkswap $swapdev 2>/dev/null 2>&1
+		log_must swapon $swapdev
+	else
+	        log_must swap -a $swapdev
+	fi
+
+	return 0
+}
+
+#
+# Cleanup a swap device on the provided device.
+#
+function swap_cleanup
+{
+	typeset swapdev=$1
+
+	if is_swap_inuse $swapdev; then
+		if is_linux; then
+			log_must swapoff $swapdev
+		else
+			log_must swap -d $swapdev
+		fi
+	fi
+
+	return 0
+}

tests/zfs-tests/tests/functional/cli_root/zfs_destroy/zfs_destroy_001_pos.ksh

@@ -161,7 +161,8 @@ function test_n_check
 	fi
 
 	# Invoke 'zfs destroy [-rRf] <dataset>'
-	log_must zfs destroy $opt $dtst
+	log_must_busy zfs destroy $opt $dtst
+	block_device_wait
 
 	# Kill any lingering instances of mkbusy, and clear the list.
 	if ! is_linux ; then

tests/zfs-tests/tests/functional/cli_root/zfs_upgrade/zfs_upgrade.kshlib

@@ -150,12 +150,12 @@ function default_check_zfs_upgrade #rootfs
 
 		if (( df_ret != 0 )); then
 			if (( spa_version != 0 )) && (( vp < spa_version )); then
-				log_mustnot eval 'zfs upgrade $opt -a > /dev/null 2>&1'
-				log_must eval 'zpool upgrade $pool > /dev/null 2>&1'
+				log_mustnot zfs upgrade $opt -a >/dev/null
+				log_must zpool upgrade $pool >/dev/null
 				vp=$(get_pool_prop version $pool)
 			fi
 
-			log_must eval 'zfs upgrade $opt -a > /dev/null 2>&1'
+			log_must zfs upgrade $opt -a
 
 			for fs in $(zfs list -rH -t filesystem -o name $rootfs) ; do
 				log_must check_fs_version $fs $newv

tests/zfs-tests/tests/functional/cli_root/zpool_add/zpool_add.kshlib

@@ -36,12 +36,18 @@
 #
 function find_vfstab_dev
 {
-	typeset vfstab="/etc/vfstab"
-	typeset tmpfile="/tmp/vfstab.tmp"
 	typeset vfstabdev
 	typeset vfstabdevs=""
 	typeset line
 
+	if is_linux; then
+		vfstab="/etc/fstab"
+		tmpfile="/tmp/fstab.tmp"
+	else
+		vfstab="/etc/vfstab"
+		tmpfile="/tmp/vfstab.tmp"
+	fi
+
 	cat $vfstab | grep "^${DEV_DSKDIR}" >$tmpfile
 	while read -r line
 	do
@@ -59,12 +65,18 @@ function find_vfstab_dev
 #
 function find_mnttab_dev
 {
-	typeset mnttab="/etc/mnttab"
-	typeset tmpfile="/tmp/mnttab.tmp"
 	typeset mnttabdev
 	typeset mnttabdevs=""
 	typeset line
 
+	if is_linux; then
+		typeset mnttab="/etc/mtab"
+		typeset tmpfile="/tmp/mtab.tmp"
+	else
+		typeset mnttab="/etc/mnttab"
+		typeset tmpfile="/tmp/mnttab.tmp"
+	fi
+
 	cat $mnttab | grep "^${DEV_DSKDIR}" >$tmpfile
 	while read -r line
 	do
@@ -84,10 +96,14 @@ function save_dump_dev
 {
 
 	typeset dumpdev
-	typeset fnd="Dump device"
 
-	dumpdev=`dumpadm | grep "$fnd" | cut -f2 -d : | \
-		awk '{print $1}'`
+	if is_linux; then
+		dumpdev=""
+	else
+		typeset fnd="Dump device"
+		dumpdev=`dumpadm | grep "$fnd" | cut -f2 -d : | \
+			awk '{print $1}'`
+	fi
 	echo $dumpdev
 }
 
@@ -97,13 +113,13 @@ function save_dump_dev
 function partition_cleanup
 {
 
-        if [[ -n $DISK ]]; then
-                partition_disk $SIZE $DISK 7
-        else
-                typeset disk=""
-                for disk in $DISK0 $DISK1; do
-                        partition_disk $SIZE $disk 7
-                done
-        fi
+	if [[ -n $DISK ]]; then
+		partition_disk $SIZE $DISK 7
+	else
+		typeset disk=""
+		for disk in $DISK0 $DISK1; do
+			partition_disk $SIZE $disk 7
+		done
+	fi
 
 }

tests/zfs-tests/tests/functional/cli_root/zpool_add/zpool_add_004_pos.ksh

@@ -45,6 +45,11 @@
 
 verify_runnable "global"
 
+# See issue: https://github.com/zfsonlinux/zfs/issues/6065
+if is_linux; then
+	log_unsupported "Creating a pool containing a zvol may deadlock"
+fi
+
 function cleanup
 {
 	poolexists $TESTPOOL && \
@@ -71,7 +76,7 @@ log_must poolexists "$TESTPOOL1"
 log_must zfs create -V $VOLSIZE $TESTPOOL1/$TESTVOL
 block_device_wait
 
-log_must zpool add "$TESTPOOL" /dev/zvol/dsk/$TESTPOOL1/$TESTVOL
+log_must zpool add "$TESTPOOL" $ZVOL_DEVDIR/$TESTPOOL1/$TESTVOL
 
 log_must vdevs_in_pool "$TESTPOOL" "$ZVOL_DEVDIR/$TESTPOOL1/$TESTVOL"