Allow test-runner to filter test groups by tag #6788

dinatale2 · 2017-10-27T00:57:11Z

Description

Enable test-runner to accept a list of tags to identify
which test groups the user wishes to run.

Motivation and Context

I would like to use these tags to generate different classes of test runs in the future.
For example, a "trivial" test run or a "fast" test run could be useful for fuzz testing.
I would also like to allow buildbot to use the tags during zfs-test execution and allow
developers to specify tags to run in their commit messages.

How Has This Been Tested?

Locally in a VM.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (a change to man pages or other documentation)

Checklist:

My code follows the ZFS on Linux code style requirements.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.
All commit messages are properly formatted and contain Signed-off-by.
Change has been approved by a ZFS on Linux member.

dinatale2 · 2017-10-27T04:32:37Z

@jwk404 Would you mind providing input on this PR?

gmelikov · 2017-10-27T05:17:15Z

Thanks! Now we can easily add travis.yml from #5786!

dinatale2 · 2017-10-27T06:52:47Z

Looks like I broke zfs-tests.sh. Will look into it later.

behlendorf

A nice addition to this would be the ability to treat each test name as a tag. That would allow you to easily do things like -T zpool_create_009_neg ' to run a specific test case. Taking that one step farther you could extend this to allow wildcards, i.e -T zpool_import_features*`.

behlendorf · 2017-10-27T17:05:17Z

scripts/zfs-tests.sh

+TAGSARG=
+if [[ -n $TAGS ]]; then
+	TAGSARG="-T $TAGS"
+fi


You could simplify this by having TAGS="functional" by default. Then you can unconditionally pass -T $TAGS to test-runner to remove the need for TAGSARG.

behlendorf · 2017-10-27T17:06:22Z

tests/runfiles/linux.run

@@ -8,7 +8,6 @@
 # source.  A copy of the CDDL is also available via the Internet at
 # http://www.illumos.org/license/CDDL.
 #
-


nit: let's keep this line of white space

codecov · 2017-10-29T03:11:18Z

Codecov Report

Merging #6788 into master will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #6788      +/-   ##
==========================================
+ Coverage   75.21%   75.24%   +0.02%     
==========================================
  Files         297      297              
  Lines       94441    94422      -19     
==========================================
+ Hits        71036    71046      +10     
+ Misses      23405    23376      -29

Flag	Coverage Δ
#kernel	`74.74% <ø> (+0.26%)`	⬆️
#user	`67.55% <ø> (+0.06%)`	⬆️

Impacted Files	Coverage Δ
module/zfs/dmu_tx.c	`81.67% <0%> (-5.38%)`	⬇️
cmd/zed/agents/zfs_diagnosis.c	`69.79% <0%> (-3.82%)`	⬇️
cmd/zed/zed_disk_event.c	`81.15% <0%> (-2.18%)`	⬇️
module/zfs/zrlock.c	`84.37% <0%> (-1.57%)`	⬇️
cmd/zed/agents/zfs_mod.c	`64.33% <0%> (-1.4%)`	⬇️
module/zfs/mmp.c	`95.7% <0%> (-1.23%)`	⬇️
module/icp/api/kcf_cipher.c	`15.35% <0%> (-0.83%)`	⬇️
module/zfs/ddt.c	`96.2% <0%> (-0.73%)`	⬇️
module/zfs/dmu_traverse.c	`94.48% <0%> (-0.69%)`	⬇️
module/zfs/zap_micro.c	`85.23% <0%> (-0.58%)`	⬇️
... and 35 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c9427c4...478edaf. Read the comment docs.

jwk404

Is zfs-test.sh still failing? I've looked over the change, and don't see anything that should cause a problem.

dinatale2 · 2017-10-30T21:49:18Z

@jwk404 zfs-test.sh is no longer failing due to recent changes to the script.

dinatale2 · 2017-10-30T22:48:18Z

I went ahead and updated the PR to include a -I flag to test-runner to control the number of times a test run is executed.

behlendorf · 2017-11-01T21:41:36Z

tests/runfiles/linux.run

@@ -18,29 +18,35 @@ timeout = 600
 post_user = root
 post = cleanup
 outputdir = /var/tmp/test_results
+tags = []


Why is a tags field needed here at all?

Refreshed. Defaulted to having the tag functional for any test group without tags specified.

Enable test-runner to accept a list of tags to identify which test groups the user wishes to run. Also allow test-runner to perform multiple iterations of a test run. Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>

Enable test-runner to accept a list of tags to identify which test groups the user wishes to run. Also allow test-runner to perform multiple iterations of a test run. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: John Wren Kennedy <john.kennedy@delphix.com> Reviewed-by: George Melikov <mail@gmelikov.ru> Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov> Closes openzfs#6788

Enable test-runner to accept a list of tags to identify which test groups the user wishes to run. Also allow test-runner to perform multiple iterations of a test run. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: John Wren Kennedy <john.kennedy@delphix.com> Reviewed-by: George Melikov <mail@gmelikov.ru> Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov> Closes #6788

Enable test-runner to accept a list of tags to identify which test groups the user wishes to run. Also allow test-runner to perform multiple iterations of a test run. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: John Wren Kennedy <john.kennedy@delphix.com> Reviewed-by: George Melikov <mail@gmelikov.ru> Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov> Closes openzfs#6788

dinatale2 added the Status: Work in Progress Not yet ready for general review label Oct 27, 2017

dinatale2 force-pushed the test-runner-groups branch 2 times, most recently from 5aa64da to a9582ff Compare October 27, 2017 04:27

gmelikov approved these changes Oct 27, 2017

View reviewed changes

behlendorf reviewed Oct 27, 2017

View reviewed changes

openzfs deleted a comment from codecov bot Oct 27, 2017

dinatale2 force-pushed the test-runner-groups branch from a9582ff to 48d7f96 Compare October 29, 2017 03:10

dinatale2 force-pushed the test-runner-groups branch from 48d7f96 to 47618e8 Compare October 29, 2017 03:14

jwk404 reviewed Oct 30, 2017

View reviewed changes

jwk404 approved these changes Oct 30, 2017

View reviewed changes

dinatale2 force-pushed the test-runner-groups branch 2 times, most recently from ca1a56e to 4904674 Compare October 30, 2017 22:47

dinatale2 force-pushed the test-runner-groups branch 2 times, most recently from 97c501e to 7893d9f Compare October 31, 2017 00:20

dinatale2 added Ready and removed Status: Work in Progress Not yet ready for general review labels Oct 31, 2017

behlendorf approved these changes Nov 1, 2017

View reviewed changes

Allow test-runner to filter test groups by tag

478edaf

Enable test-runner to accept a list of tags to identify which test groups the user wishes to run. Also allow test-runner to perform multiple iterations of a test run. Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>

dinatale2 force-pushed the test-runner-groups branch from 7893d9f to 478edaf Compare November 1, 2017 23:46

behlendorf merged commit 9a810ef into openzfs:master Nov 3, 2017

dinatale2 deleted the test-runner-groups branch November 3, 2017 16:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow test-runner to filter test groups by tag #6788

Allow test-runner to filter test groups by tag #6788

dinatale2 commented Oct 27, 2017 •

edited

dinatale2 commented Oct 27, 2017

gmelikov commented Oct 27, 2017

dinatale2 commented Oct 27, 2017

behlendorf left a comment

behlendorf Oct 27, 2017

behlendorf Oct 27, 2017

codecov bot commented Oct 29, 2017 •

edited

jwk404 left a comment

dinatale2 commented Oct 30, 2017

dinatale2 commented Oct 30, 2017

behlendorf Nov 1, 2017

dinatale2 Nov 1, 2017

Allow test-runner to filter test groups by tag #6788

Allow test-runner to filter test groups by tag #6788

Conversation

dinatale2 commented Oct 27, 2017 • edited

Description

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

dinatale2 commented Oct 27, 2017

gmelikov commented Oct 27, 2017

dinatale2 commented Oct 27, 2017

behlendorf left a comment

Choose a reason for hiding this comment

behlendorf Oct 27, 2017

Choose a reason for hiding this comment

behlendorf Oct 27, 2017

Choose a reason for hiding this comment

codecov bot commented Oct 29, 2017 • edited

Codecov Report

jwk404 left a comment

Choose a reason for hiding this comment

dinatale2 commented Oct 30, 2017

dinatale2 commented Oct 30, 2017

behlendorf Nov 1, 2017

Choose a reason for hiding this comment

dinatale2 Nov 1, 2017

Choose a reason for hiding this comment

dinatale2 commented Oct 27, 2017 •

edited

codecov bot commented Oct 29, 2017 •

edited