Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

travis-ci: quiet tests #1691

Merged
merged 3 commits into from Oct 3, 2018

Conversation

Projects
None yet
3 participants
@grondo
Copy link
Contributor

grondo commented Oct 1, 2018

This is an experimental PR to quiet the output from make check in travis (and elsewhere).
It is experimental because it adds code to tap-driver.sh and I'm not sure that's a good idea.

The main change is the addition of a quiet mode for tap-driver.sh which suppresses output for each line of TAP, instead issuing a one-line summary for each test. This dramatically reduces the output in travis from about 11.5K lines to 5500 lines. The new output looks like:

        shmem/backtoback.t: PASS: N=10  PASS=10  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
                loop/log.t: PASS: N=3   PASS=3   FAIL=0 SKIP=0 XPASS=0 XFAIL=0
             loop/handle.t: PASS: N=33  PASS=33  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
                 rpc/rpc.t: PASS: N=79  PASS=79  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
           loop/dispatch.t: PASS: N=53  PASS=53  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
           rolemask/loop.t: PASS: N=46  PASS=46  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
            loop/reactor.t: PASS: N=11  PASS=11  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
                rpc/mrpc.t: PASS: N=81  PASS=81  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
             loop/reduce.t: PASS: N=62  PASS=62  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
          t0000-sharness.t: PASS: N=27  PASS=21  FAIL=0 SKIP=5 XPASS=0 XFAIL=1
              t0008-attr.t: PASS: N=7   PASS=7   FAIL=0 SKIP=0 XPASS=0 XFAIL=0
           t0002-request.t: PASS: N=20  PASS=20  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
             t0009-dmesg.t: PASS: N=11  PASS=11  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
     t0010-generic-utils.t: PASS: N=4   PASS=4   FAIL=0 SKIP=0 XPASS=0 XFAIL=0
     t0016-cron-faketime.t: SKIP: N=1   PASS=0   FAIL=0 SKIP=1 XPASS=0 XFAIL=0
            t0003-module.t: PASS: N=44  PASS=44  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
             t0004-event.t: PASS: N=14  PASS=14  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
       t0013-config-file.t: PASS: N=13  PASS=13  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
             t0005-rexec.t: PASS: N=24  PASS=24  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
              t0005-exec.t: PASS: N=27  PASS=27  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
          t0017-security.t: PASS: N=39  PASS=39  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
              t0007-ping.t: PASS: N=33  PASS=33  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
          t0014-runlevel.t: PASS: N=8   PASS=8   FAIL=0 SKIP=0 XPASS=0 XFAIL=0
    t0012-content-sqlite.t: PASS: N=26  PASS=25  FAIL=0 SKIP=1 XPASS=0 XFAIL=0
              t0015-cron.t: PASS: N=27  PASS=27  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
     t0011-content-cache.t: PASS: N=17  PASS=16  FAIL=0 SKIP=1 XPASS=0 XFAIL=0
         t1002-kvs-watch.t: PASS: N=16  PASS=16  FAIL=0 SKIP=0 XPASS=0 XFAIL=0

Quiet mode is enabled with TAP_DRIVER_QUIET=1 in the environment before testing.

@grondo grondo force-pushed the grondo:shush-tests branch from 44e29a9 to 7512a76 Oct 1, 2018

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Oct 2, 2018

Codecov Report

Merging #1691 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #1691      +/-   ##
=========================================
+ Coverage   79.29%   79.3%   +<.01%     
=========================================
  Files         186     186              
  Lines       35059   35059              
=========================================
+ Hits        27801   27803       +2     
+ Misses       7258    7256       -2
Impacted Files Coverage Δ
src/bindings/lua/lua-hostlist/hostlist.c 58.78% <0%> (-0.23%) ⬇️
src/cmd/flux-job.c 88% <0%> (+0.66%) ⬆️
src/modules/kvs-watch/kvs-watch.c 75.96% <0%> (+0.82%) ⬆️
@SteVwonder

This comment has been minimized.

Copy link
Member

SteVwonder commented Oct 2, 2018

Awesome change! This output is so much nicer to read, especially with Travis's 10k limit.

How hard would it be to extend this to print the exact subtest that failed (if FAIL > 0)?

@grondo

This comment has been minimized.

Copy link
Contributor Author

grondo commented Oct 2, 2018

It would be trivial to print lines for each failed test. However, the full output is captured in test-suite.log, so I didn't bother this go around.

That being said, a failed test is unusual, so perhaps that is the way to go...

@grondo

This comment has been minimized.

Copy link
Contributor Author

grondo commented Oct 2, 2018

Here would be an example where "FAIL" and "ERROR" TAP output is not suppressed. This was running under make -j 8 so the lines are interspersed with the other output.

          t2009-hostlist.t: PASS: N=6   PASS=6   FAIL=0 SKIP=0 XPASS=0 XFAIL=0
               t2002-pmi.t: PASS: N=17  PASS=17  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
         t3000-mpi-basic.t: SKIP: N=1   PASS=0   FAIL=0 SKIP=1 XPASS=0 XFAIL=0
           t2201-job-cmd.t: PASS: N=20  PASS=18  FAIL=0 SKIP=2 XPASS=0 XFAIL=0
         t2100-aggregate.t: PASS: N=11  PASS=11  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
FAIL: t4000-issues-test-driver.t 5 - this test will fail
        t2200-job-ingest.t: PASS: N=9   PASS=7   FAIL=0 SKIP=2 XPASS=0 XFAIL=0
 t3001-mpi-personalities.t: PASS: N=4   PASS=4   FAIL=0 SKIP=0 XPASS=0 XFAIL=0
t4000-issues-test-driver.t: FAIL: N=5   PASS=4   FAIL=1 SKIP=0 XPASS=0 XFAIL=0
ERROR: t4000-issues-test-driver.t - exited with status 1
     lua/t0001-send-recv.t: PASS: N=34  PASS=34  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
           lua/t0002-rpc.t: PASS: N=19  PASS=19  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
        lua/t0003-events.t: PASS: N=28  PASS=28  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
            t2006-joblog.t: PASS: N=4   PASS=4   FAIL=0 SKIP=0 XPASS=0 XFAIL=0
         lua/t0007-alarm.t: PASS: N=4   PASS=4   FAIL=0 SKIP=0 XPASS=0 XFAIL=0
       lua/t0004-getattr.t: PASS: N=8   PASS=8   FAIL=0 SKIP=0 XPASS=0 XFAIL=0
     lua/t0009-sequences.t: PASS: N=54  PASS=54  FAIL=0 SKIP=0 XPASS=0 XFAIL=0
       lua/t1000-reactor.t: PASS: N=29  PASS=27  FAIL=0 SKIP=0 XPASS=0 XFAIL=2
      lua/t1001-timeouts.t: PASS: N=19  PASS=19  FAIL=0 SKIP=0 XPASS=0 XFAIL=0

@grondo grondo force-pushed the grondo:shush-tests branch from 7512a76 to 6774340 Oct 2, 2018

@grondo

This comment has been minimized.

Copy link
Contributor Author

grondo commented Oct 2, 2018

OK, pushed the change above. Now FAIL/ERROR lines are no longer suppressed during TAP_DRIVER_QUIET=1

grondo added some commits Oct 1, 2018

tap-driver.sh: add quiet mode
Problem: for CI testing and other environments, one line of TAP
output per test is becoming a bit verbose. This is causing an issue
in the Travis-CI environment specifically, where builds are failing
with truncated output, possibly due to large volume of test output.

Add a "quiet" mode to tap-driver.sh that suppresses PASS lines of
TAP output and issues a summary at the end of each test. This mode
can be initiated by setting TAP_DRIVER_QUIET=1 in the environment
before `make check`.
docker: export TAP_DRIVER_QUIET to docker runs
Add export of TAP_DRIVER_QUITE to runs of docker-run-checks.sh.
travis-ci: add TAP_DRIVER_QUIET=1 to environment
Add TAP_DRIVER_QUIET=1 to environment so that `make check`
runs much quieter, reducing the burden on Travis-CI log processing.

If tests fail, the full test output will still be emitted via
dump of the test-suite.log.

@grondo grondo force-pushed the grondo:shush-tests branch from 6774340 to ed855c5 Oct 2, 2018

@SteVwonder

This comment has been minimized.

Copy link
Member

SteVwonder commented Oct 2, 2018

Awesome! LGTM! Any feedback, @garlick? If not, I'm happy to merge this as is.

@SteVwonder SteVwonder self-requested a review Oct 2, 2018

@grondo

This comment has been minimized.

Copy link
Contributor Author

grondo commented Oct 2, 2018

As I said, I'm a bit worried about changing the autotools provided tap-driver.sh. Maybe we should rename it?

@SteVwonder

This comment has been minimized.

Copy link
Member

SteVwonder commented Oct 2, 2018

Maybe we should rename it?

If you want to, that's fine with me. I think leaving it as-is should be fine too. The last time tap-driver.sh was updated was back in 2011. So I doubt we will find ourselves pulling in changes from upstream.

@SteVwonder

This comment has been minimized.

Copy link
Member

SteVwonder commented Oct 2, 2018

The last time tap-driver.sh was updated was back in 2011.

My apologies, that is actually incorrect. It has been updated on basically a yearly basis, but the only thing that has changed in the past 7 years has been the copyright year, the url in the header comments, the emacs local variables at the bottom of the file, and some quoting in the help string (which is one of the changes we have). All that being said, I think it is still safe to say that the last major update was back in 2011.

@grondo

This comment has been minimized.

Copy link
Contributor Author

grondo commented Oct 2, 2018

Ok, fine with me for now. The impact of accidentally pulling in a new version is pretty small anyway.

@SteVwonder

This comment has been minimized.

Copy link
Member

SteVwonder commented Oct 3, 2018

Ok. Merging!

@SteVwonder SteVwonder merged commit 53a718d into flux-framework:master Oct 3, 2018

3 checks passed

codecov/patch Coverage not affected when comparing af8487d...ed855c5
Details
codecov/project 79.3% (+<.01%) compared to af8487d
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@grondo grondo deleted the grondo:shush-tests branch Feb 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.