New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fail fast reporting to test runner. #21430

Merged
merged 2 commits into from Sep 29, 2015

Conversation

Projects
None yet
6 participants
@kaspth
Member

kaspth commented Aug 30, 2015

Inline output

Any failures or errors will be reported inline during the run by default.
Skipped tests will be reported if run in verbose mode.

Any result is output with failure messages and a rerun snippet for that test. I've tried to keep it minimal and relied on Minitest's summary reporter for details:

# Running:

.....S..........................................F

This failed

bin/rails test test/models/bunny_test.rb:14

.................................E

ArgumentError: Wups! Bet you didn't expect this!
    test/models/bunny_test.rb:19:in `block in <class:BunnyTest>'

bin/rails test test/models/bunny_test.rb:18

....................

Finished in 0.069708s, 1477.6019 runs/s, 1448.9106 assertions/s.

  1) Failure:
BunnyTest#test_something_failing [/Users/kasperhansen/Documents/code/collection_caching_test/test/models/bunny_test.rb:15]:
This failed


  2) Error:
BunnyTest#test_error_me:
ArgumentError: Wups! Bet you didn't expect this!
    test/models/bunny_test.rb:19:in `block in <class:BunnyTest>'

103 runs, 101 assertions, 1 failures, 1 errors, 1 skips

You have skipped tests. Run with --verbose for details.

Note that Minitest already informs us, we can get info about skipped tests if we run with --verbose. Handy!

Fail fast

Aborts the test run on the first failure. Suit that groove with the last config option:

minitest options:
    -h, --help                       Display this help.
    -s, --seed SEED                  Sets random seed. Also via env. Eg: SEED=n rake
    -v, --verbose                    Verbose. Show progress processing files.
    -n, --name PATTERN               Filter run on /regexp/ or string.

Known extensions: pride, rails
    -p, --pride                      Pride. Show your testing pride!

Usage: bin/rails test [options] [files or directories]
You can run a single test by appending a line number to a filename:

    bin/rails test test/models/user_test.rb:27

You can run multiple files and directories at the same time:

    bin/rails test test/controllers test/integration/login_test.rb

By default test failures and errors are reported inline during a run.

Rails options:
    -e, --environment ENV            Run tests in the ENV environment
    -b, --backtrace                  Show the complete backtrace
    -f, --fail-fast                  Abort test run on first failure

Here's what it looks like:

# Run with bin/rails test --fail-fast
# Running:

................................................S......E

ArgumentError: Wups! Bet you didn't expect this!
    test/models/bunny_test.rb:19:in `block in <class:BunnyTest>'

bin/rails test test/models/bunny_test.rb:18

....................................F

This failed

bin/rails test test/models/bunny_test.rb:14

Interrupted. Exiting...


Finished in 0.051427s, 1808.3872 runs/s, 1769.4972 assertions/s.

  1) Error:
BunnyTest#test_error_me:
ArgumentError: Wups! Bet you didn't expect this!
    test/models/bunny_test.rb:19:in `block in <class:BunnyTest>'


  2) Failure:
BunnyTest#test_something_failing [/Users/kasperhansen/Documents/code/collection_caching_test/test/models/bunny_test.rb:15]:
This failed

93 runs, 91 assertions, 1 failures, 1 errors, 1 skips

cc @senny @arthurnn @dhh

@kaspth kaspth added the railties label Aug 30, 2015

@kaspth kaspth self-assigned this Aug 30, 2015

@kaspth kaspth added this to the 5.0.0 milestone Aug 30, 2015

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Aug 30, 2015

Member

Hm, it seems Travis is creating a build for a merge commit rather than the one I've written. This Travis build, https://travis-ci.org/rails/rails/builds/77903799, is running based on: db989a2

Which says "Merge 485b662 into 79f44eb" where 485b662 is the commit here. That commit mismatch must be why the build status doesn't show up here!

Member

kaspth commented Aug 30, 2015

Hm, it seems Travis is creating a build for a merge commit rather than the one I've written. This Travis build, https://travis-ci.org/rails/rails/builds/77903799, is running based on: db989a2

Which says "Merge 485b662 into 79f44eb" where 485b662 is the commit here. That commit mismatch must be why the build status doesn't show up here!

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Sep 1, 2015

Member

@joshk maybe this is the reason?

Member

rafaelfranca commented Sep 1, 2015

@joshk maybe this is the reason?

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Sep 1, 2015

Member

I think it is already possible to do that today without having to implement it. If you send SIGINFO (control + T) on Mac OS, it prints the current test failures. Not sure we should implement the same thing again.

Member

rafaelfranca commented Sep 1, 2015

I think it is already possible to do that today without having to implement it. If you send SIGINFO (control + T) on Mac OS, it prints the current test failures. Not sure we should implement the same thing again.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Sep 2, 2015

Member

Yes, it does show failures and errors but without any rerun commands:

........E........load: 2.17  cmd: ruby 8681 running 1.53u 0.42s
Current results:


  1) Failure:
BunnyTest#test_something_failing [/Users/kasperhansen/Documents/code/collection_caching_test/test/models/bunny_test.rb:16]:
This failed


  2) Error:
BunnyTest#test_error_me:
ArgumentError: Wups! Bet you didn't expect this!
    test/models/bunny_test.rb:20:in `block in <class:BunnyTest>'



Current: BunnyTest#test_this_thing_works_79 0.07s
......................

Finished in 10.386975s, 9.9163 runs/s, 9.7237 assertions/s.
Member

kaspth commented Sep 2, 2015

Yes, it does show failures and errors but without any rerun commands:

........E........load: 2.17  cmd: ruby 8681 running 1.53u 0.42s
Current results:


  1) Failure:
BunnyTest#test_something_failing [/Users/kasperhansen/Documents/code/collection_caching_test/test/models/bunny_test.rb:16]:
This failed


  2) Error:
BunnyTest#test_error_me:
ArgumentError: Wups! Bet you didn't expect this!
    test/models/bunny_test.rb:20:in `block in <class:BunnyTest>'



Current: BunnyTest#test_this_thing_works_79 0.07s
......................

Finished in 10.386975s, 9.9163 runs/s, 9.7237 assertions/s.
@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Sep 7, 2015

Member

I didn't know about Ctrl-T 😳. @kaspth there's no way to extend that with the rerun snippets? Also, would be possible to enable fail-fast by default without having to press Ctrl-T?

Member

senny commented Sep 7, 2015

I didn't know about Ctrl-T 😳. @kaspth there's no way to extend that with the rerun snippets? Also, would be possible to enable fail-fast by default without having to press Ctrl-T?

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Sep 7, 2015

Member

I believe Minitest does have a signal trap which lets you enrich a signal. I'll look into it.

I'd still like the option to not have to press Ctrl + T (which I believe is only an OS X shortcut) to see the failures fast.

Member

kaspth commented Sep 7, 2015

I believe Minitest does have a signal trap which lets you enrich a signal. I'll look into it.

I'd still like the option to not have to press Ctrl + T (which I believe is only an OS X shortcut) to see the failures fast.

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Sep 7, 2015

Member

Also fail-fast in RSpec means, stop the test execution on the first test failure so if we are going to use an option I'd recommend to not use the same name as RSpec

Member

rafaelfranca commented Sep 7, 2015

Also fail-fast in RSpec means, stop the test execution on the first test failure so if we are going to use an option I'd recommend to not use the same name as RSpec

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Sep 7, 2015

Member

Hm, right. We could go with --defer-output for disabling this and perhaps have a real fail fast that just raises Interrupt on the first failure.

Member

kaspth commented Sep 7, 2015

Hm, right. We could go with --defer-output for disabling this and perhaps have a real fail fast that just raises Interrupt on the first failure.

@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn Sep 28, 2015

Member

have a real fail fast that just raises Interrupt

👍

Member

arthurnn commented Sep 28, 2015

have a real fail fast that just raises Interrupt

👍

Show outdated Hide outdated railties/CHANGELOG.md
@@ -1,3 +1,31 @@
* Add fail fast output to `bin/rails test`

This comment has been minimized.

@arthurnn

arthurnn Sep 28, 2015

Member

this needs to change to fail slow i guess.

@arthurnn

arthurnn Sep 28, 2015

Member

this needs to change to fail slow i guess.

This comment has been minimized.

@arthurnn

arthurnn Sep 28, 2015

Member

NVM

@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn Sep 28, 2015

Member

Yeah, for sure I was confused here.
--fail-fast for me, should do the same as Rspec.. which means. exit right in the first fail/error.

The output, I guess we should not change much from what is default on Minitest.
which is.. only show fail/errors in the end, or if you pass a --verbose will show the fail/errors as they happen.

Member

arthurnn commented Sep 28, 2015

Yeah, for sure I was confused here.
--fail-fast for me, should do the same as Rspec.. which means. exit right in the first fail/error.

The output, I guess we should not change much from what is default on Minitest.
which is.. only show fail/errors in the end, or if you pass a --verbose will show the fail/errors as they happen.

Add inline failure reporting to test runner.
Any failures or errors will be reported inline during the run by default.
Skipped tests will be reported if run in verbose mode.

Any result is output with failure messages and a rerun snippet for that test.

Rerun snippets won't be output after a run, unless `--defer-output` is passed.
@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Sep 28, 2015

Member

Alright, I've added a fail fast mode which will abort the run on the first failure. I've taken a literal approach so the abort doesn't trigger on errors (which might be silly, come to think of it).

Member

kaspth commented Sep 28, 2015

Alright, I've added a fail fast mode which will abort the run on the first failure. I've taken a literal approach so the abort doesn't trigger on errors (which might be silly, come to think of it).

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Sep 28, 2015

Member

I don't know if we should do something to improve the output on Minitest < 5.6.0. Check the changelog where there's an example of a backtrace that's spewed in those versions.

Member

kaspth commented Sep 28, 2015

I don't know if we should do something to improve the output on Minitest < 5.6.0. Check the changelog where there's an example of a backtrace that's spewed in those versions.

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Sep 28, 2015

Member

I don't know if we should do something to improve the output on Minitest < 5.6.0. Check the changelog where there's an example of a backtrace that's spewed in those versions.

I'd not even mention that in CHANGELOG. I always expect people to keep their dependecies up to date.

Member

rafaelfranca commented Sep 28, 2015

I don't know if we should do something to improve the output on Minitest < 5.6.0. Check the changelog where there's an example of a backtrace that's spewed in those versions.

I'd not even mention that in CHANGELOG. I always expect people to keep their dependecies up to date.

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
Member

rafaelfranca commented Sep 28, 2015

:shipit:

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Sep 29, 2015

Member

Good suggestion, @rafaelfranca. Will amend the changelog and merge! 😁

Member

kaspth commented Sep 29, 2015

Good suggestion, @rafaelfranca. Will amend the changelog and merge! 😁

Add fail fast to test runner.
Passing `--fail-fast` to the test runner will now abort the test run
on the first failure. The run continues on any unexpected errors.

kaspth added a commit that referenced this pull request Sep 29, 2015

Merge pull request #21430 from kaspth/test-runner-fail-fast
Add fail fast reporting to test runner.

@kaspth kaspth merged commit 97b980b into rails:master Sep 29, 2015

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

@kaspth kaspth deleted the kaspth:test-runner-fail-fast branch Sep 29, 2015

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Sep 29, 2015

Member

@kaspth awesome! 💛 🎉

Member

senny commented Sep 29, 2015

@kaspth awesome! 💛 🎉

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Sep 29, 2015

Member

❤️

Member

kaspth commented Sep 29, 2015

❤️

@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn Sep 29, 2015

Member

this is awesome

Member

arthurnn commented Sep 29, 2015

this is awesome

@teoljungberg

This comment has been minimized.

Show comment
Hide comment
@teoljungberg

teoljungberg Oct 3, 2015

Contributor

Hey, I don't want to be that guy - but I think I should get some recognition for previous work: https://github.com/teoljungberg/minitest-fail-fast

/cc @kaspth

Contributor

teoljungberg commented Oct 3, 2015

Hey, I don't want to be that guy - but I think I should get some recognition for previous work: https://github.com/teoljungberg/minitest-fail-fast

/cc @kaspth

@tenderlove

This comment has been minimized.

Show comment
Hide comment
@tenderlove

tenderlove Oct 3, 2015

Member

@teoljungberg is this a derivation of your work? Or you just want people to know that you wrote one too? Just trying to understand why, since we'd probably also have to mention RSpec, Minitest::Reporters, this blog post, etc.

Member

tenderlove commented Oct 3, 2015

@teoljungberg is this a derivation of your work? Or you just want people to know that you wrote one too? Just trying to understand why, since we'd probably also have to mention RSpec, Minitest::Reporters, this blog post, etc.

@teoljungberg

This comment has been minimized.

Show comment
Hide comment
@teoljungberg

teoljungberg Oct 5, 2015

Contributor

@tenderlove I think mentioning previous work in general is not a bad idea. Though, in the view of the other implementations out there - mine's the least important.

Contributor

teoljungberg commented Oct 5, 2015

@tenderlove I think mentioning previous work in general is not a bad idea. Though, in the view of the other implementations out there - mine's the least important.

@rafaelfranca rafaelfranca modified the milestones: 5.0.0 [temp], 5.0.0 Dec 30, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment