Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Run all tests in one go (vs. quit on first fail) #5

Closed
wants to merge 1 commit into from

3 participants

@satchmorun

Hi,

I like cutest and how simple it is, but I found myself wanting a couple of things:

  1. Results per test file
  2. Not to have the suite quit on the first failure

So this is a pull request that accomplishes those things. I'm not sure if it fits in with your roadmap, but I feel like its in line with what I perceive to be the philosophy of cutest, so I thought I'd send it over.

Description of changes

  1. Added a list of exceptions to the Thread.current[:cutest] Hash
  2. Moved the logic for catching/handling exceptions from run_file into test
  3. In the output, print the file name above the error line text (because i'm going to the file anyway and the line of code is often just an assert call)
  4. Added an E result for non-Cutest::AssertionFailed exceptions that are caught.

(it sounds like a lot, but it actually isn't all that much - and I updated the relevant tests so that everything's passing)

One nice result is that assertion methods no longer have to do IO to report on their status - they just need to flunk if some conditions aren't met, so the only changes to user code this pull request might require are in custom assertions - removing any calls to success.

Feel free to let me know what you think, I'd love to hear it.

@satchmorun satchmorun Run all tests in one go (vs. quit on first fail)
Changes output to show dots for each file under test
and then display the error/traces.
6da6b5f
@frodsan

.2 is intented.

@djanowski
Owner

Sorry, fail fast is a feature.

That said, I also envision Cutest as a spec, so I encourage you to write your own runner with this behavior.

@djanowski djanowski closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Aug 17, 2012
  1. @satchmorun

    Run all tests in one go (vs. quit on first fail)

    satchmorun authored
    Changes output to show dots for each file under test
    and then display the error/traces.
This page is out of date. Refresh to see the latest.
View
6 README.markdown
@@ -6,9 +6,9 @@ Forking tests.
Description
-----------
-Each test file is run in a forked process to avoid shared state. Once a failure
-is found, you get a report detailing what failed and how to locate the error
-and the rest of the file is skipped.
+Each test file is run in a forked process to avoid shared state. After all
+tests are run, you get a report detailing what failed and how to locate the
+errors.
You can use the `scope` command around tests: it guarantees that no instance
variables are shared between tests.
View
98 lib/cutest.rb
@@ -4,42 +4,37 @@ class Cutest
CACHE = Hash.new { |h, k| h[k] = File.readlines(k) }
def self.run(files)
- files.each do |file|
- run_file(file)
+ exceptions = files.flat_map { |file| run_file(file) }
- Process.wait
-
- break unless $?.success?
+ exceptions.each do |exc, file|
+ display_trace(exc, file)
+ display_error(exc)
end
-
- puts
end
def self.run_file(file)
+ reader, writer = IO.pipe
+
fork do
+ reader.close
+
begin
+ print "#{file}: ["
load(file)
-
+ print "]\n"
+ Marshal.dump(cutest[:exceptions], writer)
rescue LoadError, SyntaxError
- display_error
- exit 1
-
- rescue StandardError
- trace = $!.backtrace
- pivot = trace.index { |line| line.match(file) }
-
- if pivot
- other = trace[0..pivot].select { |line| line !~ FILTER }
- other.reverse.each { |line| display_trace(line) }
- else
- display_trace(trace.first)
- end
-
- display_error
-
+ display_error($!)
exit 1
end
end
+
+ Process.wait
+ exit unless $?.success?
+
+ writer.close
+ exceptions = Marshal.load(reader.read)
+ exceptions.map { |e| [e, file] }
end
def self.code(fn, ln)
@@ -50,16 +45,29 @@ def self.code(fn, ln)
end
end
- def self.display_error
- print "\n#{$!.class}: "
- print "#{$!.message}\n"
+ def self.display_error(exception)
+ print "#{exception.class}: "
+ print "#{exception.message}\n"
end
- def self.display_trace(line)
+ def self.display_trace(exception, file)
+ trace = exception.backtrace
+ pivot = trace.index { |line| line.match(file) }
+
+ if pivot
+ other = trace[0..pivot].select { |line| line !~ FILTER }
+ other.reverse.each { |trace| display_trace_location(trace) }
+ else
+ display_trace_location(trace.first)
+ end
+ end
+
+ def self.display_trace_location(line)
fn, ln = line.split(":")
- puts " line: #{code(fn, ln)}"
- puts " file: #{fn} +#{ln}"
+ puts
+ puts "file: #{fn} +#{ln}"
+ puts "line: #{code(fn, ln)}"
end
class AssertionFailed < StandardError
@@ -81,7 +89,7 @@ module Kernel
# Use Thread.current[:cutest] to store information about test preparation
# and setup.
- Thread.current[:cutest] ||= { :prepare => [] }
+ Thread.current[:cutest] ||= { :prepare => [], :exceptions => [] }
# Shortcut to access Thread.current[:cutest].
def cutest
@@ -136,32 +144,30 @@ def test(name = nil, &block)
prepare.each { |blk| blk.call }
block.call(setup && setup.call)
+ success
+ rescue StandardError => exception
+ cutest[:exceptions] << exception
+ exception.kind_of?(Cutest::AssertionFailed) ? failure : other_exception
end
# Assert that value is not nil or false.
def assert(value)
flunk("expression returned #{value.inspect}") unless value
- success
end
# Assert that two values are equal.
def assert_equal(value, other)
flunk("#{value.inspect} != #{other.inspect}") unless value == other
- success
end
# Assert that the block doesn't raise the expected exception.
def assert_raise(expected = Exception)
- begin
- yield
- rescue => exception
- ensure
- flunk("got #{exception.inspect} instead") unless exception.kind_of?(expected)
- success
- end
+ yield
+ rescue => exception
+ flunk("got #{exception.inspect} instead") unless exception.kind_of?(expected)
end
- # Stop the tests and raise an error where the message is the last line
+ # Raise an error where the message is the last line
# executed before flunking.
def flunk(message = nil)
exception = Cutest::AssertionFailed.new(message)
@@ -174,4 +180,14 @@ def flunk(message = nil)
def success
print "."
end
+
+ # Executed when a cutest assertion fails.
+ def failure
+ print "F"
+ end
+
+ # Executed when a non-cutest exception is raised
+ def other_exception
+ print "E"
+ end
end
View
1  test/fixtures/fail_custom_assertion.rb
@@ -1,6 +1,5 @@
def assert_empty(string)
flunk("not empty") unless string.empty?
- success
end
test "failed custom assertion" do
View
46 test/run.rb
@@ -1,5 +1,7 @@
test "output of successful run" do
- expected = ".\n"
+ expected = <<-EXP.gsub(/^[ ]{4}/, '')
+ test/fixtures/success.rb: [.]
+ EXP
out = %x{./bin/cutest test/fixtures/success.rb}
@@ -7,19 +9,27 @@
end
test "output of failed run" do
- expected = " line: assert false\n" +
- " file: test/fixtures/failure.rb +2\n\n" +
- "Cutest::AssertionFailed: expression returned false\n\n"
+ expected = <<-EXP.gsub(/^[ ]{4}/, '')
+ test/fixtures/failure.rb: [F]
+
+ file: test/fixtures/failure.rb +2
+ line: assert false
+ Cutest::AssertionFailed: expression returned false
+ EXP
out = %x{./bin/cutest test/fixtures/failure.rb}
assert_equal(expected, out)
end
-test "output of failed run" do
- expected = " line: raise \"Oops\"\n" +
- " file: test/fixtures/exception.rb +2\n\n" +
- "RuntimeError: Oops\n\n"
+test "output of non-assertion exception" do
+ expected = <<-EXP.gsub(/^[ ]{4}/, '')
+ test/fixtures/exception.rb: [E]
+
+ file: test/fixtures/exception.rb +2
+ line: raise "Oops"
+ RuntimeError: Oops
+ EXP
out = %x{./bin/cutest test/fixtures/exception.rb}
@@ -27,9 +37,13 @@
end
test "output of custom assertion" do
- expected = " line: assert_empty \"foo\"\n" +
- " file: test/fixtures/fail_custom_assertion.rb +7\n\n" +
- "Cutest::AssertionFailed: not empty\n\n"
+ expected = <<-EXP.gsub(/^[ ]{4}/, '')
+ test/fixtures/fail_custom_assertion.rb: [F]
+
+ file: test/fixtures/fail_custom_assertion.rb +6
+ line: assert_empty "foo"
+ Cutest::AssertionFailed: not empty
+ EXP
out = %x{./bin/cutest test/fixtures/fail_custom_assertion.rb}
@@ -37,9 +51,13 @@
end
test "output of failure in nested file" do
- expected = " line: assert false\n" +
- " file: test/fixtures/failure.rb +2\n\n" +
- "Cutest::AssertionFailed: expression returned false\n\n"
+ expected = <<-EXP.gsub(/^[ ]{4}/, '')
+ test/fixtures/failure_in_loaded_file.rb: [F]
+
+ file: test/fixtures/failure.rb +2
+ line: assert false
+ Cutest::AssertionFailed: expression returned false
+ EXP
out = %x{./bin/cutest test/fixtures/failure_in_loaded_file.rb}
Something went wrong with that request. Please try again.