RunTests.jl is a test running framework for Julia. In its simplest form RunTests.jl saves you from writing test/runtests.jl
scripts that look like this:
my_tests = ["sometests.jl",
"somemoretests.jl",
"evenmoretests.jl"]
println("Running tests:")
for my_test in my_tests
include(my_test)
end
and allows you to write them, more simply, like this:
using RunTests
exit(run_tests())
Or if you wish to be explicit what test files are run and in what order (not recomended, ideally the order tests are run should not matter) you can do this:
using RunTests
exit(run_tests(["sometests.jl",
"somemoretests.jl",
"evenmoretests.jl"]))
But it has more to offer than that! RunTests.jl builds on top of Julia's Base.Test
library to make it easy to add structure to your tests. Structuring your tests with RunTests.jl gives the following advantages:
- All the tests are run - the tests script doesn't bomb out after the first failure so you can see all your test results at once.
- A summary of how many tests passed/failed is produced so you can judge at a glance how the test run went.
- The
STDOUT
andSTDERR
output from each test is captured and reported along with the details of the test failure. - You get a progress bar showing how far through the tests you are, it is green while all the tests are passing and goes red if any fail
- Using modules and functions to structure test files gives you a natural isolation between tests.
- You can selectively skip tests with
@skipif
and mark failing tests with@xfail
. - Using
@parameterize
you can run the same test again again with different parameters and see which pass and which fail.
Here is an example test file written using RunTests.jl that demonstrates a number of features of the package:
using RunTests
using Base.Test
@testmodule ExampleTests begin
function test_one()
@test true
end
function test_two()
println("seen")
@test true
println("also seen")
@test false
println("never seen")
end
@skipif false function test_not_skipped()
@test true
end
@skipif true function test_skipped()
@test true
end
@xfail function test_xfails()
@test false
end
@xfail function test_xpasses()
@test true
end
@parameterize 1:4 function test_parameterized(x)
@test x<3
end
end
Running the file will run the tests and you will get this output:
Running 10 tests 100%|##############################| Time: 0:00:01
Tests:
======
ExampleTests.test_not_skipped PASSED
ExampleTests.test_one PASSED
ExampleTests.test_parameterized[1] PASSED
ExampleTests.test_parameterized[2] PASSED
ExampleTests.test_parameterized[3] FAILED
ExampleTests.test_parameterized[4] FAILED
ExampleTests.test_skipped SKIPPED
ExampleTests.test_two FAILED
ExampleTests.test_xfails XFAILED
ExampleTests.test_xpasses XPASSED
=================================== Failures ===================================
---------------------- ExampleTests.test_parameterized[3] ---------------------
test failed: :((x<3))
in error at error.jl:21
in default_handler at test.jl:19
in do_test at test.jl:39
--------------------------------------------------------------------------------
---------------------- ExampleTests.test_parameterized[4] ---------------------
test failed: :((x<3))
in error at error.jl:21
in default_handler at test.jl:19
in do_test at test.jl:39
--------------------------------------------------------------------------------
----------------------------- ExampleTests.test_two ----------------------------
test failed: false
in error at error.jl:21
in default_handler at test.jl:19
Captured Output:
================
seen
also seen
--------------------------------------------------------------------------------
================ 3 failed 4 passed 1 skipped 1 xfailed 1 xpassed ===============
But you can also run the file along with many others by putting them under the same directory (sub directories work too) and running them all together with:
using RunTests
exit(run_tests(<path_to_directory_containing_tests>))
When you run many test files together, like this, all their tests are pooled and you get one report for them all. If you don't specify a directory run_tests
will default to running tests from the "test" folder.
RunTests.jl is extensible, in fact @xfail
, @skipif
and @parameterize
are implemented as extensions. You can extend RunTests.jl to add further types of tests or categories of test result.