A modern test suite contains many testing frameworks. It contains a suite of unit tests, integration tests, end-to-end acceptance tests. If your application is polyglot, you might have a tests for each language. Each suite can be executed with a different command.
Regularly you'll need to run them all at once. Or you'll want to run some extra commands, like setting up the database on your continuous integration server.
This gem will help you manage all your test suites and auxillary commands.
Here's an example of the test suite report you might get:
This gem is extracted from a common pattern I found in my projects. You're looking at an alpha release. The usage may change significantly in future versions.
Currently, test_suite supports the following features:
- Define a set of commands
- Define if and when a command should fail the build
- Give a nice overview of all your commands
- Provide a rake task
- Enforce cleanup commands to be run
- Multiple outputs and formatters
- Stream output to a web server, via websockets, so you can observe your test suite from everywhere
- Run commands in parrallel
Just install the gem:
gem install test_suite
Or add it to your Gemfile.
Create a configuration file, called something like
TestSuite.configure do |config| config.command :migrate do |cmd| cmd.runs "bundle exec rake RAILS_ENV=test db:migrate:reset" cmd.fails_build_immediately! end config.command :rspec do |cmd| cmd.runs "bundle exec rspec --format progress --format html --out rspec.html" end config.command :cucumber do |cmd| cmd.runs "bundle exec cucumber --format html --out cucumber.html" end # Also use it to generate metrics config.command :brakeman do |cmd| cmd.never_fails_build! end end
cmd.runs is optional and only needed if the command is different from the
By default, when a command fails, the next command will run. After all commands have run, the build will fail. This means that you can run multiple test suites, and get a nice overview of which suite failed.
If a command is setting up stuff, you'll want to stop the build immediately if
it fails. For example, if the database couldn't be migrated, it has no use
running the test suites. You can mark these commands with
If some process isn't important enough to break the build, you can mark them
You can run the test suite, by passing the filename to the test_run command.
You can create multiple test suites, one for running on your CI server, one for running locally. Simply create multiple files.
You can also split up your test suite, for more detailed results:
TestSuite.configure do |config| config.command "Unit Tests" do |cmd| cmd.runs "rspec spec/unit" end config.command "Integration Tests" do |cmd| cmd.runs "rspec spec/integration" end end
I usually make a bash script to install the required gems. This has the benefit that it can be run on your CI server more reliably.
#!/bin/bash set -e # let this script fail directly when errors occur gem which bundler > /dev/null || gem install bundler --no-ri --no-rdoc bundle check --no-color || bundle install --no-color bundle exec test_suite script/test_suite.rb
You can also create a rake task. This might be handy for CI servers that already know how to run rake, like Travis. Rake is an optional dependency though.
require 'test_suite/rake_task' TestSuite::RakeTask.new(:test, "script/test_suite.rb") # make it default task :default => :test
Let me know if and how you use it!
- Fork it
- Create your feature branch (
git checkout -b my-new-feature)
- Commit your changes (
git commit -am 'Added some feature')
- Push to the branch (
git push origin my-new-feature)
- Create new Pull Request