Test a CLI program against a black box. Intended for school projects.
Often when working on a school project, the class will provide a black box to test your programs against. Greybox provides a simple testing harness to easily add new test cases whose expected output is automatically generated by this black box. Since these might be slow, or even over an ssh connection, it will store the results so it only has to actually run the black box the first time.
Only Ruby 1.9.3 and 2.0.0 are supported.
$ gem install greybox
In your project directory, create a file called
Greyfile might look like this:
Greybox.setup do |c| c.test_command = "./my_test_command < %" c.blackbox_command = "./blackbox_command < %" c.input = "test/*.input" end
This defines a testing setup in which the command
$ ./blackbox_command < INPUT_FILE.input
is used as the reference for correctness.
% denotes where the input file will be placed.
$ ./my_test_command < INPUT_FILE.input
is verified to match the blackbox output every time
The expected results will be stored in
This is the default behaviour but can be changed.
As a more complete example, here is a Greyfile I used for a school project:
Greybox.setup do |c| c.test_command = "racket wlppscan.ss < % 2>&1" c.blackbox_command = "ssh school 'java cs241.WLPPScan' < % 2>&1" c.input = "test/*.input" # if the expected ERRORs, all we care about is that # the actual ERRORs as well. c.comparison = ->(actual, expected) do if expected =~ /ERROR/ actual =~ /ERROR/ else expected == actual end end end
test_command specifies how to run my version of the program, a racket program.
blackbox_command specifies how to generate the expected output for a specific test case.
comparison specifies how the output of
test_command should be compared to the black box output.
In this case, if part of the output contains
ERROR, all we care about is that the actual output contains
ERROR as well.
Otherwise, they must be equal.
- Fork it
- Create your feature branch (
git checkout -b my-new-feature)
- Commit your changes (
git commit -am 'Add some feature')
- Push to the branch (
git push origin my-new-feature)
- Create new Pull Request