Greybox

Test a CLI program against a black box. Intended for school projects.

Often when working on a school project, the class will provide a black box to test your programs against. Greybox provides a simple testing harness to easily add new test cases whose expected output is automatically generated by this black box. Since these might be slow, or even over an ssh connection, it will store the results so it only has to actually run the black box the first time.

Installation

Only Ruby 1.9.3 and 2.0.0 are supported.

$ gem install greybox

Usage

In your project directory, create a file called Greyfile. A Greyfile might look like this:

Greybox.setup do |c|
  c.test_command = "./my_test_command < %"
  c.blackbox_command = "./blackbox_command < %"
  c.input = "test/*.input"
end

This defines a testing setup in which the command

$ ./blackbox_command < INPUT_FILE.input

is used as the reference for correctness. % denotes where the input file will be placed.

$ ./my_test_command < INPUT_FILE.input

is verified to match the blackbox output every time

$ greybox

is run.

The expected results will be stored in INPUT_FILE.output. This is the default behaviour but can be changed.

As a more complete example, here is a Greyfile I used for a school project:

Greybox.setup do |c|
  c.test_command = "racket wlppscan.ss < % 2>&1"
  c.blackbox_command = "ssh school 'java cs241.WLPPScan' < % 2>&1"
  c.input = "test/*.input"

  # if the expected ERRORs, all we care about is that
  # the actual ERRORs as well.
  c.comparison = ->(actual, expected) do
    if expected =~ /ERROR/
      actual =~ /ERROR/
    else
      expected == actual
    end
  end
end

The test_command specifies how to run my version of the program, a racket program. blackbox_command specifies how to generate the expected output for a specific test case.

comparison specifies how the output of test_command should be compared to the black box output. In this case, if part of the output contains ERROR, all we care about is that the actual output contains ERROR as well. Otherwise, they must be equal.

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request