An integration test framework for console applications.
Currently, Smoke is in alpha, and as such is not packaged. You can use it with your own projects by downloading the smoke binary, Smoke Windows launcher and the LICENSE file and committing them with your code, or adding this repository as a Git submodule and committing the reference.
The smoke binary requires Ruby 1.9.3 or greater. It does not require any additional gems.
You can also run Smoke using Docker without any further installation. See below for more information.
Smoke is distributed under the MIT license.
Writing Test Cases
A test case consists of input and expected output. It is constructed of a number of files with the same name and different extensions.
First off, you need to specify the command itself.
- The command is specified in a file named
command, with parameters each on a new line. It is executed from the current working directory.
- The command can be overriden for each individual test case by creating a file with the
Input can come in two forms: standard input and command-line arguments.
- Standard input is specified by naming the file with the extension
- Command-line arguments are specified in a file with the
.argsextension, one per line.
Outputs that can be observed by Smoke consist of standard output, standard error and the exit status of the program.
- Expected standard output is specified with the
.outextension. Alternatively, multiple possible expected outputs can be specified by using an extension that starts with
.outc. If there are multiple outputs, a match with any of them will be considered a success.
- Expected standard error uses the
.errextension, but otherwise works in exactly the same way as expected standard output.
- The expected exit status is a file with the
.statusextension. It contains a single number between
At least one of standard output and standard error must be specified, though it can be empty. If no exit status is specified, it will be assumed to be
Our simplest calculator test case consists of three files:
2 + 2
We might want to assert that certain things fail. For example, postfix notation should fail because the second token is expected to be an operator. In this example, our calculator is expected to produce a semi-reasonable error message and exit with a status of
2 to signify a parsing error.
5 3 *
"3" is not a valid operator.
In order to run tests against an application, you simply invoke Smoke with the command required to invoke the application, and the directory containing the tests. Given an application that is invoked with
ruby bin/calculator.rb, and the tests in the test directory, we would run the tests as follows:
Tests can also be passed on an individual basis:
smoke test/addition test/postfix-notation-fails
To override the command, or to specify it on the command line in place of the
command file, you can use the
smoke --command='ruby calculator.rb' test
Smoke will exit with a code of
0 if all tests succeed,
1 if any test fails, or
2 if the invocation of Smoke itself was not understood (for example, if only one argument is provided).
Output will be in color if outputting to a terminal. You can force color output on or off with the
In order to run Smoke with Docker instead, you would change the command as follows:
docker run --rm -it -v $PWD:/var/app samirtalwar/smoke test
Enjoy. Any feedback is welcome.
We had a problem at work. It was a pretty nice problem to have. We were getting too many job applicants and we needed to screen them quickly. So we put some tests online and pointed the
guinea pigs at 'em.
We quickly found we had another problem: it was taking a lot of developer time to decide whether we should bring in the furballs for real-life interviews. So one night, while more than a little tipsy, I wrote Smoke.
We let our interview candidates write code in whatever they like: Java, C#, Python, Ruby… I needed a test framework that could handle any language under the sun. At first, I thought about ways to crowbar RSpec into running application tests. This was a stupid idea. Eventually I decided the only thing every language has in common is the command-line: every language can pretty easily support standard input and output (with the obvious exception of Java, which makes everything difficult).
I have to stress that this is not a replacement for looking over people's code. I've put people through that failed every one of my test cases because they understood the problem and mostly solved it. Similarly, someone that passes every case but writes Python like people wrote C in the 80s makes me very sad, despite all the green output from Smoke.
Issues and pull requests are very welcome. Please don't hesitate.
Developers of Smoke pledge to follow the Contributor Covenant.
We dog-food. You can run all of Smoke's smoke tests using:
On Windows, run this instead:
bin\smoke --command=bin\smoke test
Smoke development follows a few rules:
- The smoke binary is a single file without any dependencies. No gems are used.
- Smoke should work on Linux and Mac OS without any issue. Most features should also work on Windows, and we hope to get it to feature parity soon.