Writing bug-free code is very difficult. Some of the difficulty can be mitigated through the use of automated tests or executable specifications which can give programmers reasons for confidence in the code they write. With automated tests or executable specifications, the programmer expresses his intent twice – once in the code itself and once in the tests or specs. Without a double-check of programmer intent, it is easy for defects to creep into a program undetected.
Programs written in statically-typed languages are hard enough to get right. Any programmer worth her salt knows that just because her code compiles cleanly doesn’t mean that it works. An interpreted, dynamically-typed language like Ruby compounds the problem for programmers who don’t make use of automated tests or executable specifications. There are more degrees of freedom in Ruby code and therefore more dark places for bugs to hide.
So automated tests and executable specifications are good things. But lots of code doesn’t have test cases or specs. And just because there’s a test directory in a project tree doesn’t mean that the test cases it contains are worth much. They may have been wrong when they were written, they may be wrong because they haven’t been maintained along with the code, they may be difficult to maintain, or there just may not be enough of them to provide decent coverage. This is a sad reality for too many software projects.
What to do? Wouldn’t it be nice if you could have high-quality test cases generated for you automatically? At least then you would have a guard against regression when you make changes to a program.
Enter Trapeze, the safety-net generator for Ruby.
Trapeze generates a suite of unit tests or specifications for existing Ruby source code. This is accomplished through dynamic analysis, by reflecting on the public interfaces of classes, modules and top-level methods defined in the source. Trapeze then exercises each object, module and method, recording the behavior of the code. This recorded behavior is captured in a suite of test cases or specifications that can be rendered as test code or executable specifications.
There is a built-in suite renderer for the Test::Unit library. Other libraries such as RSpec, Shoulda and Jay Fields’s Expectations may be supported also at some future time.
In essence, Trapeze is a tool for characterizing Ruby source code. Trapeze lets you fly high as you maintain and enhance a Ruby codebase that lacks test or spec coverage, knowing that you have a regression safety net underneath you.
Trapeze is a pre-alpha product. Multiple major features are not yet ready for prime time. Stay tuned.
Trapeze is packaged as a RubyGem. You will need the RubyGems tool installed in order to take advantage of a one-step installation procedure. With RubyGems installed, type the following:
gem install --source http://trapeze.rubyforge.org trapeze
If the installation is successful, you should have the latest version of Trapeze installed on your system, and along with it, the trp command in your path.
Trapeze does just one thing: it generates a suite of test cases for a given Ruby codebase.
From the root of your Ruby project, type:
trp
This will search for files matching the pattern lib/*/.rb and generate a safety-net suite in the directory test/trapeze.
Type trp -h to learn about available options, including how to search for source code files matching a different pattern and how to generate a suite in a different output directory.
You can run a generated suite any time you want to check the behavior of your code against what is captured in the suite. Find the file named SUITE.rb which is located in the generated output directory. (By default this directory is test/trapeze.)
Execute it: ruby SUITE.rb.
Trapeze’s author is Nils Jonsson (nils@alumni.rice.edu). Released under the MIT License.
-
Build a system test coverage Rake task that leverages the aggregate option of RCov to aggregate the results of all the system tests
-
Search classes and modules recursively for nested classes and modules
-
Assert the values of user-defined constants and the superclasses of classes
-
Add support for –verbose/-v command-line option
-
Add support for thrown symbols and raised errors in addition to returned objects
-
Create mocks and stubs of core- and standard-library IO classes in order to make the generated suite more “unit-ish”
-
Figure out a way to probe top-level methods that have required parameters
-
Add Rake tasks for preparing RCov aggregate statistics on the system tests
-
Add support for excluding source code files and method name patterns from scanning, since the engine executes code in order to inspect it
-
Address the elephant-in-the-room question of code execution paths and how to get predictable code coverage in generated tests – create an alternative implementation of Trapeze::Probe that leverages the ParseTree gem