Unit testing with the Apple platforms, both desktop and iPhone, is very frustrating.
The unit testing framework, SenTestingKit (a.k.a OCUnit) is lovely and does just about everything I could want it to do. The set of asserts available satisfies my needs and writing test cases is pretty darn straightforward. Implement a method that starts with “test” in a subclass of SenTestCase and you’re good to go; hooray for dynamic languages!
The frustrating part is the way that Apple has set up the Unit Testing mechanisms they’ve built into Xcode. They go out of their way to separate the unit testing code from the actual application code. Then they use some fancy code loading techniques to “inject” the unit tests into the application code and run them. The upshot of lots of magic scripts and voodoo is that it is very unlikely that you will ever accidentally ship unit testing code to your application’s users. It also limits Heisenberg Uncertainty problems where your unit tests might cause unusual side effects in your running application.
But it makes working with the unit tests a pain.
On the desktop side of things, I guess it’s nice that you can set up the unit tests to run every time you build the unit test bundle. But unit tests are “just code” and code has to be debugged. When your unit tests are being run by the build scripts as part of the build process it’s really hard to attach the debugger to them. If you find the right article on Apple’s web site (Unit Testing with Xcode 3), you can set up a custom executable for your application that will inject the tests into your application. Doing so involves working with some command-line options and environment variables that instruct the loader to do the injection. The whole thing looks like voodoo incantations and black magic. It is also complex enough that I have to go look it up. Every. Single. Time. (well look it up or track down the AppleScript that does it).
On the iPhone you can use similar procedures, but the unit tests you create run at build-time have to be stand-alone tests; they don’t execute in the context of your full application. These tests can only be run if you have the current target set to the iPhone simulator. You can create unit tests in a bundle that do run in the full context of your application (through the magic of run-time bundle injection), but they can only be run on device.
In all cases, what you get when you run the unit tests is a console level transcript of which tests were run and which succeeded or failed. On the whole the transcript thing is functional, but I wouldn’t call it useable.
Now I’m a long time Mac developer and the Test Driven Development thing is relatively new to me. The hip kiddees who are slumming away from their Java programming roots and trying out Objective-C on the iPhone are constantly whining about the lack of proper unit testing tools. They bemoan the fact that there’s not a graphical browser for looking at unit test results, or the fact that there’s no XML output of those results.
They may have a point, but I can’t complain about that stuff because I don’t know any better.
Still, I don’t like combing through terminal traces; it would be nice to have better looking test results. Besides, our CIT server understands JUnit-like XML output so if we could get JUnit XML out of our unit tests that could be very handy too.
The nice thing about being a software engineer is that when software is frustrating you, you can do something about it. If it’s an interesting problem to play with, so much the better.
The first liberating thought I had about the unit test setup for the Cocoa platforms is that the SenTestingKit framework is just a framework. It’s not even Apple’s framework, it was created by a developer (or developers) in Switzerland if their domain suffix is to be believed. The framework itself existed long before Apple integrated it into the Xcode so the framework can simply be incorporated into an application like any other framework.
The second liberating thought is the realization that I don’t have to use dyld injection to get the unit tests into my application. I can compile them in just as easily. Sure there’s a chance that I might ship a build to my customers with unit testing code included… But I can put a line item on my shipping checklist to ensure that there’s no unit test code left in the application. If I’m really clever and diligent with the C preprocessor I can make sure that the unit tests are only compiled in when I want them to be.
The third liberating discovery, if you look through OCUnit, is that the framework provides a really simple mechanism for learning about the unit test life cycle. It will happily tell you when a test case begins or ends, when a suite is run, what the results were, what errors caused tests to fail etc.
The result of all this liberating thought? It’s not that hard to write a class that collects the test results and does whatever you care to do with them. In a couple of hours I put together a unit test runner that shows up as a modal panel in an iPhone app and displays the results of running the unit tests. Another hour (learning about the iPhone search box control) and I had some nice searching capabilities built into the panel.
Because Objective-C and SenTestingKit are the same on the desktop and iPhone I was able to re-use the results collector on the desktop. I haven’t created a snazzy panel for displaying the results as nice as the one I have for the iPhone but that will come in time.
Getting the code to dump out XML results took a bit more time. I had to find some kind of spec for the JUnit XML format. I eventually found an XSD file that looked promising. I had to learn enough libxml2 to dump the results out in that format and our CIT server seems to like the results. It may not be exactly what JUnit would produce, but evidently it’s close enough. Because i used libxml2, the same code runs on the desktop and the phone.
I don’t get my tests automatically running at build time, I actually have to run the application, but the build results are nice to look at. It’s easy to see which tests succeeded and failed and our CIT server has some way to examine the results as well. All in all it was not too difficult and I got satisfying results for the effort. I would encourage other developers that are disappointed with Xcode’s unit testing mechanism to learn a bit more about OCUnit/SenTestingKit.