Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Failed to load latest commit information.|
WvTest: the dumbest cross-platform test framework that could possibly work ========================================================================== I have a problem with your unit testing framework. Yes, you. The person reading this. No, don't look away guiltily. You know your unit testing framework sucks. You know it has a million features you don't understand. You know you hate it, and it hates you. Don't you? Okay, fine. Let's be honest. Actually, I don't know who you are or how you feel about your unit testing framework, but I've tried a lot of them, and I don't like any of them. WvTest is the first one I don't hate, at least sort of. That might be because I'm crazy and I only like things I design, or it might be because I'm crazy and therefore I'm the only one capable of designing a likable unit testing framework. Who am I to say? Here are the fundamental design goals of WvTest: - Be the stupidest thing that can possibly work. People are way, way too serious about their testing frameworks. Some people build testing frameworks as their *full time job*. This is ridiculous. A test framework, at its core, only does one thing: it runs a program that returns true or false. If it's false, you lose. If it's true, you win. Everything after that is gravy. And WvTest has only a minimal amount of gravy. - Be a protocol, not an API. If you don't like my API, you can write your own, and it can still be WvTest and it can still integrate with other WvTest tools. If you're stuck with JUnit or NUnit, you can just make your JUnit/NUnit test produce WvTest-compatible output if you want (although I've never done this, so you'll have to do it yourself). I'll describe the protocol below. - Work with multiple languages on multiple operating systems. I'm a programmer who programs on Linux, MacOS, and Windows, to name just three, and I write in lots of programming languages, including C, C++, C#, Python, Perl, and others. And worse, some of my projects use *multiple* languages and I want to have unit tests for *all* of them. I don't know of any unit testing framework - except maybe some horrendously overdesigned ones - that work with multiple languages at once. WvTest does. - NO UNNECESSARY OBJECT ORIENTATION. The big unit testing craze seems to have been started by JUnit in Java, which is object-oriented. Now, that's not a misdesign in JUnit; it's a misdesign in Java. You see, you can't *not* encapsulate absolutely everything in Java in a class, so it's perfectly normal for JUnit to require you to encapsulate everything in a class. That's not true of almost any other language (except C#), and yet *every* clone of JUnit in *every* language seems to have copied its classes and objects. Well, that's stupid. WvTest is designed around the simple idea of test *functions*. WvTest runs your function, it checks a bunch of stuff and it returns or else it dies horribly. If your function wants to instantiate some objects while it does that, then that's great; WvTest doesn't care. And yes, you can assert whether two variables are equal even if your function *isn't* in a particular class, just as God intended. - Don't make me name or describe my individual tests. How many times have you seen this? assertTrue(thing.works(), "thing didn't work!"); The reasoning there is that if the test fails, we want to be able to print a user-friendly error message that describes why. Right? NO!! That is *awful*. That just *doubled* the amount of work you have to do in order to write a test. Instead, WvTest auto-generates output including the line number of the test and the code on that line. So you get a message like this: ! mytest.t.cc:431 thing.works() FAILED and all you have to write is this: WVPASS(thing.works()); (WVPASS is all-caps because it's a macro in C++, but also because you want your tests to stand out. That's what you'll be looking for when it fails, after all. And don't even get me started about the 'True' in assertTrue. Come on, *obviously* you're going to assert that the condition is true!) - No setup() and teardown() functions or fixtures. "Ouch!" you say. "I'm going to have so much duplicated code!" No, only if you're an idiot. You know what setup() and teardown() are code names for? Constructor and destructor. Create some objects and give them constructors and destructors, and I think you'll find that, like magic, you've just invented "test fixtures." Nothing any test framework can possibly do will make that any easier. In fact, everything test frameworks *try* to do with test fixtures just makes it harder to write, read, and understand. Forget it. - Big long scary test functions. Some test frameworks are insistent about the rule that "every function should test only one thing." Nobody ever really explains why. I can't understand this; it just causes uncontrolled hormone-imbalance hypergrowth in your test files, and you have to type more stuff... and run test fixtures over and over. My personal theory for why people hate big long test functions: it's because their assertTrue() implementation doesn't say which test failed, so they'd like the *name of the function* to be the name of the failed test. Well, that's a cute workaround to a problem you shouldn't have had in the first place. With WvTest, WVPASS() actually tells you exactly what passed and what failed, so it's perfectly okay - and totally comprehensible - to have a sequence of five things in a row where only thing number five failed. The WvTest Protocol ------------------- WvTest is a protocol, not really an API. As it happens, the WvTest project includes several (currently five) implementations of APIs that produce data in the WvTest format, but it's super easy to add your own. The format is really simple too. It looks like this: Testing "my test function" in mytest.t.cc: ! mytest.t.cc:432 thing.works() ok This is just some crap that I printed while counting to 3. ! mytest.t.cc.433 3 < 4 FAILED There are only four kinds of lines in WvTest, and each of the lines above corresponds to one of them: - Test function header. A line that starts with the word Testing (no leading whitespace) and then has a test function name in double quotes, then "in", then the filename, and then colon, marks the beginning of a test function. - A passing assertion. Any line that starts with ! and ends with " ok" (whitespace, the word "ok", and a newline) indicates one assertion that passed. The first "word" on that line is the "name" of that assertion (which can be anything, as long as it doesn't contain any whitespace). Everything between the name and the ok is just some additional user-readable detail about the test that passed. - Random filler. If it doesn't start with an ! and it doesn't look like a header, then it's completely ignored by anything using WvTest. Your program can print all the debug output it wants, and WvTest won't care, except that you can retrieve it later in case you're wondering why a test failed. Naturally, random filler *before* an assertion is considered to be associated with that assertion; the assertion itself is the last part of a test. - A failing assertion. This is just like an 'ok' line, except the last word is something other than 'ok'. Generally we use FAILED here, but sometimes it's EXCEPTION, and it could be something else instead, if you invent a new and improved way to fail. Reading the WvTest Protocol: wvtestrun -------------------------------------- WvTest provides a simple perl script called wvtestrun, which runs a test program and parses its output. It works like this: cd python ../wvtestrun ./wvtest.py t/twvtest.py (Why can't we just pipe the output to wvtestrun, instead of having wvtestrun run the test program? Three reasons: first, a fancier version of wvtestrun could re-run the tests several times or give a GUI that lets you re-run the test when you push a button. Second, it handles stdout and stderr separately. And third, it can kill the test program if it gets stuck without producing test output for too long.) If we put the sample output from the previous section through wvtestrun (and changed the FAILED to ok), it would produce this: $ ./wvtestrun cat sample-ok Testing "all" in cat sample-ok: ! mytest.t.cc my ok test function: .. 0.010s ok WvTest: 2 tests, 0 failures, total time 0.010s. WvTest result code: 0 What happened here? Well, wvtestrun took each test header (in this case, there's just one, which said we're testing "my test function" in mytest.t.cc) and turns it into a single test line. Then it prints a dot for each assertion in that test function, tells you the total time to run that function, and prints 'ok' if the entire test function failed. Note that the output of wvtestrun is *also* valid WvTest output. That means you can use wvtestrun in your 'make test' target in a subdirectory, and still use wvtestrun as the 'make test' runner in the parent directory as well. As long as your top-level 'make test' runs in wvtestrun, all the WvTest output will be conveniently summarized into a *single* test output. Now, what if the test had failed? Then it would look like this: $ ./wvtestrun cat sample-error Testing "all" in cat sample-error: ! mytest.t.cc my error test function: . ! mytest.t.cc:432 thing.works() ok This is just some crap that I printed while counting to 3. ! mytest.t.cc.433 3 < 4 FAILED 0.000s ok WvTest: 2 tests, 1 failure, total time 0.000s. WvTest result code: 0 What happened there? Well, because there were failed tests, wvtestrun decided you'd probably want to see the detailed output for that test function, so it expanded it out for you. The line with the dots is still there, but since it doesn't have an 'ok', it's considered a failure too, just in case. Watch what happens if we run a test with both the passing, and then the failing, test functions: $ ./wvtestrun cat sample-ok sample-error Testing "all" in cat sample-ok sample-error: ! mytest.t.cc my ok test function: .. 0.000s ok ! mytest.t.cc my error test function: . ! mytest.t.cc:432 thing.works() ok This is just some crap that I printed while counting to 3. ! mytest.t.cc.433 3 < 4 FAILED 0.000s ok WvTest: 4 tests, 1 failure, total time 0.000s. WvTest result code: 0 Notice how the messages from sample-ok are condensed; only the details from sample-error are expanded out, because only that output is interesting. How do I actually write WvTest tests? ------------------------------------- Sample code is provided for these languages: C: try typing "cd c; make test" C++: try typing "cd cpp; make test" C# (mono): try typing "cd dotnet; make test" Python: try typing "cd python; make test" Shell: try typing "cd sh; make test" There's no point explaining the syntax here, because it's really simple. Just look inside the cpp, dotnet, python, and sh directories to learn how the tests are written. How should I embed WvTest into my own program? ---------------------------------------------- The easiest way is to just copy the WvTest source files for your favourite language into your project. The WvTest protocol is unlikely to ever change - at least not in a backwards-incompatible way - so it's no big deal if you end up using an "old" version of WvTest in your program. It should still work with updated versions of wvtestrun (or wvtestrun-like programs). Another way is to put the WvTest project in a subdirectory of your project, for example, using 'svn:externals', 'git submodule', or 'git subtree'. How do I run just certain tests? -------------------------------- Unfortunately, the command-line syntax for running just *some* of your tests varies depending which WvTest language you're using. For C, C++ or C#, you link an executable with wvtestmain.c or wvtestmain.cc or wvtestmain.cs, respectively, and then you can provide strings on the command line. Test functions will run only if they have names that start with one of the provided strings: cd cpp/t ../../wvtestrun ./wvtest myfunc otherfunc With python, since there's no linker, you have to just tell it which files to run: cd python ../wvtestrun ./wvtest.py ...filenames... What else can parse WvTest output? ---------------------------------- It's easy to parse WvTest output however you like; for example, you could write a GUI program that does it. We had a tcl/tk program that did it once, but we threw it away since the command-line wvtestrun is better anyway. One other program that can parse WvTest output is gitbuilder (http://github.com/apenwarr/gitbuilder/), an autobuilder tool for git. It reports a build failure automatically if there are any WvTest-style failed tests in the build output. Other Assorted Questions ------------------------ What does the "Wv" stand for? Either "Worldvisions" or "Weaver", both of which were part of the name of the Nitix operating system before it was called Nitix, and *long* before it was later purchased by IBM and renamed to Lotus Foundations. It does *not* stand for World Vision (sigh) or West Virginia. Who owns the copyright? While I (Avery) wrote most of the WvTest framework in C++, C#, and Python, and I also wrote wvtestrunner, the actual code I wrote is owned by whichever company I wrote it for at the time. For the most part, this means: C++: Net Integration Technologies, Inc. (now part of IBM) C#: Versabanq Innovations Inc. Python: EQL Data Inc. What can I do with it? WvTest is distributed under the terms of the GNU LGPLv2. See the file LICENSE for more information. Basically this means you can use it for whatever you want, but if you change it, you probably need to give your changes back to the world. If you *use* it in your program (which is presumably a test program) you do *not* have to give out your program, only WvTest itself. But read the LICENSE in detail to be sure. Where did you get the awesome idea to use a protocol instead of an API? The perl source code (not to be confused with perlunit) did a similar trick for the perl interpreter's unit test, although in a less general way. Naturally, you shouldn't blame them for how I mangled their ideas, but I never would have thought of it if it weren't for them. Who should I complain to about WvTest? Email me at: Avery Pennarun <firstname.lastname@example.org> I will be happy to read your complaints, because I actually really like it when people use my programs, especially if they hate them. It fills the loneliness somehow and prevents me from writing bad poetry like this: Testing makes me gouge out my eyes But with WvTest, it takes fewer tries. WvTest is great, wvtest is fun! Don't forget to call wvtestrun.