ArduinoUnit

ArduinoUnit is a testing framework for Arduino projects. It supports Arduino, ESP8266 and ESP32 as well as "en vitro" development system (vs embedded target) testing.

Getting Started

Install the library from the Arduino IDE. From the menu, navigate:

• Sketch->Include Library->Manage Libraries...
• Search for "arduinounit"
• Install

After this, examples should be available from File->Examples in the "Examples from Custom Libraries" section.

Here is a simple unit testing sketch:

#line 2 "sketch.ino"
#include <ArduinoUnit.h>

test(ok) 
{
  int x=3;
  int y=3;
  assertEqual(x,y);
}

test(bad)
{
  int x=3;
  int y=3;
  assertNotEqual(x,y);
}

void setup()
{
  Serial.begin(9600);
  while(!Serial) {} // Portability for Leonardo/Micro
}

void loop()
{
  Test::run();
}

Upload this sketch to the Arduino (using the 'Upload' button, File | Upload or Ctrl+U).

Turn on the Serial Monitor (using the 'Serial Monitor' button, Tools | Serial Monitor or Ctrl+Shift+M) and expect to see the following:

Assertion failed: (x=3) != (y=3), file sketch.ino, line 17.
Test bad failed.
Test ok passed.
Test summary: 1 passed, 1 failed, and 0 skipped, out of 2 test(s).

The following asserts are supported [with an optional footnote and return value]

Assertion	Description
assertEqual(a,b [,footnote [,retval]])	a == b?
assertNear(a,b,maxerr, [,footnote[,retval]])	abs(b-a)<=maxerr?
assertRelativelyNear(a,b,maxerr, [,footnote[,retval]])	abs(b-a)/abs(½(abs(a)+abs(b)))<=maxerr?
assertNotEqual(a,b [,footnote[,retval]])	a != b?
assertLess(a,b [,footnote[,retval]])	a < b?
assertLessOrEqual(a,b [,footnote[,retval]])	a <= b?
assertMore(a,b [,footnote[,retval]])	a > b?
assertMoreOrEqual(a,b [,footnote[,retval]])	a >= b?
assertTrue(p [,footnote[,retval]])	same as assertEqual(p,true)
assertFalse(p [,footnote[,retval]])	same as assertEqual(p,false)

[,footnote[,retval]]

Assertions are replaced with essentially (retval is the optional last value of an assert):

if (not assertion) { fail(); return [retval]; }

When things go wrong, it can be useful to print additional information. As of 2.3.2-alpha, this is possible with any assertXXX() method by adding an additional third parameter [footnote] to the assert. For example,

test(cases)
{
  int x=3;
  for (int k=0; k<4; ++k) {
    assertNotEqual(x,k,"case k=" << k);
  }
}

will fail with the message

Assertion failed: (x=3) != (k=3), file basic.ino, line 20 [case k=3].

The additional message is only created if the assert actually needs to generate output (usually when it fails). It appears in the [] brackets at the end of the assert message. Notice you can create fairly complex messages by chaining things you can print (like Serial.print()) between << operators. This is similar to the C++ ostream insertion operators, if you are familar with that.

The status of the test can be used (bool ok) when printing the message. Under normal verbosity settings, ok will always be false, but more verbose settings can print assert messages even if they pass.

Selecting tests

In your setup() function, you can select which tests are going to be setup and looped. The default is that all tests are included.

Test::exclude(const char *pattern) removes all the tests that match the given *-pattern.

Test::include(const char *pattern) includes all the tests that match the given *-pattern.

Here are some examples:

Select examples:

A single test my_test

void setup()
{
  Serial.begin(9600);
  while(!Serial) {} // Portability for Leonardo/Micro
  Test::exclude("*");
  Test::include("my_test");
}

All tests named dev_-something, but not the ones ending in _skip or _slow, or have the word eeprom in them:

void setup()
{
  Serial.begin(9600);
  while(!Serial) {} // Portability for Leonardo/Micro
  Test::exclude("*");
  Test::include("dev_*");
  Test::exclude("*_slow");
  Test::exclude("*_skip");
  Test::exclude("*eeprom*");
}

Re-running tests (advanced)

Typically, you just want to run all tests once and then show the result. If so, you can skip this section.

In more advanced situations, you might want to run the entire test suite multiple times (for example if your tests can be configured with different parameters). To facilitate this, you can use the Test::resetDoneTests() function.

Calling this function will reset all completed tests (passed, failed or skipped) back to their initial state. For tests that define a setup method, this will be run again on the next Test::run(). If any tests were not completed yet, these are unaffected. The statistics (number of passed, failed and skipped tests) are also reset to 0.

Note that excluded tests (using Test::exclude()) are treated as skipped tests, so these are also re-run (you will need to call Test::exclude() again after resetDoneTests() if you want keep these tests excluded). Tests removed with their remove() method are really removed, so not re-run when using resetDoneTests().

Typically, you would use this after all tests are completed (but if you call resetDoneTests() when some tests are still pending, those tests are unaffected). You must never call resetDoneTests() from inside a test, only between calls to Test::run().

Below is an example that runs all tests once, then changes a global variable and runs all tests again. To have a bit more direct control over running the tests, this example does not call Test::run() infinitely in the loop(), but instead uses Test:runUntilDone() which repeatedly calls Test::run() until all tests are completed.

bool some_global_setting = false;

void setup() {
  Serial.begin(9600);
  while(!Serial) {} // Portability for Leonardo/Micro

  Test::runUntilDone();

  some_global_setting = true;
  Test::resetDoneTests();
  Test::runUntilDone();
}

void loop() { }

Output

The Test::out value is the shared value for all tests describing where output for all tests goes. The default is

Test::out = &Serial;

But you can set to the address of any Print stream. For example, if you want the output to be on the Serial3 device on the arduino mega, use

Test::out = &Serial3;

in your setup(). Note the library does not set the baud rate - you have to do that in your setup().

Verbosity

Normal ArduinoUnit verbosity reports only failed assertions, the status (pass,skip,fail) of completed tests, and a summary.

Seeing more.

It is often useful to see the results of assertions [and footnotes] even when they pass. If you want to trace everything in this way, you can turn on all output with Test::min_verbosity = TEST_VERBOSITY_ALL in your setup.

Seeing less more.

The previous choice is great until you are lost in an ocean of messages for tests you do not want to watch at the moment. Instead of globally setting min_verbosity/max_verbosity in your setup(), you can instead use verbosity = TEST_VERBOSITY_ALL in a given test to see everything about that test.

MockPrint and MockStream (intermediate)

MockPrint is provided by ArduinoUnit to mimic a real output device, like Serial, but is also a String which happens to contain the information printed to it. This can be used to test output formatting, as in:

void format(Print &out, int value) {
  out.print("decimal ");
  out.print(value);
  out.print(" is hex ");
  out.println(value,HEX);
}

test(format) {
  MockPrint mp;
  format(mp,32); // test as mock
  assertEqual(mp,"decimal 32 is hex 20\r\n");
}

void setup() {
  Serial.begin();
  while (!Serial) {}
  format(Serial,100); // format to serial
}

void loop() {
  Test::run();
}

MockStream is provided by ArduinoUnit to mimic a real input/output device, like Serial. It contains two MockPrint parts, input contains the input that will be read from the MockStream, and output which contains the output that was written. This can be used to test input and output, as in:

void square(Stream &io) {
  io.print("value? ");
  int x = io.parseInt();
  out.print(value);  
  out.print("*");
  out.print(value);
  out.print("=");
  out.println(x*x);
}

test(square) {
  MockStream ms;
  ms.input.print(10);
  square(ms);
  assertEqual(ms.output,"value? 10*10=100\r\n");
}

void setup() {
  Serial.begin();
  while (!Serial) {}
  square(Serial); // format to serial
}

void loop() {
  Test::run();
}

The mockstream example shows a convenient way to switch between real and mock streams for testing.

En Vitro Testing (advanced)

ArduinoUnit will compile in a standard C++ environment (LLVM or GCC) with -std=gnu++11. The advanced example has a makefile and main.cpp to support this.

Note ArduinoUnit has very limited mocking features; you will have to include the mocking features you need to simulate the embedded environment. The main.cpp file in the advanced example illustrates minimal mocking. In particular the only features provided (because of dependencies on these by ArduinoUnit) are:

F()
millis()
micros()
String
Print
Printable
Stream # public components only

These are available via #include "ArduinoUnitMock.h". In the mock environment, there are two additional objects, CppStreamPrint and CppIOStream, which wrap C++ std::ostream (and std::istream for CppIOStream). This simplifies creating tests in the mocking environments. Look at the advanced example and test firmware for guidance.

Verbosity (advanced)

Just how much information is generated on each test is fairly flexible, and designed to address these rules:

1. How much code is generated (TEST_MAX_VERBOSITY)
2. Global maximums subject to 1 (static Test::max_verbosity)
3. Global minimums subject to 1 and 2 (static Test::min_verbosity)
4. Per-test requirements subject to 1, 2 and 3 (Test::verbosity)

The rules are as follows for each kind of possible output flag:

1. Is the flag set in TEST_MAX_VERBOSITY? If not set, then the output is suppressed and we are done.

2. Is the flag set in Test::max_verbosity? If not set, then the output is suppressed and we are done.

3. Is the flag set in Test::min_verbosity? If set, then the output is generated and we are done.

4. Are we in a test context (Test::run())?

   If so, the output is generated if the corresponding per-test verbosity flag is set.

The default values are as follows

   TEST_MAX_VERBOSITY = TEST_VERBOSITY_ALL

   static Test::max_verbosity = TEST_VERBOSITY_ALL
   static Test::min_verbosity = TEST_VERBOSITY_TESTS_SUMMARY

   Test::verbosity = (TEST_VERBOSTY_TESTS_ALL|TEST_VERBOSITY_ASSERTIONS_FAILED)

This amounts to asking for a summary of each test (skip, pass, fail), an overall summary when all tests are resolved, and a more detailed report on each of the failed assertions.

The verbosity flags are the bitwise-or of the following values

TEST_VERBOSITY_TESTS_SUMMARY      (0x01)
TEST_VERBOSITY_TESTS_FAILED       (0x02)
TEST_VERBOSITY_TESTS_PASSED       (0x04)
TEST_VERBOSITY_TESTS_SKIPPED      (0x08)
TEST_VERBOSITY_TESTS_ALL          (0x0F)
TEST_VERBOSITY_ASSERTIONS_FAILED  (0x10)
TEST_VERBOSITY_ASSERTIONS_PASSED  (0x20)
TEST_VERBOSITY_ASSERTIONS_ALL     (0x30)
TEST_VERBOSITY_ALL                (0x3F)
TEST_VERBOSITY_NONE               (0x00)

Built-in Assertions (details)

The following assertions are supported

assertLess(arg1,arg2)
assertLessOrEqual(arg1,arg2)
assertEqual(arg1,arg2)
assertNotEqual(arg1,arg2)
assertMoreOrEqual(arg1,arg2)
assertMore(arg1,arg2)

Anything that can be compared via a '<' comparison between them can be used.

All the string-like types (String, char *, char[] and flash string literals) can be used interchangeably in assertions, i.e.:

test(strings) {
   const char *cOk="ok";
   char aOk[3]; 
   String sOk(cOk);
   // two underbars (_) for a flash string literal...
   const __FlashStringHelper *fOk = F("ok"); 

   strcpy(aOk,cOk);

   assertEqual(cOk,aOk,F("char* vs char[]"));
   assertEqual(aOk,sOk,F("char[] vs String"));
   assertEqual(sOk,fOk,F("String vs flash"));
   assertEqual(fOk,cOk,F("flash vs char*"));   
}

__FlashStringHelper ?

Note that using a flash string literal directly (except the footnote) in an assert is not supported. You must declare and use them separately as above. The main reason for this is the F() macro expands to a large useless expression, which is then represented in flash as part of the assert message. The alternate version keeps the assert message small and readable.

There are addtionally some boolean assertions:

assertTrue(arg)
assertFalse(arg)

These are shorthands for assertEqual(arg,true) and assertEqual(arg,false).

See the section below for assertions on tests.

The output from these assertions is to print a string represenation of the arguments, and the value of the arguments, as in:

Assertion passed/failed: (arg1=value1) op (arg2=value2), file name, line #.

These assertions are defined in a way that the problem of multiple evaluations is avoided. The arguments are only evaluated once in these assertions.

All the assert macros expand to a test that creates an optional message, and, if false, calls fail() on the current test and returns.

Meta Assertions (Advanced)

You can make assertions on the outcome of tests as well. The following meta-assertions are supported:

Meta Assertion	Description
assertTestDone(test [,footnote[,retval]])	test done (skip, pass or fail)?
assertTestNotDone(test [,footnote[,retval]])	test not done?
assertTestPass(test [,footnote[,retval]])	test passed?
assertTestNotPass(test [,footnote[,retval]])	test not passed (fail, skip, or not done)?
assertTestFail(test [,footnote[,retval]])	test failed?
assertTestNotFail(test [,footnote[,retval]])	test not failed (pass, skip, or not done)?
assertTestSkip(test [,footnote[,retval]])	test skipped?
assertTestNotSkip(test [,footnote[,retval]])	test not skipped (pass, fail, or not done)?

Meta Assertion	Description
assertCurrentTestDone([footnote[,retval]])	current test done (skip, pass or fail)?
assertCurrentTestNotDone([footnote[,retval]])	current test not done?
assertCurrentTestPass([footnote[,retval]])	current test passed?
assertCurrentTestNotPass([footnote[,retval]])	current test not passed (fail, skip, or not done)?
assertCurrentTestFail([footnote[,retval]])	current test failed?
assertCurrentTestNotFail([footnote[,retval]])	current test not failed (pass, skip, or not done)?
assertCurrentTestSkip([footnote[,retval]])	current test skipped?
assertCurrentTestNotSkip([footnote[,retval]])	current test not skipped (pass, fail, or not done)?

These can be used in conjunction with the boolean check-only macros

checkTestDone(test) / checkCurrentTestDone()
checkTestNotDone(test) / checkCurrentTestNotDone()
checkTestPass(test) / checkCurrentTestPass()
checkTestNotPass(test) / checkCurrentTestNotPass()
checkTestFail(test) / checkCurrentTestFail()
checkTestNotFail(test) / checkCurrentTestNotFail()
checkTestSkip(test) / checkCurrentTestSkip()
checkTestNotSkip(test) / checkCurrentTestNotSkip()

These behave like the other asserts, but they work only in the context of other tests. The most likely place you would have such a test would be in a testing meta-test as so:

test(ok) { pass(); }
test(bad) { fail(); }
testing(slow) { if (millis() > 1000) pass(); }

testing(passed) 
{
  if (checkTestDone(ok)) {
    assertTestPass(ok);
    pass();
  }
}

testing(too_slow)
{
  if (millis() > 100) {
    assertTestDone(slow);
    pass();
  }
}

Test and TestOnce (advanced)

You can create your own modular tests by deriving from these classes.

class MyTest : public Test {
private:
  void construct() {
    // TODO: construct named test.
    // This should be lightweight - it may be excluded
    //
    // You can set verbosity.
  }
public:
  MyTest(const char *name) : Test(name) {construct();}
#if defined(F)
  MyTest(const __FlashStringHelper *name) {construct();}
#endif
  void setup() {
    // TODO: setup test
    // You can call pass(), fail(), or skip() to immediately resolve test
    // You can make assertions.
    // You can set verbosity.
  }
  void loop() {
    // TODO: run test on each loop
    // You can call pass(), fail(), or skip() to resolve test
    // You can make assertions.
    // You can set verbosity.
  }
};

class MyTestOnce : public TestOnce
{
public:
  MyTestOnce(const char *name) : TestOnce(name) {
  // same as MyTest
  }
  void setup() {
  // same as MyTest
  }
  void once() {
  // same as MyTest::loop(), but will only be called once from loop()
  // if included in the active tests and was not resolved in setup().
  }  
}

// create instances of the custom test 
MyTest myTest1(F("myTest1"));
MyTest myTest2(F("myTest2"));

MyTestOnce myTestOnce1("myTestOnce1");
MyTestOnce myTestOnce2("myTestOnce2");

Note that Test::run() only calls the active unresolved tests.

Known Bugs

• The assertCurrentTestXXXX([,footnote [,retval]) macros do not compile on ESP8266 boards with no footnote. Use an empty footnote assertCurrentTestXXXX(""), or use assertCurrentTestXXXX_0() for no footnote. You do not have to specify a return value.

FAQ

Q. The line number of the asserts do not match the source file.

A. As far as I can tell, this is a bug in the compiler -- look two lines up. I do not know why the __LINE__ macro does not match the actual line of code.

Q. What's with the # 2 "file.ino" business in the examples?

A. This is to address question 1 above, and, without this line, the filename will be a very long and mostly useless name in the asserts, like,

/var/folders/gr/n9s7qtcs2qqbdnmcgm6gjzrm0000gp/T/build2118014134542174575.tmp/sketch_mar17a.ino

This uses up flash memory space and doesn't give any useful information when something goes wrong.

Q. I get these link errors about multiply defined test_XXXX_instance.

A. You have defined two tests with the same name XXXX using either the test() or testing() macro.

Q. I get no output

A. Here is a troubleshooting guideline:

• Make sure you call Serial.begin() in your setup. Or, if you redirect output by changing the value of Test::out, make sure you configure the Print stream you direct it to.
• If you are using an Arduino Leonardo/Micro: don't forget to add while(!Serial) {} after Serial.begin(9600) in the setup(). Without this line nothing will be printed in the serial monitor.
• Make sure you call Test::run() in your loop().
• Make sure you did not exclude the test(s) with Test::exclude(pattern). By default all tests are included.
• Make sure your tests complete.
  • Each single-check test() test not be in an infinite loop.
  • Each continous testing() test do a small amount of work on each call, and should eventually invoke pass(), fail() or skip().
• Make sure verbosity is adequate. You can generate all possible output by
  • Assuring that TEST_MAX_VERBOSITY in ArduinoUnit.h is TEST_VERBOSITY_ALL (the default).
  • Assuring that Test::max_verbosity is TEST_VERBOSITY_ALL (the default).
  • Setting Test::min_verbosity = TEST_VERBOSITY_ALL (the default is TEST_VERBOSITY_TESTS_ALL | TEST_VERBOSITY_ASSERTIONS_FAILED, generating output only for failed assertions, completions of tests, and an overall summary).
  • With these settings, the per-test verbosity has no effect.