Integration with AutoTools (and other test frameworks)?

[sbseltzer]

Hi there. CMock is wonderful. Thanks for making it!

I've been evaluating a number of mocking/testing frameworks for an application-layer embedded Linux library suite, and so far the CMock API is my favorite. However, we use AutoTools for builds and it doesn't seem to integrate well with the bureaucratic nature of Automake.

I think the problem is that it doesn't quite philosophically align well with Automake since it builds sources in a way that doesn't play nice with Automake's dependency tracking, which in turn makes test drivers less happy. I think this is a necessary evil of how Automake handles distributions, but it would be nice if we could find a way to make integration easier. Maybe we could collaborate to find an elegant pattern and perhaps a way to automate it?

My basic theory is, depending on test strategy, to either conditionally include the generated sources in the real libraries or build mocked sources into their own counterparts of the real libraries. These mocked implementations could then be linked into tests with much less fuss.

Idea A - Facilitate use of the __wrap_ prefix by allowing the user to specify an arbitrary function prefix. This way the user could use the --wrap linker flag on GNU compliant linkers. This has a number of advantages in test environments, but would require CMock to have a feature like the one described in #32

• User could (optionally) provide an arbitrary prefix for the mocked function (such as __wrap_).

It seemed to be given a rain-check due to it being compiler specific, but I believe that having an option for it in the cmock.yml could be beneficial in portable use-cases as well. For example, it would enable the user to implement their own stubbing of CMock'd functions even without the --wrap a la ifdef. It wouldn't be as flexible as --wrap, but it would be the most portable. A user may even be able to do this stubbing dynamically at runtime, which is another candidate for code generation, but I haven't fleshed that idea out at all.

Idea B - There's also weak attributes, which are a little more portable than --wrap, and even available in MSVC despite being undocumented. That's another potential solution I've considered exploring since it could provide functionality similar to --wrap that's slightly more portable. This could also be used in conjunction with the aforementioned prefix stubbing to do one-off mock substitution in test suites. This kind of thing could alternatively be complemented by having some kind of build automation strategy that stubs out individual functions as their own shared libraries that follow some kind of naming convention, which would simulate --wrap behaviour in a portable way. This idea is kind of vague, but might inspire something more complete.

In the case of --wrap, testing could be tuned to the function-level. In the case of weak attributes, it could be tuned to the library-level (at least). In the case of stubbing prefixed mocks the result would be similar to weak attributes, though possibly more cumbersome to implement, but also potentially quite powerful for clever users.

I agree that portability is a good priority for this framework, but I also think that having the facilities available to make it fit cleanly in build frameworks besides Ceedling is equally important. At the very least, having a clean integration strategy is what I'm looking for. Thoughts?

[mvandervoord]

I'm not terribly familiar with Automake. What is it about the way CMock builds things that doesn't integrate well with it already? Is it an issue of knowing WHEN to make a mock?

[sbseltzer]

Hey, thanks for the prompt reply. I really appreciate your taking the time to review and respond to these massive walls of text. :)

Yes, the WHEN is a big part of it. There are two other parts.

1. When you tell Automake about a source file, it is generally sacred to the distribution/dependency process. Built sources have a strange interaction this, and can add significant complexity.
2. Mocking library interfaces wholesale is almost never desirable when testing interactions within a single library.

The TLDR is that granular mocking is currently impossible without a massive explosion of header files, and since generated sources need their dependencies managed manually in Automake, maintenance gets out of hand rather quickly. Otherwise you risk name conflicts and outright inability to mock interactions between functions in the same interface. Putting a tiny bit more control in the user's hands for fine-tuning the compilation/linking process seems like an overall win no matter what pipeline you're using.

I'm not an autotools guru, but I'll share what I think I know about it if you care to read it for a more informed decision. :)

A detailed look at the problem

Take the following abridged Makefile.am snippet that builds unit tests without mocking.

check_PROGRAMS = test_Something

test_Something_SOURCES  = test_Something.c
test_Something_CPPFLAGS = -I. -I$(top_builddir)/include
test_Something_LDADD    = $(top_builddir)/src/mylib.la

The am extension just tells Automake to turn it into a Makefile with a matching name.

Let's take it line by line.

1. The first line is a list of rules representing executables to build. The check_ prefix means make check, a standardized rule for testing distributions, is dependent upon these rules. The PROGRAMS part means these are compiled executables that have their dependencies automated, which uses the current project compiler configuration.
2. The next few lines start with test_Something, which Automake recognizes from the check_PROGRAMS. These are used to generate Makefile rules. The only mandatory one is the SOURCES part, which as you might imagine tells Automake what to compile for the executable. If something from SOURCES doesn't exist, it will throw a fit and ask that you either add it, or add an inline Makefile segment to build it. Doing so means you need to do your own dependency tracking, which is cumbersome and sort of defeats the purpose of Automake. In this case it informs Automake that test_Something.c is a distributed source file to be compiled with its own automated dependencies.
3. The CPPFLAGS are, as you might have guessed, the C preprocessor flags.
4. The LDADD is a special part that more or less specifies libraries to link against. Here it represents a byproduct of libtool that was built in a previous phase of compilation as part of the shared library creation. Libtool gives you a facility for library interface versioning - a very desirable trait for distributed shared libraries.

We would repeat this for every test.

To integrate CMock into this, we might need to add quite a bit to all of those lines. We need to build cmock.c and unity.c somewhere. Let's say we build it into a libtest.la. Now we need to have the mocks included in some way.

Now, we have a few options. The one I proved out was the surgical-strike, which mocks only what is absolutely necessary for a test.

Let's say I'm testing a function from my library called getClockTime which returns CPU clock time in milliseconds. Let's also suppose that on POSIX compliant systems, we implement it using clock_gettime. We want to ensure it's calculating properly on such systems by mocking clock_gettime, since it needs to convert from second + nanosecond precision to millisecond precision.

check_PROGRAMS = test_getPosixClockTime

test_getPosixClockTime_SOURCES  = mock_clock_gettime.c \
                                  test_getPosixClockTime.c \
                                  test_getPosixClockTime_runner.c
test_getPosixClockTime_CPPFLAGS = -I. -I$(top_builddir)/include
test_getPosixClockTime_LDADD    = $(top_builddir)/src/libtest.la \
                                  $(top_builddir)/src/libtime.la

BUILT_SOURCES = mock_clock_gettime.c test_getPosixClockTime_runner.c

mock_clock_gettime.c: clock_gettime.h
	ruby $(top_srcdir)/CMock/lib/cmock.rb -o$(top_srcdir)/cmock.yml clock_gettime.h

test_getPosixClockTime_runner.c: test_getPosixClockTime.c
	ruby $(top_srcdir)/CMock/scripts/create_runner.rb test_getPosixClockTime.c $@

# More stuff is needed to inform Automake of the built stuff to clean up.

Yeesh, that got out of hand quickly.

Here we have a new Automake concept: BUILT_SOURCES. In the scope of one Automake file, it's often used for sources that aren't distributed, but must be available before all others are compiled. These require manually defined Makefile segments to build them, as seen following that line.

This gets pretty cumbersome when we want to mock more individual functions.

Now let's expand this with a more problematic example. Let's say we have a getUpTime using getClockTime in the library header libtime.h, and we want to mock getClockTime to test it. Doing so could result in link-time problems. The mocked implementations will either conflict with the real ones, or they'll replace all of them wholesale. We can't test the real getUpTime with a mocked getClockTime because they'll both get mocked due to sharing a header file. The same thing would happen if we tried to consolidate mocked implementations into their own library and linked to it for tests. There's currently no way to test that kind of interaction without stubbing out each function into individual header files. We can't mock write and read separately unless we give each of them their own header/source file to be built and linked. Yuck.

Let's say that our generated, mocked implementations had a wrapper prefix to avoid name conflicts, and let's also suppose they are generated and built in a separate Automake file into their own libraries so our Automake test rules don't explode in size. Now the above could become this:

AM_CPPFLAGS = -I. -I$(top_srcdir)/include

check_PROGRAMS = test_getPosixClockTime test_getUpTime

test_getPosixClockTime_SOURCES = test_getPosixClockTime.c test_getPosixClockTime_runner.c
test_getPosixClockTime_LDADD   = $(top_builddir)/src/libtest.la \
                                 $(top_builddir)/src/libtime.la \
                                 $(top_builddir)/src/libc_mock.la
test_getPosixClockTime_LDFLAGS = -Wl,--wrap=clock_gettime

test_getUpTime_SOURCES  = test_getUpTime.c test_getUpTime_runner.c
test_getUpTime_LDADD    = $(top_builddir)/src/libtest.la \
                          $(top_builddir)/src/libtime.la \
                          $(top_builddir)/src/libtime_mock.la
test_getUpTime_LDFLAGS  = -Wl,--wrap=getClockTime

BUILT_SOURCES = test_getPosixClockTime_runner.c test_getUpTime_runner.c

test_getPosixClockTime_runner.c: test_getPosixClockTime.c
	ruby $(top_srcdir)/CMock/scripts/create_runner.rb test_getPosixClockTime.c $@
test_getUpTime_runner.c: test_getUpTime.c
	ruby $(top_srcdir)/CMock/scripts/create_runner.rb test_getUpTime.c $@

# More stuff is needed to inform Automake of the built stuff to clean up.

Here we've replaced the program CPPFLAGS with a special Automake one, AM_CPPFLAGS. This cleans things up a little bit.

Now we have everything mocked in libc_mock.la and libtime_mock.la, but only replacing the real versions that we specify using the --wrap flag.

We still have the same problem with the runners. It's not ideal, but at least those are 1:1 relationships, and mocking doesn't explode in complexity. We've technically just kicked the can down the road with mocks, but consolidating them in larger chunks elsewhere makes building/cleaning/dependency-tracking much simpler. Just imagine if we want to mock other system calls, or other libraries that are depended on, or in this case mock the same library we're testing!

As I mentioned in the issue statement, there are other clever alternatives to --wrap that could still benefit from this process.

This might seem quite bureaucratic, but it automates source distribution in a way that ensures you don't accidentally distribute things you don't mean to, nor leave a mess behind you when building, installing, and uninstalling. All at the cost of developers needing to be more explicit. It's partly a philosophical thing, but it's also for portability reasons. Despite being very GNU, Automake ensures the built Makefile syntax doesn't rely on GNU make extensions, which is why adding "clever" Makefile segments to Automake is frowned upon. Autogen exists to remedy this, but that's somewhat out of scope here.

If you decided to read that explanation, I commend you and hope it helped you understand my struggle a bit better. :)