Question: Mocking

[eerimoq]

Hello!

Are there any plans on adding support for mocking?

Thanks!

[vberlier]

Mocking can be quite tricky in C, and depending on the situation, you might want to be able to choose between different approaches. Mocking also inevitably ends up involving the build system, which is something I tried to avoid with the test framework itself.

I mean I've been thinking about it but I'm not sure what the API should even look like. So far I haven't been able to come up with something I'd be willing to bake into the framework.

If you have ideas let me know, mocking is definitely a lacking feature so if you have something in mind... :)

[eerimoq]

This is how I implemented mocking in a project of mine:

The function under test.

library.c

void xmount(const char *source_p,
            const char *target_p,
            const char *type_p)
{
    int res;

    res = mount(source_p, target_p, type_p, 0, "");

    if (res != 0) {
        perror("error: mount: ");
        exit(1);
    }
}

The test suite.

mock_push_mount() and __wrap_mount() were generated by a Python script in my project based on the mocked header file. The script did a good job, but some manual adaptions were often needed. Though, once the mock functions had been implemented once, they could be used in all test suites. Basically, for every module in the system, a mock module was implemented as well.

There are certainly cases where this approach is not suitable, but it's pretty flexible. What is needed by the mock framework is just functions to push, pop and assert named data in queues. The stub implementation can be manual, and still useful in my opinion.

There are of course other mocking frameworks available that might be better, but unfortunately I have little to no experience from them.

main.c

#include "mock.h"

TEST(xmount_ok)
{
    /* Prepare expected input and output for the next call to
       mount(). */
    mock_push_mount("a", "b", "c", 0, "", 1, 0);

    /* The function to be tested will call __wrap_mount() instead of
       mount(). */
    xmount("a", "b", "c");
}

mock.h

void mock_push_mount(const char *source_p,
                     const char *target_p,
                     const char *type_p,
                     unsigned long flags,
                     const void *data_p,
                     size_t data_size,
                     int res);

mock.c

void mock_push_mount(const char *source_p,
                     const char *target_p,
                     const char *type_p,
                     unsigned long flags,
                     const void *data_p,
                     size_t data_size,
                     int res)
{
    mock_push("mount(source_p)", source_p, strlen(source_p) + 1);
    mock_push("mount(target_p)", target_p, strlen(target_p) + 1);
    mock_push("mount(type_p)", type_p, strlen(type_p) + 1);
    mock_push("mount(flags)", &flags, sizeof(flags));
    mock_push("mount(data_p)", data_p, data_size);
    mock_push("mount(): return (res)", &res, sizeof(res));
}

/* Called instead of mount() if (-Wl,)--wrap=mount is given to the
   linker. */
int __wrap_mount(const char *source_p,
                 const char *target_p,
                 const char *type_p,
                 unsigned long flags,
                 const void *data_p)
{
    int res;

    mock_pop_assert("mount(source_p)", source_p);
    mock_pop_assert("mount(target_p)", target_p);
    mock_pop_assert("mount(type_p)", type_p);
    mock_pop_assert("mount(flags)", &flags);
    mock_pop_assert("mount(data_p)", data_p);
    mock_pop("mount(): return (res)", &res, sizeof(res));

    return (res);
}

[vberlier]

Thanks a bunch! I've been thinking about all of this, did some research and I think I have an idea. I need to experiment with it first but it might lead to a pretty clean mocking workflow. I'll try to keep the thread updated.

[eerimoq]

Maybe make the mocked functions save input and return reasonable output by default, with the possibility to override the default behavior if required to test a specific case.

[vberlier]

The mocked functions will record the parameters they get called with. I don't know what kind of default output would be considered reasonable though, you mean every mocked function would return 0 unless you override it or something like that?

[eerimoq]

I'm not sure how it would be implemented, all I know is that the user want to write as little code as possible. A default behavior that works for most cases could reduce the test code size.

[vberlier]

There's quite a lot of things involved with mocking so yeah I want to keep the API as small as possible. It's just that there's no robust way of taking a function and determining a "reasonable" default return value. The mock would have to rely on heuristics to differentiate between returning an int as an error code or as an actual number for example. And what about returning structs? Or pointers? I obviously don't want the user to drown in boilerplate but I'm not sure if I really want to go down the rabbit hole. I'll see what I can do.

[vberlier]

Hey just wanted to post an update on this. It's a bit different from your approach but I think I settled on an API and this is how far I've gone with my prototype:

int dummy_mount_implementation(const char *arg1,
                               const char *arg2,
                               const char *arg3,
                               unsigned long int arg4,
                               const void *arg5)
{
    printf("test\n");
    return 0;
}

TEST(xmount_ok)
{
    MOCK(mount)->mock_return(0);

    // __wrap_mount doesn't call mount and just returns 0
    xmount("a", "b", "c");
    xmount("a", "b", "c");
    xmount("a", "b", "c");

    MOCK(mount)->mock_implementation(dummy_mount_implementation);

    // __wrap_mount calls dummy_mount_implementation
    xmount("a", "b", "c");

    MOCK(mount)->disable_mock();

    // __wrap_mount calls the original mount function
    xmount("a", "b", "c");
}

I think it's pretty flexible and I'm already working on implementing a bunch of other features:

TEST(xmount_ok)
{
    MOCK(mount)
        ->mock_return_once(0)
        ->mock_implementation_once(dummy_mount_implementation)
        ->mock_return_once(0);

    xmount("a1", "b", "c"); // return 0
    xmount("a2", "b", "c"); // call dummy_mount_implementation
    xmount("a3", "b", "c"); // return 0
    xmount("a4", "b", "c"); // call original mount function

    ASSERT_EQ(MOCK(mount)->calls[0].arg3, "c");
    ASSERT_EQ(MOCK(mount)->calls[0].return_value, 0);
    ASSERT_EQ(MOCK(mount)->calls[1].arg1, "a2");
    ASSERT_EQ(MOCK(mount)->last_call.arg1, "a4");
}

I'll try to push something usable on a separate branch soon.

[eerimoq]

Looks interesting!

A question, what's the reason for asserting input arguments after the tested function has been called, instead of preparing the mock with expected input before the calls? I prefer tests to fail early as secondary failures are of little to no interest.

[vberlier]

Hmm I've worked with mocking libraries in other languages and most of the time they're not responsible for making assertions themselves. The mocking utilities are just here to swap implementations and collect information about invocations, and it's always up to you to decide what you want to do with it afterwards. I'm just replicating what I'm the most familiar with.

I think it's overall more flexible though, since you can check arguments individually and only on the calls you're interested in, but i agree that it would be useful to be able to make assertions about the expected input beforehand.

[eerimoq]

You can add expected values of individual arguments on beforehand as well. No technical problem there.

[vberlier]

Yeah it should be completely possible to add it on top of everything else. It just means that mocks will be much more intertwined with the assertion library, which I guess is fine.

[eerimoq]

I like your mocking strategy more and more and want to use it in a project of mine. What's the status of the mocking branch? Do you have any plans on merging it to master and release it?

[vberlier]

Oh thanks! I just got back from vacation so there wasn't much activity recently, but I turned the mocking utility into its own project https://github.com/vberlier/narmock. The mocking branch is where I experimented with the API and different strategies for making it easy to set up.

Other than that the functions I implemented are pretty much ready to use, I just need to work on the documentation. I'm planning to wrap things up in the next few days and make a release before implementing more mocking functions.

[eerimoq]

Sounds great! Thank you very much!

[vberlier]

Alright so I changed a few final things and made a release. I'm planning to add more functions to the mock API but everything else should be relatively stable.

The Narwhal README now also mentions mocking and links to Narmock 2969fc4

Thank you for your help! I'm glad to finally be able to close this issue! All further mocking-related things are now happening at https://github.com/vberlier/narmock