No error details reported via Platform.io

[trullock]

I have a failing test - because I haven't mocked/faked all the Arduino bits fully.

When I run the tests I get a Failed (correctly) reported for the test, but theres no error details. Should there be?

I have to debug each time to find which thing I've not mocked/faked properly.

I'm new to both PlatformIO and ArduinoFake

Any help much appreciated!

platformIO.ini snippet:

[env:native]
platform = native
build_type = debug
lib_deps =
  ArduinoFake
debug_test = mytest
build_flags = -std=gnu++11

[trullock]

Note that I get errors when Verify and other such assertions fail, just not when theres a "missing" mock

[FabioBatSilva]

Hi..

How are you running it ?
platformio cli or idea ?

Can you please paste the output you get ?

[trullock]

Running it via the Test button in VSC, haven't tried the cli directly although the button just invokes the cli in a terminal.

I'll post an example later when I'm at my PC

[trullock]

Repro:

#include <ArduinoFake.h>
#include <unity.h>

using namespace fakeit;

void testa()
{
	Serial.println("Foo");
}

void setUp(void)
{
    ArduinoFakeReset();

    // dont stub Serial.println here
}

int main(int argc, char **argv)
{
    UNITY_BEGIN();
    RUN_TEST(testa);
    UNITY_END();
    return 0;
}

Platformio.ini

[env:native]
platform = native
build_type = debug
lib_deps =
  ArduinoFake
debug_test = testy

Run:

>platformio.exe test --environment native

Output:

Processing testy in native environment
--------------------------------------------------------------------------------------------------------------------------------------------Building...
Testing...
======================================================== [FAILED] Took 3.60 seconds ========================================================
Test          Environment    Status    Duration
------------  -------------  --------  ------------
testy         native         FAILED    00:00:03.605
================================================== 1 failed, 0 succeeded in 00:00:13.036 ================================================== 
The terminal process "platformio.exe 'test', '--environment', 'native'" terminated with exit code: 1.

So its correct that its failed, because Serial.println hasn't been mocked, but the output doesnt tell me so, I have to debug and I end up deep in the internals of ArduinoFake/FakeIt and have to go back up the call stack of find the offending unmocked line.

Would be great if the output showed what was wrong as the debugging cycle is laborious

Thanks!

[trullock]

Actaully, I think its simply because an exception is being thrown and PlatformIO/Unity aren't outputting anything useful.
I guess this isnt an ArduinoFake problem then, so feel free to close.
But, it would be good to solve it I think as newcomers to Arduino and PIO end up here and presumably have this same problem 🤷

https://community.platformio.org/t/getting-output-of-failed-test/10940

[matthewturner]

I've hit a similar issue and it looks like PlatformIO now outputs the error code from the failing test. In my case, when I forget to mock an Arduino method I get a seg fault:

Program errored with 3221225477 code

[r89m]

Running your tests with -v or --verbose, ie platformio test -e native -v gives you a tiny bit more information, for example:

Example failing test
---------------------------------------------------------------------
test/test-example.cpp:43
.....................................................................

test/test-example.cpp:43: FAILED:
due to unexpected exception with message:
  Unknown exception

This at least gives you some information in that the test that failed was declared on line 43 of test-example.cpp

This still wasn't enough for me, so I patched ArduinoFake.h and FunctionFake.h and replaced the line

#include "fakeit/fakeit.hpp"

with

#ifndef ARDUINOFAKE_CUSTOM_FAKEIT
#include "fakeit/fakeit.hpp"
#else
#include "custom-fakeit.hpp"
#endif // ARDUINOFAKE_CUSTOM_FAKEIT

I edited platform.io to look like the following:

[env:native]
platform=native
lib_extra_dirs=
    ./test_includes
build_flags=
    -DARDUINOFAKE_CUSTOM_FAKEIT
    -Wno-deprecated		; Hide errors related to fakeit
lib_deps=
    fabiobatsilva/ArduinoFake@^0.3.1

I then created a test_includes folder next to my test folder in the project root.
Inside test_includes I created a folder called catch and downloaded catch_amalgamated.hpp and catch_amalgamated.cpp from the latest release of Catch2 - the unit testing framework I use.

Again, inside test_includes I created a folder called fakeit and downloaded the latest Catch-compatible version of FakeIt - saving it as custom-fakeit.hpp

This then gives a little more output on error:

Unknown file:0: FAILED:
explicitly with message:
  Unexpected method invocation: unknown()
    An unmocked method was invoked. All used virtual methods must be stubbed!

Annoyingly FakeIt cannot provide you with the name of the method that you tried to call as, if it hasn't been mocked yet, it cannot possibly know its name due to the compiled nature of the code and C++'s lack of support for reflection.

Helpfully, if it is a method that you have mocked, but perhaps using Return instead of AlwaysReturn or with parameters that don't match, you can get a bit more information, for example:

Unknown file:0: FAILED:
explicitly with message:
  Unexpected method invocation: ArduinoFake().digitalWrite(
  , )
    Could not find any recorded behavior to support this method call.

It would be nice if all of the Arduino methods in FunctionFake could be wrapped by a When(Method(ArduinoFake(), method)).Throw("Unmocked method 'method'") call but I'm not sure how feasible that would be

[r89m]

Following on from the above I edited ArduinoFake.h and replaced _ArduinoFakeInstanceGetter1 with the below:

#define _ArduinoFakeInstanceGetter1(mock) \
    mock##Fake* mock() \
    { \
        if (!this->Instances->mock){ \
            this->Instances->mock = &this->Mocks->mock.get(); \
            mock##Fake::prepare(this->Mocks->mock); \
        } \
        return this->Instances->mock; \
    }

This adds a call to each Interface that we're mocking to prepare the mock with known methods - this allows FakeIt to known the names of any methods that haven't been called. This requires a prepare method to be added to each interface (FunctionFake, SerialFake, WireFake, StreamFake, ClientFake, PrintFake), for example for FunctionFake I added the following:

static void prepare(Mock<FunctionFake> &functionFakeMock) {
        Method(functionFakeMock, init);
        Method(functionFakeMock, loop);
        Method(functionFakeMock, setup);

        Method(functionFakeMock, pinMode);
        Method(functionFakeMock, digitalWrite);
        Method(functionFakeMock, digitalRead);

        Method(functionFakeMock, analogRead);
        Method(functionFakeMock, analogReference);
        Method(functionFakeMock, analogWrite);
        
        Method(functionFakeMock, millis);
        Method(functionFakeMock, micros);

        Method(functionFakeMock, delay);
        Method(functionFakeMock, delayMicroseconds);

        Method(functionFakeMock, pulseIn);
        Method(functionFakeMock, pulseInLong);

        Method(functionFakeMock, shiftOut);
        Method(functionFakeMock, shiftIn);

        Method(functionFakeMock, detachInterrupt);
        Method(functionFakeMock, attachInterrupt);
        Method(functionFakeMock, cli);
        Method(functionFakeMock, sei);

        Method(functionFakeMock, tone);
        Method(functionFakeMock, noTone);
        
        OverloadedMethod(functionFakeMock, random, long(long));
        OverloadedMethod(functionFakeMock, random, long(long, long));
        Method(functionFakeMock, randomSeed);

        Method(functionFakeMock, map);

        Method(functionFakeMock, yield);
    }

This then gives you quite a useful message without having to do any mocking in the tests themselves, for example:

Unknown file:0: FAILED:
explicitly with message:
  Unexpected method invocation: functionFakeMock.digitalWrite(
  , )
    Could not find any recorded behavior to support this method call.

For some methods you actually get a really nice message with the exact parameters passed, delay for example:

Unknown file:0: FAILED:
explicitly with message:
  Unexpected method invocation: functionFakeMock.delay(40)
    Could not find any recorded behavior to support this method call.

If anyone is better at macros than me I'm sure this could be significantly simplified

[FabioBatSilva]

@r89m, Thanks for looking into it.. that looks promising.

The "fakeit/fakeit.hpp" file is just a directly copy of the the standalone version of https://github.com/eranpeer/FakeIt. We should be able to upgrade to the latest version..

[r89m]

Upgrading to the latest fakeit would be a good idea but I'm not certain it would fix the issue with the warning messages...
There are several versions of fakeit for different test frameworks: https://github.com/eranpeer/FakeIt/tree/master/single_header

I happen to use Catch so I download the version in the catch directory and saved it as custom-fakeit.hpp but I don't know if it is popular enough to make the default in your library?