Unit testing impossible for frameworks including Unity themselves

[maxgerhardt]

What kind of issue is this?

• [X ] PlatformIO Core.
  If you’ve found a bug, please provide an information below.

You can erase any parts of this template not applicable to your Issue.

---

Configuration

Operating system: Windows 10 x64

PlatformIO Version (platformio --version): version 5.2.0a6

Description of problem

The PlatformIO "test" procedure is such that PlatformIO adds its own Unity library to the build.

platformio-core/platformio/builder/tools/piomisc.py

Lines 353 to 361 in 834c7b0

	def ConfigureTestTarget(env):
	env.Append(
	CPPDEFINES=["UNIT_TEST", "UNITY_INCLUDE_CONFIG_H"],
	CPPPATH=[os.path.join("$BUILD_DIR", "UnityTestLib")],
	)
	unitylib = env.BuildLibrary(
	os.path.join("$BUILD_DIR", "UnityTestLib"), get_core_package_dir("tool-unity")
	)
	env.Prepend(LIBS=[unitylib])

this however directly conflicts with any framework which already has Unity compiled into it. Such is the case with the https://github.com/arduino/ArduinoCore-mbed in which the precompiled mbed-os library (libmbed.a) has the Unity object files.

Thus it is impossible to e.g. run unit tests with a board = nano33ble framework = arduino configuration without modifications.

Refer to the community topic here

Steps to Reproduce

1. Create a new Arduino Nano 33 BLE project
2. Add test\main.cpp from below
3. Run the "Test" project task

Actual Results

A bazillion linker errors due a conflict with .pio\build\nano33ble\libUnityTestLib.a and .platformio\packages\framework-arduino-mbed\variants\ARDUINO_NANO33BLE\libs\libmbed.a

c:/users/max/.platformio/packages/toolchain-gccarmnoneeabi@1.80201.190214/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/bin/ld.exe: C:\Users\Max\.platformio\packages\framework-arduino-mbed\variants\ARDUINO_NANO33BLE\libs\libmbed.a(unity.o): in function `UnityBegin':
unity.c:(.text.UnityBegin+0x0): multiple definition of `UnityBegin'; .pio\build\nano33ble\libUnityTestLib.a(unity.c.o):unity.c:(.text.UnityBegin+0x0): first defined here
c:/users/max/.platformio/packages/toolchain-gccarmnoneeabi@1.80201.190214/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/bin/ld.exe: C:\Users\Max\.platformio\packages\framework-arduino-mbed\variants\ARDUINO_NANO33BLE\libs\libmbed.a(unity.o): in function `UnityEnd':
unity.c:(.text.UnityEnd+0x0): multiple definition of `UnityEnd'; .pio\build\nano33ble\libUnityTestLib.a(unity.c.o):unity.c:(.text.UnityEnd+0x0): first defined here
c:/users/max/.platformio/packages/toolchain-gccarmnoneeabi@1.80201.190214/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/bin/ld.exe: C:\Users\Max\.platformio\packages\framework-arduino-mbed\variants\ARDUINO_NANO33BLE\libs\libmbed.a(unity.o):(.rodata.UnityStrErr64+0x0): multiple definition of `UnityStrErr64'; .pio\build\nano33ble\libUnityTestLib.a(unity.c.o):(.rodata.UnityStrErr64+0x0): first defined here
c:/users/max/.platformio/packages/toolchain-gccarmnoneeabi@1.80201.190214/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/bin/ld.exe: C:\Users\Max\.platformio\packages\framework-arduino-mbed\variants\ARDUINO_NANO33BLE\libs\libmbed.a(unity.o):(.rodata.UnityStrErrDouble+0x0): multiple definition of `UnityStrErrDouble'; .pio\build\nano33ble\libUnityTestLib.a(unity.c.o):(.rodata.UnityStrErrDouble+0x0): first defined here
c:/users/max/.platformio/packages/toolchain-gccarmnoneeabi@1.80201.190214/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/bin/ld.exe: C:\Users\Max\.platformio\packages\framework-arduino-mbed\variants\ARDUINO_NANO33BLE\libs\libmbed.a(unity.o):(.rodata.UnityStrErrFloat+0x0): multiple definition of `UnityStrErrFloat'; .pio\build\nano33ble\libUnityTestLib.a(unity.c.o):(.rodata.UnityStrErrFloat+0x0): first defined here
c:/users/max/.platformio/packages/toolchain-gccarmnoneeabi@1.80201.190214/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/bin/ld.exe: warning: size of symbol `Unity' changed from 132 in .pio\build\nano33ble\libUnityTestLib.a(unity.c.o) to 160 in C:\Users\Max\.platformio\packages\framework-arduino-mbed\variants\ARDUINO_NANO33BLE\libs\libmbed.a(unity.o)
collect2.exe: error: ld returned 1 exit status
*** [.pio\build\nano33ble\firmware.elf] Error 1

Expected Results

Running unit tests is possible

If problems with PlatformIO Build System:

The content of platformio.ini:

[env:nano33ble]
platform = nordicnrf52
board = nano33ble
framework = arduino

Source file to reproduce issue:
test\main.cpp

#include <Arduino.h>
#include <unity.h>

void simple_test(void) {
    TEST_ASSERT_EQUAL(33, 33);
}

void setup() {
    delay(2000);

    UNITY_BEGIN();
    RUN_TEST(simple_test);
    UNITY_END();
}

void loop() {
    delay(1000);
}

Additional info

[valeros]

Hi @maxgerhardt ! Thanks for reporting. Indeed, the reasons for this behavior are obvious, but I'm not sure how we can solve this problem in a reasonable way. We cannot simply omit our .pio\build\nano33ble\libUnityTestLib.a in favor of the Unity located in mbed.a. We need a complete control over the Unity configuration, this is the main reason why PlatformIO also adds several additional files that implement standard routines (like redirecting test output, etc.) when compiling the Unity sources. On the other hand, we a bit reluctant to dynamically manipulate the libmbed.a archive and remove all references of the Unity code. I believe the best way out here is to ask the maintainers of ArduinoCore-mbed to exclude Unity from their framework package.

[maxgerhardt]

We need a complete control over the Unity configuration, this is the main reason why PlatformIO also adds several additional files that implement standard routines (like redirecting test output, etc.)

There might be special cases where e.g the pre-compiled Unity version is "functionally correct" and would not differ from what PlatformIO would use. Or when there are frameworks which won't remove their internal Unity version. It would make sense to add a fallback option to the platformio.ini to disable using PlatformIO's Unity version completely, like test_use_builtin_unity = no -- the user has to of course understand the implications of this then.

In the Arduino-mbed case (and probably for framework = mbed too?), no adverse consequences were observed by the user when removing the Unity object files from libmbed.a during normal firmware compilation, since the normal firmware does not use Unity, only the unit-testing firmware does. So again, an option to really make PlatformIO remove known Unity object files from the build (while backing up the old file) via an explicit "please do so" configuration option may come in handy if frameworks with precompiled blobs won't remove Unity.

The ArduinoCore-mbed people will surely see a problem with removing Unity since then no Arduino IDE user can use Unity (if they did). They might resolve that though be refactoring Unity into a library and compile it from source. That would be a good alternative.