Add Support for local ("PC") unit tests #519

Closed
neuhalje opened this Issue Feb 15, 2016 · 12 comments

Comments

Projects
None yet
5 participants
@neuhalje

For CI and testing reasons I would really like to run unit tests. For performance reasons these should run on the local machine (PC), and not on the device (in contrast to #408).

I managed to hack together an external_script (see here) that compiles and runs a single cpp file as unit test.

Adding multiple test files does NOT work, bc the compilation of 'program' lumps together every testfile in one executable and the linker fails with duplicate symbols.

If someone with SCons skills can help me, we could extend platformio ci to actually run unit tests.

@neuhalje neuhalje changed the title from Local ("PC") unit tests to Add Support for local ("PC") unit tests Feb 16, 2016

@malachib

This comment has been minimized.

Show comment
Hide comment
@malachib

malachib Mar 23, 2016

Although not exactly the same thing, have you looked into using travis.yml? Works well if your project is on github & open-source

Although not exactly the same thing, have you looked into using travis.yml? Works well if your project is on github & open-source

@ivankravets ivankravets added this to the 3.0.0 milestone Mar 27, 2016

@ivankravets ivankravets added the feature label Mar 27, 2016

@neuhalje

This comment has been minimized.

Show comment
Hide comment
@neuhalje

neuhalje Mar 30, 2016

@malachib you are right. In fact that is one of the use cases I am thinking of. The current platformio ci infrastructure only allows compilation, but no test execution. That last bit is missing.

@malachib you are right. In fact that is one of the use cases I am thinking of. The current platformio ci infrastructure only allows compilation, but no test execution. That last bit is missing.

@malachib

This comment has been minimized.

Show comment
Hide comment
@malachib

malachib Mar 30, 2016

Test execution of an embedded device is a very situation-specific thing. Even if Ivan put in full support for something like simavr, it would fall short of a complete suite. Or are you thinking of just IO-less testing which could all run on say an x86?

Test execution of an embedded device is a very situation-specific thing. Even if Ivan put in full support for something like simavr, it would fall short of a complete suite. Or are you thinking of just IO-less testing which could all run on say an x86?

@ivankravets

This comment has been minimized.

Show comment
Hide comment
@ivankravets

ivankravets Mar 30, 2016

Member

Guys, I hope that I will think something as for this issue.

Member

ivankravets commented Mar 30, 2016

Guys, I hope that I will think something as for this issue.

@malachib

This comment has been minimized.

Show comment
Hide comment
@malachib

malachib Mar 30, 2016

Reviewing this, you seem to be wanting an x86-only test run situation. That seems fully feasible.

I do it with a distinct directory (test, alongside src) and custom makefiles that travis picks up, then use Catch.hpp unit test framework because it's light and easy with C++

I bet the same approach can work with pio+scons, however my scons skills are too nub-ish to make further recommendations

example: https://github.com/malachib/util.embedded

note: there is also a test.pio folder, but that is not automated testing and not relevant to our discussion, so ignore it

Reviewing this, you seem to be wanting an x86-only test run situation. That seems fully feasible.

I do it with a distinct directory (test, alongside src) and custom makefiles that travis picks up, then use Catch.hpp unit test framework because it's light and easy with C++

I bet the same approach can work with pio+scons, however my scons skills are too nub-ish to make further recommendations

example: https://github.com/malachib/util.embedded

note: there is also a test.pio folder, but that is not automated testing and not relevant to our discussion, so ignore it

@neuhalje

This comment has been minimized.

Show comment
Hide comment
@neuhalje

neuhalje Mar 31, 2016

@malachib yes, the goal is to run tests on x86.

A dedicated makefile for tests is a good possibility. For starters I will borrow your Makefile :-)

Having worked with java build tools I came to enjoy automatic (test) dependency resolution.

@malachib yes, the goal is to run tests on x86.

A dedicated makefile for tests is a good possibility. For starters I will borrow your Makefile :-)

Having worked with java build tools I came to enjoy automatic (test) dependency resolution.

@malachib

This comment has been minimized.

Show comment
Hide comment
@malachib

malachib Mar 31, 2016

Glad the makefile can provide some interim help !

Which tools did you use for your Java testing? I did a fair bit of Java back in the day, but have been a C# guy for the last 10 years

Glad the makefile can provide some interim help !

Which tools did you use for your Java testing? I did a fair bit of Java back in the day, but have been a C# guy for the last 10 years

@neuhalje

This comment has been minimized.

Show comment
Hide comment
@neuhalje

neuhalje Apr 1, 2016

@malachib Mainly JUnit and Mockito. At times also Spock.

neuhalje commented Apr 1, 2016

@malachib Mainly JUnit and Mockito. At times also Spock.

@ivankravets ivankravets referenced this issue Jun 13, 2016

Closed

PlatformIO 3.0 Roadmap #423

28 of 37 tasks complete

@ivankravets ivankravets referenced this issue in platformio/platformio-examples Jun 26, 2016

Closed

Error: No such command "test" #1

@ohhorob

This comment has been minimized.

Show comment
Hide comment
@ohhorob

ohhorob Jul 22, 2016

Is there any indication wether the testing configuration can handle using or emulating the same types as the target platform? Or is that just common, and independent of the target??

i.e. some variables I want to be uint8_t, and other uint32_t. Will the unit test compilation be able to ensure correctly sized integers/floats?

ohhorob commented Jul 22, 2016

Is there any indication wether the testing configuration can handle using or emulating the same types as the target platform? Or is that just common, and independent of the target??

i.e. some variables I want to be uint8_t, and other uint32_t. Will the unit test compilation be able to ensure correctly sized integers/floats?

@ivankravets

This comment has been minimized.

Show comment
Hide comment
@ivankravets

ivankravets Jul 22, 2016

Member

@ohhorob No emulation/cross-compilation. All tests will bu run on your machine using OS's GCC compiler.

Member

ivankravets commented Jul 22, 2016

@ohhorob No emulation/cross-compilation. All tests will bu run on your machine using OS's GCC compiler.

@eerimoq

This comment has been minimized.

Show comment
Hide comment
@eerimoq

eerimoq Jul 27, 2016

I can't resist to promote Simba in this thread since it has the host PC test execution functionality already in place. I'm not saying you should use Simba, but instead I'm giving an example of how I work developing for embedded devices.

When I started the development of Simba RTOS I compiled and tested everything on my Linux PC. After all, the big job was to set the interfaces, not to execute the binary on an embedded device. Executing the binary on my Linux PC gave the possibility to use Linux tools, for example gdb, gprof, code coverage, etc. Very useful indeed!

Even today I usually implement and test everything on my Linux PC. When I'm satisfied I cross-compile the code and upload the binary to the embedded deivce.

Of course, certain functionality requires the embedded device, but a surprisingly big part does not.

Here is an example of how to build and run for two boards, linux and arduino_due:

   # Linux test compilation and execution (linux is the default board).
   $ make -s run
   <test soutput from linux stdout/stderr>

   # Arduino Due compilation and execution.
   $ make -s BOARD=arduino_due run
   <upload output>
   <test output from serial port from the embedded device>

The output from the test suite today is rather big, but if you have a PC with lots of RAM you can open it here: https://travis-ci.org/eerimoq/simba

Code coverage (press "src" at the bottom of the page): https://codecov.io/gh/eerimoq/simba

Simba project page on GitHub: https://github.com/eerimoq/simba

eerimoq commented Jul 27, 2016

I can't resist to promote Simba in this thread since it has the host PC test execution functionality already in place. I'm not saying you should use Simba, but instead I'm giving an example of how I work developing for embedded devices.

When I started the development of Simba RTOS I compiled and tested everything on my Linux PC. After all, the big job was to set the interfaces, not to execute the binary on an embedded device. Executing the binary on my Linux PC gave the possibility to use Linux tools, for example gdb, gprof, code coverage, etc. Very useful indeed!

Even today I usually implement and test everything on my Linux PC. When I'm satisfied I cross-compile the code and upload the binary to the embedded deivce.

Of course, certain functionality requires the embedded device, but a surprisingly big part does not.

Here is an example of how to build and run for two boards, linux and arduino_due:

   # Linux test compilation and execution (linux is the default board).
   $ make -s run
   <test soutput from linux stdout/stderr>

   # Arduino Due compilation and execution.
   $ make -s BOARD=arduino_due run
   <upload output>
   <test output from serial port from the embedded device>

The output from the test suite today is rather big, but if you have a PC with lots of RAM you can open it here: https://travis-ci.org/eerimoq/simba

Code coverage (press "src" at the bottom of the page): https://codecov.io/gh/eerimoq/simba

Simba project page on GitHub: https://github.com/eerimoq/simba

ivankravets added a commit that referenced this issue Aug 10, 2016

ivankravets added a commit that referenced this issue Sep 9, 2016

Merge branch 'develop' into feature/unicode-issue-771
* develop:
  Fix incorrect line order when converting from INO to CPP and pointer is used
  Fix unit test
  Notify about `version` field when creating library
  Add support for SparkFun Blynk Board
  Return valid exit code from ``plaformio test`` command
  Disable SSL Server-Name-Indication for Python < 2.7.9
  Version bump to 3.0.1 (issue #772)
  Disable temporary SSL for PlatformIO services // Resolve #772
  Version bump to 3.0.0 (issues #770, #766, #747, #730, #765, #640, #659, #742, #459, #542, #763, #759, #753, #757, #749, #748, #745, #519, #709, #743, #413, #498, #410, #740, #361, #414, #554, #732, #588, #475, #461, #101, #719, #721, #537, #415, #522, #289, #556, #570, #456, #617, #432, #408, #479, #667, #510)
  Fix menu height for  docs
  Fix issue with multiple archives when linking firmware
  Add migration guide for PIO2 to PIO3
  Search libraries by headers/includes with ``platformio lib search --header`` option
  Update pio run command examples
  Add Unit Testing Demo
  Update PIO Plus badge title and link
  Add PlatformIO Plus badge
  Add links to PlatformIO Plus
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment