Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support generation of code coverage data #882

Open
r89m opened this issue Jan 20, 2017 · 12 comments
Open

Support generation of code coverage data #882

r89m opened this issue Jan 20, 2017 · 12 comments
Labels
Milestone

Comments

@r89m
Copy link

r89m commented Jan 20, 2017

Tutorials


Hi Ivan,

I searched for this kind of topic and didn't find anything but let me know if this is a duplicate.

I have my Transition library being tested on Travis using my ArduinoFrameworkFake library which has been working well since you gave me directions on how to import the library.

I'd now like to gather some information regarding coverage so that I can identify areas where I am not testing my code sufficiently. I tried using the build_flags section in platformio.ini to pass --coverage but was met with a lot of linker errors that I tried (unsuccessfully) to resolve.
Here's a few of the combinations of options I tried using
-g -O0 --coverage -fPIC -shared -Wl,-fprofile-generate -Wl,-shared -Wl,-fPIC
-fprofile-arcs -ftest-coverage -fprofile-generate -Wl,-lgcov --coverage

I am successfully generating coverage using: https://github.com/r89m/Transition/blob/master/.travis.yml#L56
which is giving me the output I'd like (https://coveralls.io/jobs/22116632) but I'd love to be able to do something along the lines of pio test --coverage or similar and generate the necessary gcno and gcda files.

Any help would be appreciated.

Richard

@ivankravets ivankravets changed the title [Feature Request] Support generation of code coverage data Support generation of code coverage data Jan 21, 2017
@ivankravets ivankravets added this to the Backlog milestone May 23, 2019
@tanishqaggarwal
Copy link

tanishqaggarwal commented Oct 15, 2019

I know this has probably fallen by the wayside by now, but here's a simple fix I found. Posting this here in case someone, like me, searches the ends of the Earth for how to configure this functionality but can't figure out how:

  • Add --coverage as a build option
  • Add an extra_script with the following content:
Import("env")

env.Append(
  LINKFLAGS=[
      "--coverage"
  ]
)

Works like a charm.

@tanishqaggarwal
Copy link

tanishqaggarwal commented Oct 15, 2019

It's worth noting that I haven't been able to get coveralls to satisfactorily exclude random nonsense files from the coverage report, although this is probably more of a coveralls problem than a PlatformIO one. So far I've been trying:

coveralls -r . -b . -e "./test/output_export.cpp" -e ".pio/libdeps"

This invocation tries to print a test coverage for libdeps within the .pio folder as well. Not sure why.

@tanishqaggarwal
Copy link

@XNinety9
Copy link

I'd love to see that feature added to the next release's roadmap ^^

@arduhe
Copy link

arduhe commented Jun 19, 2020

So do I :-)

@jcw
Copy link

jcw commented Aug 28, 2020

With some web searches, I've been able to generate coverage data and visualise it (using lcov + genhtml), but one puzzle still remains: I'm not just running the app (natively, in Linux), I also want to include coverage data from all the test suites I've written. The data from these tests gets generated fine (in .pio/build/native/test/testname/...), but so far I've found it impossible to use it with lcov/gcov, because the test application binaries get overwritten: the final apps build with pio run and pio test all end up being the same executable, i.e. .pio/build/native/program.

Coverage data is really only meaningful when you can take the code though all its paces and then examine the aggregated reports. I suspect that anyone using pio test will run into the same issue.

In short: how can I generate and combine all coverage data, i.e. several runs of the real app as well as all test apps?

P.S. For reference, here is a trimmed version of my extra_script.py:

# Import("env") etc ...

def generateCoverageInfo(source, target, env):
    for file in os.listdir("verify"):
        call([".pio/build/native/program", "verify/"+file])
    call(["lcov", "-d", ".pio/build/native/", "-c", "-o", "lcov.info"])
    call(["genhtml", "-o", "cov/", "--demangle-cpp", "lcov.info"])

env.AddPostAction(".pio/build/native/program", generateCoverageInfo)

This works fine for the app, without the test suites, when launched as pio run -t debug.

@MBerka
Copy link

MBerka commented Apr 8, 2021

I don't have an answer for @jcw , but that script helped me produce the following that works with an existing pio test -e native test command.
platformio.ini:

[env:native]
platform = native
lib_compat_mode = off
src_filter = +<*> -<*dirs*> -<to/> -<exclude/>
; basically lib and test folders
build_flags =
  -lgcov
  --coverage
extra_scripts = test-coverage.py

test-coverage.py:

import os

Import("env")

def generateCoverageInfo(source, target, env):
    for file in os.listdir("test"):
        os.system(".pio/build/native/program test/"+file)
    os.system("lcov -d .pio/build/native/ -c -o lcov.info")
    os.system("lcov --remove lcov.info '*/tool-unity/*' '*/test/*' '*/MockArduino/*' -o filtered_lcov.info")
    os.system("genhtml -o cov/ --demangle-cpp filtered_lcov.info")

env.AddPostAction(".pio/build/native/program", generateCoverageInfo)

@abanik96
Copy link

I don't have an answer for @jcw , but that script helped me produce the following that works with an existing pio test -e native test command.
platformio.ini:

[env:native]
platform = native
lib_compat_mode = off
src_filter = +<*> -<*dirs*> -<to/> -<exclude/>
; basically lib and test folders
build_flags =
  -lgcov
  --coverage
extra_scripts = test-coverage.py

test-coverage.py:

import os

Import("env")

def generateCoverageInfo(source, target, env):
    for file in os.listdir("test"):
        os.system(".pio/build/native/program test/"+file)
    os.system("lcov -d .pio/build/native/ -c -o lcov.info")
    os.system("lcov --remove lcov.info '*/tool-unity/*' '*/test/*' '*/MockArduino/*' -o filtered_lcov.info")
    os.system("genhtml -o cov/ --demangle-cpp filtered_lcov.info")

env.AddPostAction(".pio/build/native/program", generateCoverageInfo)

Doesn't seems to work, ended up in compilation isse. Can anyone guide me from here??
image

@MBerka
Copy link

MBerka commented Jun 3, 2021

@abanik96 Hard to help without knowing what you actually wrote in the config file. However, it does look like you are trying to run in a different environment instead of the native plaform that I used, and PlatformIO's ESP32 libraries may not include gcov in the same way.

@abanik96
Copy link

abanik96 commented Jun 3, 2021

@abanik96 Hard to help without knowing what you actually wrote in the config file. However, it does look like you are trying to run in a different environment instead of the native plaform that I used, and PlatformIO's ESP32 libraries may not include gcov in the same way.

@MBerka , You are correct did not use native environement, used the environment created by default in platform.ini while creating the project. Can you tell me the significance of this 'native' platform.
Also, how can i include gcov in ESP32 platform?

@MBerka
Copy link

MBerka commented Jun 4, 2021

@abanik96 The native platform allows the C++ to be compiled and run on the same machine where PlatformIO is installed. I have only tried it on Ubuntu, but it probably works similarly on other systems with gcc. This makes the writing and testing of critical logic much faster. I do not know how to run the tests on ESP32 - I am doing the same thing as the creator of this issue, using a library to simulate Arduino framework functions on the dev machine.

RoguedBear added a commit to RoguedBear/esp32-touch-ble-keyboard that referenced this issue Jun 11, 2022
@leon0399
Copy link

Btw I also added proper coverage calculation since @MBerka's solution only calculates coverage for tests themselves. I'm building firmware binaries with coverage flags also, but it required changing memory segments and collecting coverage with lcov --initial (it only requires gcno files)

I wrote an article on it, it is quite complex for single reply: https://blog.leon0399.ru/platformio-coverage-github-actions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

9 participants