Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to build osal with coverage flags so functional test coverage can be reported #936

Open
skliper opened this issue Mar 26, 2021 · 2 comments
Labels
unit-test Tickets related to the OSAL unit testing (functional and/or coverage)

Comments

@skliper
Copy link
Contributor

skliper commented Mar 26, 2021

Is your feature request related to a problem? Please describe.
Ideally functional tests should strive for full coverage, with the exception being impossible paths based on the inability to force return codes or other lower level behaviors that can't be triggered when using the "full stack". Additional "coverage" testing (utilizing stubs) is really only necessary to cover this final % where calls need to be stubbed out to force alternate paths.

Coverage report on functional tests helps for reporting test traceability (coverage tests don't count for API verification), and to easily identify missing functional tests or test cases. It's better to strive for coverage w/ functional tests first (higher value tests, checking for correct full-stack behavior), then fill in the gaps with targeted coverage tests.

Describe the solution you'd like
Add the option to include coverage reporting on OSAL. Currently thinking this could be independent from ENABLE_UNIT_TESTS, since it may be useful to build or not build with these flags for higher level analysis (easily show what parts of OSAL are exercised by cFE tests, etc).

Describe alternatives you've considered
Debated adding a coverage (osal-coverage) target along side the non-coverage target (osal), but who wants twice the compile time? Easy enough to build one or the other depending on need. Could just add the (osal-coverage) target when unit tests are enabled and always use it with the functional tests... debatable. Different targets could avoid accidentally linking a coverage-enabled osal.

Additional context
None

Requester Info
Jacob Hageman - NASA/GSFC

@skliper skliper added the unit-test Tickets related to the OSAL unit testing (functional and/or coverage) label Mar 26, 2021
@skliper skliper added this to the 6.0.0 milestone Mar 26, 2021
@jphickey
Copy link
Contributor

Note that the way coverage instrumentation is implemented by gcc/gcov is very unfriendly to cross compilation environments - it requires runtime access to the build tree to read/update the gcda/gcdo files so it generally breaks if the binaries are copied to another system for execution.

For instance even though we currently support building coverage tests in cross builds, we disable coverage flags.

AFAIK there are other coverage tools that run unmodified binaries (i.e. no extra flags) in a VM of sorts such that they can determine coverage/branch path info this way. This may be a viable alternative to the gcov instrumentation, if we can find a tool that is usable.

Building the normal system with gcov options is certainly possible but should never be a default because of all the quirks/limitations. I would think we can also accomplish gcov instrumentation by putting options in arch_build_custom.cmake and/or making a separate "arch" which is like native but has coverage enabled.

@skliper
Copy link
Contributor Author

skliper commented Mar 26, 2021

https://gcc.gnu.org/onlinedocs/gcc-4.2.4/gcc/Cross_002dprofiling.html#Cross_002dprofiling

Seems possible if the files of interest are moved to the target.

EDIT - I picked 4.2.4 since our target compiler is 4.3, so should be there
EDIT1 - Even if not done on target, the native functional run instrumented for coverage provides a measure of API coverage at least.

skliper added a commit to skliper/osal that referenced this issue Apr 1, 2021
@skliper skliper self-assigned this Apr 2, 2021
@skliper skliper removed this from the 6.0.0 milestone Jul 12, 2021
@skliper skliper removed their assignment May 11, 2022
jphickey pushed a commit to jphickey/osal that referenced this issue Aug 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
unit-test Tickets related to the OSAL unit testing (functional and/or coverage)
Projects
None yet
Development

No branches or pull requests

2 participants