infrastructure for code coverage?

[DanAlbert]

https://stackoverflow.com/q/48033153/632035

Looks like the stuff I'd made to do this in the platform is close to working for the NDK. Should look in to what we need to do to make this work out of the box.

[DanAlbert]

Adding --coverage to the cflags and ldflags seems to work already (I don't think this has always been the case, but it's true for r19).

Are non-app tools for this useful to people? I suspect the most useful thing for this is Studio integration, but that's something that we'd track as part of Studio rather than the NDK. Would command-line tooling to automate:

1. Push ndk-build built test executables to the device
2. Run your unit tests
3. Pull the coverage data back to the host
4. Build a coverage report

be useful? I suspect this is probably of some use to people, but probably minimal when compared to having this integrated into Studio?

[alexcohn]

@DanAlbert, I kindof repeat myself, but it would be much more efficient if run on the host, with emulation runtime. Also, such scenario is much easier to get integrted into AS.

[DanAlbert]

Sounds like not really any interest in non-Studio tooling then. I'll leave this open but shift it over to unscheduled and we can re-triage it some day if that changes.

[JonForShort]

I have a few projects that only use the ndk itself (not the sdk) so separate tooling would be useful in my situation. Not sure if this too common though. I'd imagine most would want direct integration with studio. Is there a way to create the separate tooling and then maybe have studio interact with those separate tools? That way, developers have both options.

PS: I thought it was kind of funny that the original stackoverflow post in this thread is actually my question :) It look to me a whole year before I found it.

[krysanify]

Adding --coverage to the cflags and ldflags seems to work already (I don't think this has always been the case, but it's true for r19).

Are non-app tools for this useful to people? I suspect the most useful thing for this is Studio integration, but that's something that we'd track as part of Studio rather than the NDK. Would command-line tooling to automate:

1. Push ndk-build built test executables to the device
2. Run your unit tests
3. Pull the coverage data back to the host
4. Build a coverage report

be useful? I suspect this is probably of some use to people, but probably minimal when compared to having this integrated into Studio?

Hi, @DanAlbert

I would appreciate if you can give some feedback. I'm trying to write a gradle task to do just this, below is my attempt so far:

afterEvaluate {
    tasks.externalNativeBuildDebug.doLast {
        def adbPath = android.adbExe.absolutePath
        AndroidDebugBridge.initIfNeeded(false /*clientSupport*/)
        AndroidDebugBridge bridge = AndroidDebugBridge.createBridge(adbPath, false /*forceNewBridge*/)
        if (!bridge.hasInitialDeviceList() || 0 == bridge.devices.length) {
            return
        }

        def buildType = 'debug'
        def deviceAbi = bridge.devices[0].abis.get(0)// 'arm64-v8a'
        exec {
            commandLine adbPath, 'push', "${buildDir}/path/to/resources/file.bin", '/data/local/tmp/'
        }
        exec {
            commandLine adbPath, 'push', "${buildDir}/../.externalNativeBuild/cmake/${buildType}/${deviceAbi}/libgtest.a", '/data/local/tmp/'
        }
        exec {
            commandLine adbPath, 'push', "${buildDir}/path/to/lib/${deviceAbi}/libnative-lib.so", '/data/local/tmp/'
        }
        exec {
            commandLine adbPath, 'push', "${buildDir}/intermediates/cmake/${buildType}/obj/${deviceAbi}/unit_run", '/data/local/tmp/'
        }

        exec {
            commandLine adbPath, 'shell', 'chmod', '775', '/data/local/tmp/unit_run'
        }
        exec {
            commandLine adbPath, 'shell', 'LD_LIBRARY_PATH=/data/local/tmp /data/local/tmp/unit_run'
        }

        exec {
            commandLine adbPath, 'shell', 'rm', '/data/local/tmp/file.bin'
        }
        exec {
            commandLine adbPath, 'shell', 'rm', '/data/local/tmp/libgtest.a'
        }
        exec {
            commandLine adbPath, 'shell', 'rm', '/data/local/tmp/libnative-lib.so'
        }
        exec {
            commandLine adbPath, 'shell', 'rm', '/data/local/tmp/unit_run'
        }
    }
}

The script was able to push and run the unit test on a connected device successfully. However, when I add the --coverage, the profiling failed as it tries to find *.gcda files in host machine under the .externalNativeBuild/cmake/debug/arm64-v8a/CMakeFiles/unit_run.dir/src/test/cpp/ folder. Furthermore, I don't see these files created under /data/local/tmp/ of the connected device.

I guess what confuses me is where does the *.gcda files located if I run the unit test via gradle task? Or did the unit test run wrongly? What did I miss? Once I'm able to locate them, I should be able to pull them and generate the report.

Any feedback is greatly appreciated, and Happy New Year!

Edit: FWIW, I'm using CMake 3.6.4111459 and NDK 18.1.5063045

[DanAlbert]

iirc you need to use GCOV_PREFIX to get the data files to dump to the right place. https://gcc.gnu.org/onlinedocs/gcc-7.2.0/gcc/Cross-profiling.html

[JonForShort]

Hi @DanAlbert , that's correct. That's how I was able to do it in my scripts.

One thing to note @krysanify is that the GCOV_PREFIX is the path that is used where the executable is being run. So if you are running this on an Android device, it needs to be a valid path on the device itself. This is probably obvious but just wanted to mention it just in case :)

[krysanify]

hi @DanAlbert and @thejunkjon,

thanks, guys. you're right, adding the gcov_prefix will generate the gcda files.

[AmitKaushikGIT]

Hi @DanAlbert , I am able to generate gcda files, but I am not able to generate coverage info due to gcov tool version mismatch, even though I am using gcov tool from ndk toolchain.

This fails in ubuntu docker

Processing CMakeFiles/TestSuiteRunner.cpp.gcda
.externalNativeBuild/cmake/debug/x86/CMakeFiles/TestSuiteRunner.cpp.gcno:version '402*', prefer '603*' when using system's gcov tool. (version 6.3)
.externalNativeBuild/cmake/debug/x86/CMakeFiles/TestSuiteRunner.cpp.gcno:version '402*', prefer '409*' when using ndk toolchain's gcov tool.

But locally it works with apple clang's gcov (clang-1001.0.46.4)

[DanAlbert]

I think what that's saying is that Clang generates the data in a format that is too old for that gcov.

@stephenhines what's the state of the art for coverage with Clang? iirc they had their own llvm-cov or something?

[pirama-arumuga-nainar]

Can you try -XClang -coverage-version='409*' compiler flags?

Another alternative is to use the llvm-cov gcov command, like @DanAlbert suggested.