parallelize main loop of gcovr

[whart222]

It takes a long time to run gcovr on a large project, especially when used as part of an automated analysis system. It would be nice if this loop were parallel:

for file in datafiles:
  process_datafile(file,covdata,options)

(from gcovr ticket 3956)

[mrx23dot]

Max number of parallel jobs should depend on numof CPU cores
eg.:
import joblib
print joblib.cpu_count()

others: http://stackoverflow.com/questions/1006289/how-to-find-out-the-number-of-cpus-using-python

[JamesReynolds]

We've hit this issue running tests on a Sparc box so I've created a fork here:

https://github.com/JamesReynolds/gcovr

When I'm satisfied this is working and I've got all the tests working I'll submit a PR. Its necessarily python 2.7 for us as we're rather constrained by our environment.

[latk]

Thank you @JamesReynolds for looking into this. A bit of advice:

• Have you run gcovr under a profiler? Where did you find was most time spent? While calling gcov, or while parsing gcov reports? If it is the parsing code, threads are not going to be helpful because of CPython's Global Interpreter Lock: Python threads don't actually execute in parallel.

• If you find that you need deeper architectural changes, please open an issue so that they can be discussed beforehand. Such changes can then be implemented independently of the parallelization work. If it helps, I can also create a Gitter chat for this repo so that we can discuss questions without having to use GH issues.

  One change that may be necessary is that the covdata is only updated in the main thread, not in any worker threads.

• It's absolutely no problem that you use Python 2.7. But I recommend that you enable Travis and Appveyor for your fork so that you get test reports across multiple Python versions (incl. pypy and 3.x) whenever you push to GH. Simply sign in on their websites with you GH account, and toggle the "on" switch for your repo.

I can understand your desire for a faster gcovr. When I worked with Solaris/Sparc this was so painful that we only ran gcovr on a separate Linux/x86 box.

[JamesReynolds]

Thank you! As far as the changes go:

• I didn't run a full profiler on gcovr but instead timed runs of gcov and they average at 7s for our code. Running them in parallel produced a 3x speed up for 4 threads. I might be a special case though... our sparc box has a lot of threads (64) but each of which is pretty slow.

• I was going to run this for a while to see whether collisions occur with covdata and whether the data is as I expect. At a glance it looks like the updates should be atomic - but that is definitely only a glance.

• Its on my to do list now, good suggestion.

We do our full build with a cross-compiler with only the tests running on Solaris Sparc. It could well be worth us investigating pulling the result back and running gcovr on a Linux/x86 box.

[marco-c]

If the time is actually spent parsing gcov files, to avoid the GIL, you could 1) use multiprocessing; 2) use Cython. Using Cython might require a large set of changes.

I've written something similar to gcovr in Rust (https://github.com/marco-c/grcov). You might also experiment making gcovr use grcov for parsing.

[JamesReynolds]

@latk I've got all of the tests parsing and it is a lot faster - but that is for my use case (~100 *.cpp files, same number (ish) of headers, boost & cmake):

Linux (M-5Y71, Fedora 27)

$ time gcovr -r .. --object-directory . --xml-pretty >/dev/null
real	0m3.492s / user	0m2.816s
$ time gcovr -r .. --object-directory . --xml-pretty -j  >/dev/null
real	0m2.803s / user	0m4.223s

Solaris Sparc (T4-1, Solaris 11.3, Virtualised with 8 vcpus)

$ time gcovr -r .. --object-directory . --xml-pretty -j >/dev/null
real    2m4.542s / user    2m49.113s
$  time gcovr -r .. --object-directory . --xml-pretty >/dev/null
real    8m27.017s / user    1m34.838s

It turns out parallelizing gcov isn't easy - it dumps all its output in the current directory so I've had to create a mutex that locks each directory. I think this limits the usefulness of the code as we're going to be serialized on the locked directory for some build systems. I've added in a temporary directory per thread though, so for some build systems (CMake...) it can output to its temporary directory and run all the gcov's at the same time.

I've only added one threading test, which may not be good enough and I've only my benchmarks to go on - but I could start a PR now?

@marco-c I might take a look. My issue appears to be mainly gcov though (and I'm now happy with the performance) but cython or your project might give me an additional boost.

[marco-c]

It turns out parallelizing gcov isn't easy - it dumps all its output in the current directory so I've had to create a mutex that locks each directory

The easiest solution to this is to run gcov in different directories.
There's also a new gcov intermediate format that you can use to avoid this problem (as it generates one gcov file per gcno file, so it won't overwrite files from other gcov runs), but you would need to change the parsing code.

[marco-c]

@marco-c I might take a look. My issue appears to be mainly gcov though (and I'm now happy with the performance) but cython or your project might give me an additional boost.

grcov executes gcov in a parallel way (or parses the gcda/gcno files directly if you are using LLVM) and then parses its results. I'm pretty sure it's faster because 1) the parsing code is written in Rust; 2) it's parsing the gcov intermediate format, much faster to parse; 3) there's no mutex for the directory; 4) if you're using LLVM, it doesn't even need to execute gcov.

It would be nice to see grcov integrated in gcovr, not sure what would be the best way to do that.

[latk]

@JamesReynolds Those are fantastic performance improvements on the Sparc! Great work :)
Many projects will not notice these improvements (smaller projects, better single-threaded performance), but I'd still like to include these changes for those projects where it does matter.

Please do open a Work-In-Progress PR. As there are some architectural changes that need to be discussed first this will take some time. However:

• Your fork contains some changes that I will not merge (changed links in the Readme, a Dockerfile, …).
• Refactor gcov parsing code, support upcoming GCC 8 changes #228 refactors the gcovr.gcov module significantly. I'll merge that later today.

So it may be better to start a new branch on top of master+228, open the PR, and gradually add changes while they are discussed. I'll assist in any way that I can, but mostly by asking dumb questions :)

[latk]

@marco-c Thank you for building grcov, that looks interesting. Note that gcovr is currently tightly coupled to the gcov human-readable report format, so adding more backends is not immediately possible. If you're interested in doing the necessary work, we would first have to establish some kind of coverage backend interface. The current high-level flow is:

1. gcovr searches for gcno and gcda files
2. for each data file:
   1. guess the directory from which gcov should be invoked
   2. run gcov on the data file
   3. filter the generated files
   4. parse the generated gcov files and update the CoverageData object for that source file (and perform additional processing, e.g. for GCOVR_EXCL_START comments)
3. the CoverageData objects are summarized into some report

I assume only some parts of step 2 would have to change to accomodate grcov? What kind of info would be needed by all backends? Can some phases be generalized? Once you have familiarised yourself with the current code structure, you're welcome to submit a design for a stable coverage backend interface as an issue.

Note that while I am open to making more backends possible, grcov will never become a required or recommended backend; the gcov tool that already ships with the compiler is preferable.

TBH, it would probably be more beneficial to move more of gcov's functionality into gcovr itself, i.e. parsing the raw coverage directly. This should be faster by avoiding launching so many extra processes, and would be more easily parallelizeable. As you already have some experience with parsing raw coverage data, how would one approach this? Do gcc and llvm have a stable format for their raw coverage files?

[JamesReynolds]

@marco-c How do you avoid the requirement for gcov to be run in the same directory as the original compilation? That is what required me to add a directory lock as multiple gcov invocations required the same folder and produced the same files.

Unfortunately, getting rust running on Solaris sparc is (by the looks of things) non-trivial. I do have some other projects though that require tooling on Solaris/AIX so if I hit one that also needs a rust cross-compilation environment I'll revisit.

@latk Thanks! I've got a bunch of stuff on today and tomorrow - so if I rebase from a branch with #228 on Wednesday then submit a PR that should work?

[latk]

@JamesReynolds That would be great! By that time, you can probably just rebase on the master branch. There's no hurry, I'll be grateful whenever you may find some time.

[latk]

Fixed by #239.

[marco-c]

I assume only some parts of step 2 would have to change to accomodate grcov? What kind of info would be needed by all backends? Can some phases be generalized? Once you have familiarised yourself with the current code structure, you're welcome to submit a design for a stable coverage backend interface as an issue.

grcov would replace step 1 as well.
I don't know when I will have time for this, but I'd like to do it at some point as I think it can be beneficial for your users.

Note that while I am open to making more backends possible, grcov will never become a required or recommended backend; the gcov tool that already ships with the compiler is preferable.

grcov uses gcov too for GCC, and uses the LLVM API for LLVM.
I agree it would be nice to have grcov as an option, and not force it. I would like to see it a recommended backend for large projects (where the parsing speed is important), but it's of course up to you.

TBH, it would probably be more beneficial to move more of gcov's functionality into gcovr itself, i.e. parsing the raw coverage directly. This should be faster by avoiding launching so many extra processes, and would be more easily parallelizeable. As you already have some experience with parsing raw coverage data, how would one approach this? Do gcc and llvm have a stable format for their raw coverage files?

I'd advise against this, the GCC format is not stable. The LLVM format has been stable so far (it's an old version of the GCC format), hasn't changed since when it was introduced, but if you intend to use the LLVM API you would need a C extension.
One suggestion would be to use the gcov intermediate format, which is faster to parse. Of course, the parsing speed in Python will never be able to match a Rust implementation.

@marco-c How do you avoid the requirement for gcov to be run in the same directory as the original compilation? That is what required me to add a directory lock as multiple gcov invocations required the same folder and produced the same files.

Are you sure about this requirement? IIRC, as long as you have the gcno and gcda files, you can run gcov anywhere (in fact, for Firefox we run gcov on an entirely different machine than the machine where the build is made).