search_paths seems to overwrite --object_directory

[bdevorem]

Prepare for major information dump! If this question has already been asked, please let me know. I tried looking for anything related to this, with no luck, but definitely point me in the right direction if I missed it!

For my project, my build directory is separate from my source directory, and I also need the .gcda files in a separate directory. The reason for this is my source and build directories are located on a shared filesystem, and I have workers that will each test the executable and then run gcovr. If the .gcda, .gcov, and output files aren't in different locations for each worker, then I'll have a race condition because the workers run in parallel.

Here is an example file structure. I'm working with a larger project, so there are actually more subdirectories, but I think this gets the point across:

├── target
│   └── build
│       └── build_flavor
│           ├── subdir 1
│           │   └── .gcda files
│           ├── subdir 2
│           │   └── .gcda files
│           └── subdir 3
│               └── .gcda files
├── src
│   ├── makefile
│   ├── README.md
│   ├── subdir 1
│   │   ├── .c files
│   │   └── .h files
│   ├── subdir 2
│   │   ├── .c files
│   │   └── .h files
│   └── subdir 3
│       ├── .c files
│       └── .h files
└── build
    └── build_flavor
        ├── subdir 1
        │   ├── .o files
        │   ├── .gcno files
        │   ├── .dso files
        │   └── .d files
        ├── subdir 2
        │   ├── .o files
        │   ├── .gcno files
        │   ├── .dso files
        │   └── .d files
        └── subdir 3
        │   ├── .o files
        │   ├── .gcno files
        │   ├── .dso files
        │   └── .d files

Basically, src/ has my source code (shocker!). When I build the source, the object files go into build/, with the file structure retained. Additionally during compilation, I have two environment variables set, GCOV_PREFIX and GCOV_PREFIX_STRIP, such that the .gcda files for each source file are placed in target/, with the file structure retained. target/ consists of a subdirectory that is the exact same as build/, except that it only contains .gcda files (and not object files).

Now, where the problem exists, I'm running the following gcovr command:

gcovr -v -r /absolute/path/to/src --html --html-details -o test-details.html --object-directory=/absolute/path/to/build /absolute/path/to/target/build

I expected this to check /absolute/path/to/src for my source code, /absolute/path/to/build for my object files, and /absolute/path/to/target/build for the .gcda files (as gcovr documentation says search_paths: Search these directories for coverage files). Looking at the output from this command (since I ran with -v), I'm seeing output and multiple gcov commands of the following form:

Processing file: /absolute/path/to/target/build/build_flavor/subdir1/filename.gcda
Running gcov: 'gcov /absolute/path/to/target/build/build_flavor/subdir1/filename.gcda --branch-counts --branch-probabilities --preserve-paths --object-directory /absolute/path/to/target/build/build_flavor/subdir1' in '/tmp/tmphbIP3v'

The output for that gcov command is: /absolute/path/to/target/build/build_flavor/subdir1/filename.gcno:cannot open notes file. That's because the .gcno files are under the top-level build/ and not target/build/.
It seems like the search_paths path overwrote the --object-directory path from my original gcovr command. I expected a gcov output like the following (only difference is in the --object-directory path):

Processing file: /absolute/path/to/target/build/build_flavor/subdir1/filename.gcda
Running gcov: 'gcov /absolute/path/to/target/build/build_flavor/subdir1/filename.gcda --branch-counts --branch-probabilities --preserve-paths --object-directory /absolute/path/to/build/build_flavor/subdir1' in '/tmp/tmphbIP3v'

And when I run this command, the output is as I expect, with the proper coverage percentages as well.

Is this expected behavior?

[latk]

That's a detailed question! That behaviour of gcovr is kind of expected. I'm not sure whether it can be fixed without breaking lots of existing workflows.

What is gcovr doing?

(I'm writing this up partially for my own benefit, as the behaviour is mildly WTF.)

The gcovr --object-directory option does not directly affect the gcov option of the same name but has related behaviour: it influences where the program looks for coverage data.

Gcovr looks in any search paths that you specify as positional arguments. If none are given, gcovr falls back to the root path and any --object-dir. If you specify both search paths and --object-dir, the --object-dir path will be ignored. (It could make sense to change this part, I'm not sure).

Gcovr then searches for all .gcda and .gcno files under the selected search paths (but discards .gcno if there's a corresponding .gcda).

Next, gcovr tries to guess from which directory gcov should be invoked to process that coverage data. There are a couple of organically grown heuristics, and they are all bad. I don't really understand them.

• By default gcovr tries all parent directories of the gcda file which should be somewhere in your build directory, and tries the --root directory.

• However, if --object-directory is specified the normal heuristics are overridden. I really don't understand that code.

It has been suggested that these heuristics could be removed entirely, but I don't understand gcov sufficiently well.

Finally gcovr tries to run gcov in all the directories suggested by the heuristics and sees what sticks. This is …slow. Gcov is given the base directory of the .gcda file as the --object-directory arguments, since the .gcno file should be right next to it.

The takeaway is that there are few or no reasons to continue using the --object-directory option. Historically, the --object-directory option is older (and the search paths weren't documented for a long time).

At no point are the actual .o object files of interest, only the .gcno files that are generated during compilation.

Possible workarounds

In your case the .gcda and .gcno files are not next to each other. The simplest way to fix this is to copy the .gcno files into your coverage target directory structure.

The less simple way is to give me (a series of) pull requests that:

• (optional) figure out what --object-directory is actually doing and remove/document that code as appropriate.
• (optional, and after more discussion) deprecate the --object-directory option.
• add a separate option that is passed through to gcov's --object-directory option. The name --gcov-option-directory might be appropriate.

If you (or anyone else) would like to tackle this, please see the contribution guide for general advice and how to set up a development environment. For this kind of option it would also be really important to include a test case; a simplified version of this scenario with only one source file would do. That test case would be great anyway because it has related problems as cross-profiling, compare #259 (comment).

[bdevorem]

Thanks for such a detailed response!

I'd be interested in pursuing the --gcov-option-directory option in the long-haul. I may not get around to it soon though, due to other priorities.

For now, I'll copy the gcno files to the location of the gcda files. Here's a semi-related question- does gcovr care about the relative paths between the source dir and the build dir? I.e. is anything dependent on their locations relative to one another?

[latk]

I'll be glad for a pull request if and when you get around to it! Could also be a small project for a student assistant?

Unfortunately, yes, the relative path between the .gcda/.gcno files and the source files can matter.

• gcov needs to find the source file in order to write the intermediate .gcov file. All those working directory heuristics are intended to make that work. The result should be that gcov is invoked in the same directory as the compiler (so that relative paths to the source file that are stored in the .gcno can be resolved).

  You may be able to prevent any problems if you change the gcov command that gcovr uses, so that you explicitly pass the gcov --source-prefix option. E.g.:

  gcovr --gcov-executable 'gcov --source-prefix /absolute/path/to/src' $normal_options

• later, gcovr also needs to find the absolute source file path in order to apply filters. This should typically work fine, assuming you have set the --root option correctly. Again there are some heuristics involved that might break.

[latk]

I did some further digging during the weekend:

1. Gcovr's --object-directory heuristics are partially dead code. In most cases it will suggest the following gcov working directories:

   • if the objdir is absolute: {objdir}
   • if the objdir is relative: {dirname(path_to_gcda_file)}/{objdir} and {cwd}/{objdir}, if any of those exist
   • if the objdir is relative but none of those exists, the default heuristics are used instead.

2. Gcov's --object-directory option names a directory which immediately contains the .gcno file. It is a prefix, not a search path: Given the gcov --object-directory /build/flavor and a data file /profile/build/flavor/subdir/foo.gcda, gcov would look for /build/flavor/foo.gcno, i.e. would disregard the subdir.

   Therefore, simply adding a --gcov-object-directory option that is passed through will not work except for the most simple builds. Instead, a more sophisticated approach that translates the path (in reverse of the GCOV_PREFIX variable) would be necessary. I.e. of any found gcda file, strip the prefix and add the prefix of the original build directory to find the gcno file.

   But this still will not work because gcov necessarily requires the gcda and gcno file to be right next to each other. This is not fixable by gcovr unless we first copy the corresponding files into a tempdir.

3. Gcov's --source-prefix option doesn't work like I would have hoped. It does not make it possible to invoke gcov from arbitrary directories. The gcov working directory MUST be the directory from which the compiler was invoked, in your case this is the original build directory. Running gcovr and by extension gcov is closely linked to the compilation process and will necessarily create (temporary) files in that directory.

   It is therefore not possible to support your shared-compilation, individual-coverage approach. AFAIK no gcov-based tool (e.g. lcov) can work under those constraints.

Your options seem to be:

• Stop doing this kind of shared compilation and start compiling on those systems where you want to process coverage. If you are doing this for performance reasons you still might be able to do distributed compilation or use something like ccache. It may also be sufficient to copy the build & source directories to a different location.
• Mount the shared directories so that the relevant paths between the object files to the source and the coverage data to the source is the same. I.e. your target and build directories should be immediate siblings and have the same structure. You will still need to copy all gcno file from the build dir to the target dir.
• Use a file-system level approach such as overlay file systems, so that gcov can write files into the shared directory structure without modifying the shared file system.
• Use a non-gcov based coverage tool such as kcov.

I am now closing this issue because any of those approaches is out of scope for gcovr.