--exclude behavior is inconsistent

[ReenigneArcher]

I spent entirely way too long trying to figure this out and it ended up being a configuration issue on codecov. So posting here for anyone who might find it in a search.

I have the following code, which I run for Windows, Linux, and macOS inside of GitHub runners. For all 3 OSes a bash shell is used.

cd build
python -m gcovr -r .. \
  --exclude ../tests/ \
  --exclude ../third-party/ \
  --json-pretty \
  -o coverage.json

The exclude worked for Windows, but didn't for macOS or Linux.

tests is contained in both my repository root, as well as my build directory.

I've tried many combinations from various posts, with no luck getting consistent behavior. Other combinations I have tried

• (.*/)?tests/
• tests/
• .*tests/.*
• .*/tests/
• $(realpath -- ../tests/) (but this isn't available on macOS... and still didn't work for Linux)

They all work on regex101... https://regex101.com/r/LNtRyx/1

I noticed that I didn't actually see any coverage data on codecov for third-party, perhaps because everything there is a submodule and not actually part of my repo? In any event, this led to me to investigate the issue on codecov's side... and I found that when I actually looked at the json report created by gcovr, that there were no files from tests or third-party... and that codecov was actually added them to their generated report.

To solve you can add ignore paths to codecov's config file. https://docs.codecov.com/docs/ignoring-paths
They support regex as well, but I just used the directory names, like so:

ignore:
  - "tests"
  - "third-party"

Hope this helps someone else.

[Spacetown]

Can you explain a little bit more? The .. stands for two characters and not parent directory.

[ReenigneArcher]

I don't know what else there is to explain.

I found that when I actually looked at the json report created by gcovr, that there were no files from tests or third-party... and that codecov was actually added them to their generated report.

So, the gcovr report was actually correct, but codecov was adding the missing files in anyway.

I know the .. is any two characters in regex. While that still matches, I ended up using the format of .*tests/.* instead.

[Spacetown]

Please can you check the log of the upload?
This sounds like #914 (comment).

[ReenigneArcher]

I do see some gcov steps being performed by codecov... that's interesting.

For some reason on Windows, it's using the gcovr report though... even though it still does something on it's own with gcov. Besides the files being excluded as they were supposed to be on Windows, I also noticed if I use a json report, codecov fails to use it... but switching to xml and it works fine.

I can move any additional findings to that issue though. Thanks!

[Spacetown]

And codecov doesn't finde the provided coverage.xml file. Can it be a output path issue? What is if you remove the instrumentation files before running the upload?

[ReenigneArcher]

It does find it, because I have one project that gcov doesn't run on, because the tests are run by by Macports (the directory is screwy). e.g. https://github.com/LizardByte/Sunshine/actions/runs/8682672118/job/23807558159#step:17:47

But here are logs for a rather simple project.

macOS: https://github.com/LizardByte/tray/actions/runs/8696957670/job/23851261476#step:11:44
Linux: https://github.com/LizardByte/tray/actions/runs/8696957670/job/23851261686#step:11:46
Windows: https://github.com/LizardByte/tray/actions/runs/8696957670/job/23851262112#step:11:45

I think codecov is using what's provided by the coverage.xml, but also combining it with whatever they find.

It looks like they have a disable_search option, that should probably be used when providing your own reports. https://github.com/codecov/codecov-action?tab=readme-ov-file#arguments

[Spacetown]

We also use this option in our own tests, see

gcovr/.github/workflows/test.yml

Lines 198 to 210 in 0574ae5

	- name: Upload coverage to Codecov
	if: ${{ env.USE_COVERAGE == 'true' }}
	uses: codecov/codecov-action@v4
	with:
	env_vars: OS,PYTHON
	fail_ci_if_error: true
	disable_search: true
	plugins: pycoverage
	files: ./coverage.xml
	name: ${{ matrix.os }}-${{ matrix.gcc }}
	verbose: true
	env:
	CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

[ReenigneArcher]

I can confirm that disabling the search indeed changes the results.

I don't know whether the results are accurate or not. I get a lot of partials, on lines that I don't see how they could be partial at all.

One example being:

from (https://app.codecov.io/gh/LizardByte/Sunshine/pull/2430/indirect-changes)

I definitely have a test for when the file is missing. -> https://github.com/LizardByte/Sunshine/blob/ff1341c9edefaa1642fde3f53f783060ec07ae67/tests/unit/test_file_handler.cpp#L18

Maybe I need to use -00 instead of -01? https://github.com/LizardByte/Sunshine/blob/ff1341c9edefaa1642fde3f53f783060ec07ae67/tests/CMakeLists.txt#L22

[Spacetown]

In C++ there are branches generated by the compiler for e.g. exception handling. Please try the options --exclude-unreachable-branches, --exclude-throw-branches and --exclude-unreachable-branches?

[ReenigneArcher]

Thank you for the response. I really appreciate the support!

Unfortunately, that also doesn't seem to provide what I would expect to be covered. The behavior is also different between Windows, Linux, and macOS.

Looking at the read_file function. The log message, is showing partially covered all 3 OSes.

Linux is showing some lines as not covered at all, Windows is showing as fully covered, and macOS is showing as partial.

Windows:
https://app.codecov.io/gh/LizardByte/Sunshine/pull/2335/blob/src/file_handler.cpp?flags%5B0%5D=Windows

Linux:
https://app.codecov.io/gh/LizardByte/Sunshine/pull/2335/blob/src/file_handler.cpp?flags%5B0%5D=Linux

macOS:
https://app.codecov.io/gh/LizardByte/Sunshine/pull/2335/blob/src/file_handler.cpp?flags%5B0%5D=macOS-14&flags%5B1%5D=macOS-12&flags%5B2%5D=macOS-13

Did you mean to list --exclude-unreachable-branches twice, or should there be a third argument?

[Spacetown]

Sorry, the third one is --exclude-noncode-lines. And you should also turn off the optimizations.

[ReenigneArcher]

Hmm... the results are the same. https://github.com/LizardByte/Sunshine/pull/2335/files

Links above are the same and show the same coverage.

[Spacetown]

Can you check on the raw gcov files?

[ReenigneArcher]

Sorry for the delay. I am pretty new to testing in C++, so could you elaborate how I am supposed to check them and what I should be checking for?

[Spacetown]

You can run gcovr with the option --gcov-keep which will keep the files after processing them. Search the file corresponding to your source file and check the data for this lines.

[Spacetown]

Closing because of no response.