Code coverage runtime

[myrrc]

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

• Build/Testing/Packaging Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Сode coverage runtime for functional tests (replaces coverage CI check).
Uses clang SanitizerCoverage callbacks to store data on a per-test basis and dump them after testing.
Introduces critical edges coverage metric.

[myrrc]

Looks like it's the harmful contrib that was causing the bugs, removed it from the coverage builds as we won't need it anyway.

[myrrc]

A word about the IPC between the tester script and the server -- currently we have 3 main options:

1. Utilizing the tester's --testname option which executes a SELECT testname query before running the test.
   The issue here is that we need to interfere with the query process pipeline -- either write a hook or handle the special coverage case with a macro.

2. Setting some option like SET coverage_test_name = name; Same issue with the pipeline.

3. Sending a signal from the tester to the server. This option is "cheaper" in a way, the only issue here is how to attach the test_id to the signal. The server uses the POSIX signal handler with the SA_SIGINFO. The tester preloads libc and executes the sigqueue function.

[myrrc]

Depends on #27228

[myrrc]

Depends on #27361

[myrrc]

I think this PR won't be done in near future, so I'm closing it.

If you operate entirely on the information provided by the binary itself (with or without PC-table), all you can get is addresses that were hit (or some basic blocks indices that were hit). You can symbolize this data and get the source file and line, but the problem is that, you can't really do anything with this information.

Multiple basic blocks can correspond to some lines (e.g. function template instantiations). Basic blocks can correspond to lines not belonging to functions (e.g. macro expansion like settings traits implementation).
If you instrument the code at critical edges level, you lose the access to basic blocks structure, these addresses (of edges) could as well be generated randomly as there's no profit from using them.
You won't be able to get lines ranges that were hit as C++ is hard to parse and you need a compiler.

If you involve the compiler (.gcno), things don't get better. clang does not enumerate its basic blocks, so there's no way to match the basic block that was hit, for example, in the boolean counters array (the PC-table) with a basic block parsed from a gcno file. You get stuck at the function level. Sorting-transforming basic blocks line ranges still does not solve the problem (moreover, .gcno produces strange results like a line belonging to a basic block that's empty). gcno parsers like lcov use perl scripts containing thousands of lines to turn all this information into a human-readable format.

And if one day you want to try the source-based code coverage, you'll also fail. There's no way to tweak what data are stored and tracked (unless you patch the compiler which obviously is not a good idea here), so you just collect everything (and it takes >= 9 hours currently for a general run without per-test data). All you can do is use builtins provided to write the report to some other location, but it won't make everything substantially faster.

You may wonder: why can't you simply collect addresses that were hit and display them? Well, for CH purposes it's a) useless, as one needs to get the coverage percentage, and b) useless, as you reinvent the sancov wheel.

I doubt there's a good way to resolve this PR.

[alexey-milovidov]

You may wonder: why can't you simply collect addresses that were hit and display them?

I'm going to do exactly this in #56102

useless, as one needs to get the coverage percentage

The percentage can be calculated as the percentage of all instrumented edges, the percentage of instrumented symbols in the binary, functions, or source files. The percentage of lines can be calculated roughly if you take the assumption that every basic block spans from the line corresponding to its address to the next instrumented line.