Skip to content

coverage: scripts/coverage.sh profiles three test binaries out of nine, so include/morph/net is scored by nothing at all #403

Description

@Yaraslaut

What is wrong

scripts/coverage.sh names the binaries it profiles by hand. It names three
families:

TEST_EXE="$OUT/tests/morph_tests"                              # positional
LADDER_TEST_EXE="$OUT/examples/common/ladder_common_tests"     # -object
$OUT/examples/${_rung}/ladder_${_rung}_tests                   # -object, per rung

The tree builds nine test executables:

target profiled by coverage.sh?
morph_tests yes
ladder_common_tests, ladder_<rung>_tests yes
morph_net_tests no
morph_offline_sqlite_tests no
morph_qt_tests no
morph_net_qt_interop_tests no
morph_soak no
morph_bench no (correctly — a benchmark)

Each of the four missing test binaries calls apply_coverage(...) under
AF_COVERAGE, so they are instrumented and they do write .profraw that
llvm-profdata merge folds into merged.profdata. But llvm-cov maps counters
through the binary, and a binary passed via neither the positional argument
nor -object contributes no regions. The profile data is produced and then
silently dropped.

What it costs, measured

include/morph/net contributes zero files to the uploaded report:

$ jq -r '[.files[].name|split("/")[0:3]|join("/")]|unique|.[]' <codecov totals> | grep ^include/
include/morph/core      include/morph/detail    include/morph/forms
include/morph/journal   include/morph/offline   include/morph/qt
include/morph/render    include/morph/session   include/morph/util

include/morph/net is absent. It is not a stub: socket_backend.hpp (28.8 kB)
and socket_server.hpp (12.0 kB) are 955 lines between them, plus net/detail/
(tcp_socket, ws_handshake, ws_frame, sha1, base64), and the subsystem
holds 42 of the library's 148 throw sites — the second-largest concentration
after core. tests/net/ has eight test files driving exactly those headers.
So the network stack is tested and scored by nothing.

Two more numbers are suspect for the same reason, and should be re-read rather
than believed:

  • include/morph/offline/sqlite_offline_queue.hpp57.04%, the worst file
    in the library, while morph_offline_sqlite_tests (its dedicated suite) is
    unprofiled.
  • include/morph/qt — 13 lines total in the report, while morph_qt_tests is
    unprofiled.

Why this is the third time

This is the same defect as morph#141 (rungs 2-4 shipped without ever being added
to coverage.sh, leaving ~15k lines outside the number) and morph#179
(coverage.sh's hand-copied rung list had drifted, so ledger and lims
contributed nothing while codecov.yml's ledger component scored a set of
files no report contained). Both were fixed by removing a hand-maintained list —
the rung loop now reads examples/rungs.txt.

The test-executable list was never given the same treatment, and coverage.sh's
own comment predicts this precisely:

Nothing fails when a rung is forgotten -- the script runs, the report uploads,
and the figure is simply computed over a shrinking fraction.

The constraint

morph_bench must stay out (a benchmark is not a test), and morph_soak is a
judgement call — it is a test binary but a long-running one that no ordinary
coverage leg runs. The fix is not "profile every executable"; it is "derive the
list from something that cannot drift, and state the exclusions".

Also: adding four binaries will move every number in codecov.yml, in both
directions — net appears from nothing, and offline/qt should rise. Every
target in that file was set from a measured ceiling, so those measurements have
to be retaken, not adjusted.

Verification status: reproduced

Codecov API, branch master: zero files under include/morph/net; nine
add_executable targets found by grep -rn add_executable tests/ --include=CMakeLists.txt;
apply_coverage confirmed present in tests/net/CMakeLists.txt,
tests/offline_sqlite/CMakeLists.txt and tests/qt/CMakeLists.txt;
TEST_EXE/OBJECT_ARGS assignments read from scripts/coverage.sh on master.

Not verified: that adding the four binaries actually raises the three
suspect numbers. It should, but a coverage run is the only proof and I did not
run one. It is possible sqlite_offline_queue.hpp's 57% is real and its suite
is thin — that outcome would be a different and more serious finding, not a
disappointment.

What would close this

  1. coverage.sh derives its -object list from the build tree or from CMake
    rather than from hand-written names, on the same principle that fixed
    morph#179. Any deliberate exclusion (morph_bench, and morph_soak if it
    stays out) is named in the script with its reason.
  2. include/morph/net appears in the uploaded report with a real percentage.
  3. Something fails when a new coverage-instrumented test executable is added
    and not profiled — in the spirit of scripts/check_rung_filters.sh, which
    already does exactly this for the codecov components a script cannot read.
  4. The three suspect numbers above are re-measured and codecov.yml's affected
    targets are re-derived from the new ceilings, with the old and new figures
    both recorded.

Blocks: every other coverage ticket in this sprint. A target set against
today's denominator is a target set against the wrong tree.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: ciSubsystem: cibugSomething isn't workingtriage: validWell-framed; implement as written

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions