Skip to content

testing: morph has SQLite-shaped test instruments and no charter stating what they collectively guarantee #407

Description

@Yaraslaut

What this asks for

An explicit, written testing charter for morph, modelled on SQLite's — naming
which of its techniques this project adopts, which it declines, and why. Today
the repository has an unusually strong set of test instruments and no document
that says what they are collectively supposed to guarantee
, so gaps are found
one at a time by whoever happens to look.

What morph already has

More than the absence of a charter suggests:

SQLite technique morph's equivalent today
Very high statement coverage 95.69% lines on include/morph
100% branch / MC-DC coverage branch data measured, no target — morph#404
Out-of-memory injection tests/oom_injector.{hpp,cpp} — narrow, and disabled under ASan/TSan/Valgrind
I/O error injection examples/common/testkit/fault_proxy.hpp — ladder-side only
Fuzzing tests/fuzz/fuzz_wire_decode, fuzz_dispatch_execute, with a corpus
Valgrind / leak checks Valgrind memcheck CI leg
Sanitizers ASan, UBSan, TSan CI legs
Long-running / soak tests/soak/
Compile-time contract checks tests/compile_checks/
Multiple independent harnesses Catch2 suite + the scenario corpus (wire-level, example rungs)
Mutation testing of the library none — morph#405
Crash / power-loss simulation none

What is missing beyond the two filed gaps

  • No stated guarantee. SQLite's value is not any single technique; it is a
    published claim ("100% MC/DC") that every technique serves and that a reader
    can check. morph has instruments without a thesis.
  • Instrument reach is undocumented. oom_injector is disabled on three CI
    legs; fault_proxy is ladder-only. Neither limitation is recorded anywhere a
    reader would look before trusting the suite.
  • No I/O-failure seam for the framework itself. The network stack
    (include/morph/net, 42 throw sites) has no equivalent of SQLite's
    per-syscall error injection.

The constraint

SQLite's headline coverage is produced by TH3, which is proprietary and not
shipped with SQLite. Copying the claim without the harness is how a charter
becomes aspirational. This document must say what morph will actually do with
the tools it can obtain, and must not assert a coverage class the repository
does not measure. Where a technique is declined — crash simulation, for a
library with no storage engine of its own — the decline is recorded with its
reason, which is as useful to a reader as an adoption.

Verification status: inferred from reading, not run

The table above is assembled from the tree on master: the directories and
files named all exist, and the oom_injector limitation is quoted from its own
header comment. The coverage figure is from the Codecov API for master.

Not verified: SQLite's current internal practice beyond what its public
documentation states. This ticket should cite that documentation rather than
recollection.

What would close this

A document under docs/ that:

  1. States morph's testing guarantee in a checkable form — a coverage class and a
    number, not an adjective.
  2. Lists each technique as adopted (with where it runs and where it does not) or
    declined (with why).
  3. Names, for each adopted technique, the check that fails when it regresses. A
    technique with no failing check is listed as unenforced, honestly.
  4. Is referenced from CONTRIBUTING.md, so it is reachable by someone deciding
    what tests a change needs.

Not closable by a document asserting a guarantee nothing measures. Every
claim in it must name its enforcing check or be marked unenforced.

Sequenced after morph#403, morph#404 and morph#405 — the charter should
describe what is true once the denominator is right and the two gaps have
measurements, not restate the aspiration that produced them.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: docsSubsystem: docsdocumentationImprovements or additions to documentationtriage: validWell-framed; implement as written

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions