What this asks for
An explicit, written testing charter for morph, modelled on SQLite's — naming
which of its techniques this project adopts, which it declines, and why. Today
the repository has an unusually strong set of test instruments and no document
that says what they are collectively supposed to guarantee, so gaps are found
one at a time by whoever happens to look.
What morph already has
More than the absence of a charter suggests:
| SQLite technique |
morph's equivalent today |
| Very high statement coverage |
95.69% lines on include/morph |
| 100% branch / MC-DC coverage |
branch data measured, no target — morph#404 |
| Out-of-memory injection |
tests/oom_injector.{hpp,cpp} — narrow, and disabled under ASan/TSan/Valgrind |
| I/O error injection |
examples/common/testkit/fault_proxy.hpp — ladder-side only |
| Fuzzing |
tests/fuzz/ — fuzz_wire_decode, fuzz_dispatch_execute, with a corpus |
| Valgrind / leak checks |
Valgrind memcheck CI leg |
| Sanitizers |
ASan, UBSan, TSan CI legs |
| Long-running / soak |
tests/soak/ |
| Compile-time contract checks |
tests/compile_checks/ |
| Multiple independent harnesses |
Catch2 suite + the scenario corpus (wire-level, example rungs) |
| Mutation testing of the library |
none — morph#405 |
| Crash / power-loss simulation |
none |
What is missing beyond the two filed gaps
- No stated guarantee. SQLite's value is not any single technique; it is a
published claim ("100% MC/DC") that every technique serves and that a reader
can check. morph has instruments without a thesis.
- Instrument reach is undocumented.
oom_injector is disabled on three CI
legs; fault_proxy is ladder-only. Neither limitation is recorded anywhere a
reader would look before trusting the suite.
- No I/O-failure seam for the framework itself. The network stack
(include/morph/net, 42 throw sites) has no equivalent of SQLite's
per-syscall error injection.
The constraint
SQLite's headline coverage is produced by TH3, which is proprietary and not
shipped with SQLite. Copying the claim without the harness is how a charter
becomes aspirational. This document must say what morph will actually do with
the tools it can obtain, and must not assert a coverage class the repository
does not measure. Where a technique is declined — crash simulation, for a
library with no storage engine of its own — the decline is recorded with its
reason, which is as useful to a reader as an adoption.
Verification status: inferred from reading, not run
The table above is assembled from the tree on master: the directories and
files named all exist, and the oom_injector limitation is quoted from its own
header comment. The coverage figure is from the Codecov API for master.
Not verified: SQLite's current internal practice beyond what its public
documentation states. This ticket should cite that documentation rather than
recollection.
What would close this
A document under docs/ that:
- States morph's testing guarantee in a checkable form — a coverage class and a
number, not an adjective.
- Lists each technique as adopted (with where it runs and where it does not) or
declined (with why).
- Names, for each adopted technique, the check that fails when it regresses. A
technique with no failing check is listed as unenforced, honestly.
- Is referenced from
CONTRIBUTING.md, so it is reachable by someone deciding
what tests a change needs.
Not closable by a document asserting a guarantee nothing measures. Every
claim in it must name its enforcing check or be marked unenforced.
Sequenced after morph#403, morph#404 and morph#405 — the charter should
describe what is true once the denominator is right and the two gaps have
measurements, not restate the aspiration that produced them.
What this asks for
An explicit, written testing charter for morph, modelled on SQLite's — naming
which of its techniques this project adopts, which it declines, and why. Today
the repository has an unusually strong set of test instruments and no document
that says what they are collectively supposed to guarantee, so gaps are found
one at a time by whoever happens to look.
What morph already has
More than the absence of a charter suggests:
include/morphtests/oom_injector.{hpp,cpp}— narrow, and disabled under ASan/TSan/Valgrindexamples/common/testkit/fault_proxy.hpp— ladder-side onlytests/fuzz/—fuzz_wire_decode,fuzz_dispatch_execute, with a corpusValgrind memcheckCI legtests/soak/tests/compile_checks/What is missing beyond the two filed gaps
published claim ("100% MC/DC") that every technique serves and that a reader
can check. morph has instruments without a thesis.
oom_injectoris disabled on three CIlegs;
fault_proxyis ladder-only. Neither limitation is recorded anywhere areader would look before trusting the suite.
(
include/morph/net, 42throwsites) has no equivalent of SQLite'sper-syscall error injection.
The constraint
SQLite's headline coverage is produced by TH3, which is proprietary and not
shipped with SQLite. Copying the claim without the harness is how a charter
becomes aspirational. This document must say what morph will actually do with
the tools it can obtain, and must not assert a coverage class the repository
does not measure. Where a technique is declined — crash simulation, for a
library with no storage engine of its own — the decline is recorded with its
reason, which is as useful to a reader as an adoption.
Verification status: inferred from reading, not run
The table above is assembled from the tree on
master: the directories andfiles named all exist, and the
oom_injectorlimitation is quoted from its ownheader comment. The coverage figure is from the Codecov API for
master.Not verified: SQLite's current internal practice beyond what its public
documentation states. This ticket should cite that documentation rather than
recollection.
What would close this
A document under
docs/that:number, not an adjective.
declined (with why).
technique with no failing check is listed as unenforced, honestly.
CONTRIBUTING.md, so it is reachable by someone decidingwhat tests a change needs.
Not closable by a document asserting a guarantee nothing measures. Every
claim in it must name its enforcing check or be marked unenforced.
Sequenced after morph#403, morph#404 and morph#405 — the charter should
describe what is true once the denominator is right and the two gaps have
measurements, not restate the aspiration that produced them.