Repository navigation
v4.10.0
COLEMAN 4.10 — Inspect controlled faults across Java codebases
Reproduce a small, explicit matrix of real upstream fixes before making a
benchmark claim. COLEMAN now collects CLI-13, CLI-51 and CODEC-65 with frozen
sources, common test fixtures and the existing bounded JUnit producer. Every
selected result remains inspectable, including errors and failures that persist
in both revisions.
What you can do
- Collect a declared matrix. One typed plan describes three focal fixes across
Apache Commons CLI and Codec. Pinned source views, original notices, JARs,
runtime image, architecture, timezone and harness make the protocol explicit. - Keep negative evidence. The initial qualification retained 714 role
observations / 357 same-test pairs: 348 both-pass, four both-fail, four
assertion-failed-to-passed and one inconclusive comparison. CLI-51's test error
remains inconclusive under the existing classifier. Three Codec triggering
tests do not become three faults. - Resume without inventing observations. Existing role acknowledgments and
raw packets retain interruption semantics. Typed indices distinguish observed
pairs from entirely unobserved tests; incomplete collection has no completion
marker. A verified completed matrix reopens offline without querying Docker. - Verify the evidence behind the index. Offline validation checks actual
source/context bytes, raw protocols, normalized observations and diagnostic
references before accepting counts, identities or collection status.
The complete matrix guide
provides installation, explicit runtime preparation, source acquisition, typed
Python use, CLI collection/validation, recovery and the installed-package workflow.
Foreign sources, dependencies and captures stay in operator-owned external storage.
Compatibility and limits
The new optional transport adds versioned matrix plans/indices while reusing the
existing generic controlled domain, source cache, JUnit producer and immutable
artifact writer. Old single-pair recipes, controlled artifacts, native results,
RunSpec identities and learning checkpoints retain their semantics and require
no migration. Unknown matrix schemas or producers reject; there is no implicit
reinterpretation of prior evidence.
The producer identity conservatively binds all COLEMAN packaged code/resources,
so even an unrelated packaged change selects a new generation. Incidental
bytecode, hidden files and interpreter caches do not affect that identity.
Codec's common fixture is explicitly partial. Before outcomes were observed,
three fixture files were excluded because of separately noticed third-party data,
non-UTF8 source bytes or dependencies on excluded classes. Remaining source bytes
and original LICENSE/NOTICE are preserved. This is direct-upstream reproduction,
not the complete original build or a full Defects4J reconstruction.
Upstream Base64/Hex/Digest fixtures also generate unseeded random inputs whose
bytes/seeds are not captured. Those tests remain included with an explicit
limitation; deterministic saved-artifact replay does not reconstruct fresh inputs.
The CLI fixes share project history. Multiple tests, seeds or trials do not
establish independent projects or unique faults. Collection clocks and harness
nanoseconds are not historical CI events, runner work or parallel wall-clock
savings. Shared-JVM test-order independence remains unverified. All outputs retain
not_evaluated and insufficient_evidence; diversified source admission, baseline
comparisons and confirmation in #261 remain open.
Validation covers versioned goldens, old recipes, strict units/missingness,
non-finite rejection, deterministic identities, duplicate accounting, negative
results, source/raw/index correspondence, interrupted publication, concurrency,
offline continuation and domain import boundaries. Installed-package checks use
actual Java source collection and a separate native CI exact-answer/checkpoint
workflow. This validates the evidence pipeline, not algorithm effectiveness.