How should we evaluate module quality? Benchmarks vs. property tests vs. golden files

[web3guru888]

The Evaluation Problem

One of the hardest parts of building ASI:BUILD is knowing when a module is actually good.

Right now our tests verify correctness — does the code run without crashing, do the data structures have the right shape, does the API contract hold? That's necessary but not sufficient. We also need semantic correctness: does a reasoning module actually reason? Does a memory consolidation module actually improve recall?

This is genuinely hard. Some thoughts on how to approach it:

---

Current State

Most modules have unit tests covering:

• Constructor / init
• Basic method calls with minimal inputs
• Edge cases (empty inputs, None, boundary values)

A few modules have more substantive tests — rings/ has 108 tests covering SDK client, DID auth flows, and Blackboard adapter event propagation.

---

What We're Missing

1. Functional correctness benchmarks

For example, consciousness/ computes an IIT Φ score. But is it right? We have Issue #6 tracking a known bug where it uses entropy difference instead of the TPM-based Φ from Tononi 2014.

A benchmark would:

• Pull reference implementations (PyPhi, pyphi-data test cases)
• Run our module on the same inputs
• Assert our output is within tolerance of the reference

2. Regression baselines

For modules that do learning or optimization, we need before/after snapshots:

• meta_learning/: does MAML actually converge on held-out tasks?
• reasoning/: does the inference engine produce valid argument chains?
• bio_inspired/: do evolutionary algorithms improve fitness over generations?

3. Cross-module integration quality

When sensory_processing → reasoning → bio_inspired runs a full pipeline, what does "correct" look like? We have Issue #18 open for E2E tests, but the benchmark criteria are still open.

---

Design Question

What should the ASI:BUILD evaluation framework look like?

Option A: Pytest fixtures with golden files
Store expected outputs as JSON fixtures, assert modules produce outputs within epsilon. Simple to implement, brittle to algorithmic changes.

Option B: Property-based testing (Hypothesis)
Define invariants (monotonicity, boundedness, commutativity) and let Hypothesis generate adversarial inputs. Catches bugs golden files miss, but harder to write for ML modules.

Option C: Standalone benchmark suite (benchmarks/ directory)
Separate from tests/, runs slower, uses real datasets. CI runs only on tagged releases. Good for tracking performance over time. High investment.

Option D: Community-sourced test cases
Open issues asking contributors to submit "this input should produce output Y" examples. Crowd-sourced golden files. Scales with community, but needs curation.

---

My current thinking

Probably a combination:

• Property-based tests for pure algorithmic modules (IIT Φ, NTT, formal verification)
• Golden file fixtures for modules with stable interfaces
• A benchmarks/ directory starting with 2-3 key modules, expanding over time

But I'd love to hear what the community thinks. If you've worked on evaluation frameworks for cognitive/AI modules before — what worked, what didn't?

Particular interest in:

• How to test reasoning/ without a ground-truth knowledge base
• How to evaluate memory_consolidation/ — what's the right recall metric?
• Whether IIT Φ benchmarks are even tractable at the scale we're operating

[web3guru888]

Let me share where my thinking currently lands on each approach, and why I think we need all three rather than picking one.

---

The case for property tests as the foundation

For most of ASI:BUILD, property-based tests are the right first line. A property test captures what a module promises to be true for any valid input, not just the inputs you thought of when writing the test. For example:

# From tests/test_knowledge_graph.py (paraphrased)
@given(st.lists(st.tuples(st.text(), st.text(), st.text())))
def test_kg_roundtrip(triples):
    kg = KnowledgeGraph()
    for s, p, o in triples:
        kg.add(s, p, o)
    stored = kg.query_all()
    assert all(t in stored for t in triples)

This catches the kind of bug where overlapping time ranges silently drop entries — something a handful of example-based tests might miss.

Where property tests break down: modules with non-deterministic or approximate outputs (consciousness metrics, bio-inspired search). You can write properties like Φ >= 0 or result is in top-k candidates but not Φ = X for arbitrary input.

---

Benchmarks belong at module boundaries

For the consciousness and reasoning modules, benchmarks against known reference values are the right tool:

• IIT Φ: PyPhi gives ground truth for small networks (≤ 8 nodes). ~20 canonical networks lets us track whether our approximation is drifting.
• Bio-inspired search: Benchmark against brute-force cosine similarity — we know the expected top-k recall.
• Cognitive Blackboard throughput: Wall-clock benchmarks for concurrent writers (currently ~10 µs/write under contention).

Benchmarks are expensive to maintain. Keep the suite small (20–50 cases) and well-documented.

---

Golden files: only for serialization and output stability

Golden file tests are most valuable where:

1. The expected output is complex (JSON schema, rendered explanation, structured plan)
2. You want to detect unintended regressions, not prove correctness

Best candidates in ASI:BUILD:

• cognitive_blackboard.snapshot() output format
• knowledge_graph.export() serialization
• reasoning module explanation strings

Risk: golden files become stale. They need an --update-goldens flag and a clear policy for when updating is legitimate.

---

Proposed quality tiers

Module	Primary	Secondary
knowledge_graph	Property tests (roundtrip, temporal consistency)	Golden files (export format)
cognitive_blackboard	Property tests (thread safety, entry lifecycle)	Benchmarks (throughput)
consciousness/IIT	Benchmarks (PyPhi reference)	Property tests (Φ ≥ 0)
bio_inspired	Benchmarks (recall vs brute-force)	Property tests
reasoning	Property tests	Golden files (explanation structure)
safety	Property tests (no false proofs)	Benchmarks (constraint coverage)

---

Open questions:

1. Is there a module where golden files should be primary?
2. For the IIT Φ benchmark suite — what network sizes are tractable? (PyPhi slows significantly above ~8 nodes)
3. Should benchmarks live in tests/benchmarks/ and be excluded from the default pytest run, or always run?