[Community] Conformance Testing -- Help Validate Memory Engines Against MF-001 Rules M0-M12

[jarvispectrayan]

The MF-001 specification defines 13 evaluation tests (Section 10) that any system claiming to be a "memory engine" should pass. Spector currently implements 3 of these as conformance test fixtures. We need help implementing the remaining 10 -- and each one is a well-scoped, self-contained task perfect for a first contribution.

---

Currently Implemented (3/13)

These live in bench/spector-bench/src/test/resources/mf-conformance/:

Test ID	Name	What It Validates
MF-T01	Truncation Trap	Plants an old, high-importance, low-cosine trace as the only correct answer. Single-signal top_k misses it; fused recall finds it. (Validates M2: Guaranteed Access)
MF-T03	Valence Window	Traces outside the cue's emotional window are hard-gated. Inside the window, closer valence ranks higher at equal similarity.
MF-T10	Isolation	Rememberer A cannot recall Rememberer B's traces by cue collision. (Validates M10: Rememberer Isolation)

---

Needs Implementation (10/13)

Each test is defined in MF-001 Section 10. Here is what needs building -- pick one and submit a PR!

Test ID	Name	What to Validate	Difficulty
MF-T02	Emotional Enhancement	Two traces with equal age and importance but different arousal. After time passes, the high-arousal trace should rank higher because its retrieval strength D decayed more slowly.	Beginner
MF-T04	Desirable Difficulty	Rehearse a trace at low D vs high D. The strength gain (delta-S) should be larger when D is low: delta-S = s_gain * (1 - D).	Beginner
MF-T05	Monotonic S	Call forget on a trace. Verify that retrieval strength D decreases but storage strength S does not decrease. S is non-decreasing except on tombstone.	Beginner
MF-T06	Zeigarnik / Unresolved	An unresolved trace (open task loop) should decay more slowly than a resolved one. Pin is a separate flag from unresolved.	Medium
MF-T07	Availability vs Accessibility	Tombstoned traces are unavailable (gone). Decayed traces are available but inaccessible. introspect must distinguish between the two.	Medium
MF-T08	Consolidation Fidelity	After consolidate, the resulting semantic trace must have lineage pointing back to the source episodic traces.	Medium
MF-T09	Operation Partition	recall does not rewrite payload. rehearse does not return payload. reconsolidate is the only operation that rewrites payload, and it writes lineage.	Medium
MF-T11	Source Gate	After simulate, default recall returns no hypothetical traces. recall with allow_simulated = true may. commit_simulation never yields experienced source.	Advanced
MF-T12	Dream Write Budget	After dream, verify no new experienced episodic traces exist. Only simulated traces, association updates, and strength updates are permitted.	Advanced
MF-T13	Signal Ablation	Disabling decay or the arousal term of the decay law changes rank ordering after time has passed. This validates that signals are genuinely participating in scoring, not no-ops.	Advanced

---

How to Contribute a Test

Test Fixture Format

Each test lives in its own directory under mf-conformance/:

bench/spector-bench/src/test/resources/mf-conformance/
+-- MF-T01-truncation-trap/
|   +-- setup.json        # Traces to pre-load
|   +-- cue.json          # The recall cue
|   +-- expected.json     # Expected behavior (which trace should rank where)
+-- MF-T02-emotional-enhancement/   # <-- YOU BUILD THIS
|   +-- setup.json
|   +-- cue.json
|   +-- expected.json

Step-by-Step

1. Read the test definition in MF-001 Section 10 (linked above)
2. Create the fixture directory with setup.json, cue.json, expected.json
3. Write the JUnit test in bench/spector-bench/src/test/java/ that loads the fixtures, runs the scenario, and asserts the expected behavior
4. Submit a PR -- we will review and merge!

Example: MF-T02 (Emotional Enhancement)

// setup.json -- Two traces with equal age and importance, different arousal
{
  "traces": [
    {
      "id": "trace-high-arousal",
      "text": "The server crashed during the product demo",
      "tier": "EPISODIC",
      "importance": 5.0,
      "valence": -50,
      "arousal": 200,
      "age_hours": 720
    },
    {
      "id": "trace-neutral",
      "text": "Regular weekly team sync meeting notes",
      "tier": "EPISODIC",
      "importance": 5.0,
      "valence": 0,
      "arousal": 10,
      "age_hours": 720
    }
  ]
}

// cue.json
{
  "query": "what happened at the demo",
  "top_k": 2
}

// expected.json -- High-arousal trace should rank first (D decayed less)
{
  "ranking": ["trace-high-arousal", "trace-neutral"],
  "constraint": "trace-high-arousal.D > trace-neutral.D"
}

---

Should Conformance Tests Be Language-Agnostic?

One open question: should these test fixtures be defined as pure JSON (setup traces + cue + expected output) so that any memory engine in any language can validate against MF-001?

This would make MF-001 not just a spec document but a runnable conformance suite -- like SQL conformance tests or HTTP/2 h2spec.

---

Discussion Questions

1. Want to implement a test? Comment with the test ID (MF-T02 through MF-T13) and we will assign it to you!
2. Should conformance tests be language-agnostic JSON fixtures or language-specific test code?
3. How would you test M2 (Guaranteed Access) at scale -- 1M+ traces -- without making the test take 10 minutes?
4. What edge cases should we add beyond the 13 defined in Section 10?
5. Should there be a conformance badge ("MF-001 Conformant") for systems that pass all 13 tests?