Skip to content

MemKraft 3.0.1

Choose a tag to compare

@seojoonkim seojoonkim released this 16 Jul 16:07

MemKraft 3.0.1

Released 2026-07-17. This record describes the source tree tagged v3.0.1;
public package and release URLs are listed in the repository release metadata.

Gym gate evidence: docs/bench/baselines/3.0.1.json.

MemKraft 3.0.1 is a reliability patch for deterministic claim precision,
Preview candidate governance and local sidecar maintenance, and observable
compiled-truth freshness.

  • Improves English claim precision and Korean claim extraction, backed by
    stricter Memory Gym precision, retrieval, extraction, and freshness gates.
  • Filters governed structured candidates, supports auditable candidate
    forgetting, and adds dry-run-first compaction for local event and candidate
    sidecars. Compaction does not claim erasure from backups, snapshots, version
    control, or external copies.
  • Adds Preview truth_status(), fail-closed compiled-truth cache invalidation
    and governance checks, and cold/warm derived-view benchmark coverage.
  • Adds public compile_evidence_context(...) wiring, with temporal
    latest/past/compare selection based on validated source timestamps and
    preservation of numeric evidence with provenance.
  • Adds public aggregate_numeric_evidence(...) for strict sum, count, and
    duration aggregation. Unsupported or ambiguous inputs fail closed, and
    successful aggregates retain source provenance.

Verification evidence

Evidence Result Notes
Memory Gym 11/11 gates passed Current final code; baseline regenerated 2026-07-17 and includes truth_freshness.
Full regression 2,016 passed, 3 skipped, 258 warnings Current final tree, 2026-07-17; fail/error 0.
External benchmark (historical LongMemEval oracle candidate) N=50, 2026-07-14 Pre-final dirty candidate at HEAD 1bef5b0; it predates the final evidence-context, temporal, and numeric changes and does not validate them. See the reproducibility record.

Historical LongMemEval candidate record

The following measurements are retained exactly as historical pre-final
candidate evidence. They are not clean release evidence and must not be used to
claim validation of compile_evidence_context(...), validated-timestamp
temporal selection, numeric-evidence preservation, or
aggregate_numeric_evidence(...) in the final code.

  • Dataset/run: local LongMemEval oracle, seeded stratified N=50 (seed=42), run on 2026-07-14.
  • System under test: dirty MemKraft 3.0.1 candidate at HEAD 1bef5b0, before the current evidence/temporal/numeric changes; top_k=15.
  • Generation metadata: harness adapter litellm-vhh, model gpt-5.6-terra; the full non-secret provider configuration and execution settings are in the evidence manifest.
  • Automatic scores: contains-match 68.0% (34/50); exact-match 6.0%; execution errors 0/50.
  • End-to-end latency: p50 2,899.741 ms; p95 6,951.518 ms; min 1,097.256 ms; max 9,206.981 ms. This includes the full run_sample() path, not only retrieval.
  • Semantic score: an LLM judge using the same gpt-5.6-terra model marked 48/50 = 96.0% correct, with 0 judge errors. This is a self-judge score, not an independent-model evaluation; the automatic contains-match result is retained alongside it to make that limitation explicit.
  • Artifact: local judged JSON SHA-256 80ccd83b4162def0275835e7f01b29ae6287467e699e22c7c3b418370ca4c4b3. It is intentionally ignored with the LongMemEval artifacts. No public URL is currently available.
  • Excluded run: a later attempt returned APIConnectionError for all 50/50 samples and is invalid, so it is excluded from release evidence. A final-code rerun is currently blocked because DNS for litellm-vhh fails.

Latency baseline exception

The only prior committed Gym baseline is 2.13.0. Its search_recall workload
uses top_k=5, while the 3.0.1 baseline uses top_k=20; the workloads are not
comparable for the checklist's +15% latency rule. This release therefore does
not claim a cross-version latency improvement or regression result. The 3.0.1
baseline establishes the comparable starting point for the next release; a
matching-workload rerun is required before public release if the historical
comparison is needed.

The patch is additive and read-compatible. It does not rewrite existing memory
and requires no destructive migration.