Skip to content

MemKraft 3.0.2

Choose a tag to compare

@seojoonkim seojoonkim released this 17 Jul 13:41

MemKraft 3.0.2

Release date: 2026-07-17. This patch packages the evidence-context and benchmark
reliability work completed after 3.0.1.

Gym gate evidence: docs/bench/baselines/3.0.2.json.

What changed

  • Compiled evidence now retains validated UTC source_time provenance for
    temporal interval reasoning. Missing, invalid, and weekday-mismatched source
    dates fail closed.
  • Hard context budgets preserve evidence text before optional timestamps.
  • Selective context compilation applies only to standard and temporal routes;
    aggregation, preference, ordinal, and empty-compiler cases keep the legacy
    full-context fallback.
  • LongMemEval evaluation preserves historical exact/contains metrics and adds a
    conservative canonical metric for surface-form equivalence only.
  • LongMemEval artifacts now record sample-level routing, latency, context size,
    and output size.
  • MK_SPLIT_RECOMMENDATION_ROUTE=1 optionally sends explicit past
    recommendation recall to the factual route. New recommendations and
    meta-reminders stay on the preference route. The default remains disabled.

External benchmark evidence

The paired runs below used the same N=50, seed 42, top_k=15, endpoint,
provider, and gpt-5.6-sol model. They are benchmark-harness evidence, not a
full LongMemEval claim.

  • 3.0.1 control: exact 8%, contains 64%, canonical 80%, p50 4,067 ms,
    p95 11,069 ms, execution errors 0.
  • Selective evidence candidate: contains 66%, canonical 80%, paired
    canonical loss 0, p50 3,779 ms. Its p95 was 12,579 ms, so this release does
    not claim a p95 improvement.
  • Recommendation split candidate: exactly three explicit past-recall cases
    changed route. In the paired runs, those cases showed output length of 996
    versus 145 characters (-85.4%) and median latency of 2,961 versus 2,701 ms
    (-8.8%), with canonical loss 0. These are observed run differences, not an
    isolated causal estimate of routing alone. The whole-run contains change on
    an unaffected sample is treated as model/run variance, not a routing
    regression.

Reproduction details and artifact names are recorded in
docs/bench/longmemeval-3.0.1.md. Benchmark
result JSON remains local and is not claimed as a public release artifact.

Compatibility and migration

  • No destructive migration is required.
  • Existing exact/contains evaluator outputs remain available.
  • Selective evidence compilation fails open to legacy context when it cannot
    produce safe evidence.
  • Recommendation route splitting is opt-in; existing default behavior is
    unchanged.

Validation

  • Full test suite: 2,031 passed, 3 skipped, 0 failed.
  • Memory Gym: 11/11 scenarios passed; see the committed baseline above.
  • twine check: passed for wheel and sdist.
  • Fresh isolated wheel install: CLI version/init/doctor, entity write/search,
    search_v2 content retrieval, and evidence/numeric API checks passed.
  • Artifact SHA-256:
    • wheel: a66c3408a55376511bc4e4c5caa2217ad34e383c9577a41a33241b6b128357c2
    • sdist: b3521ab6ff305f757bcfe2c51e1300842478a78cf7e1ed3ee7af9dd46c1b165d
  • Hermes provider smoke is performed against the published PyPI artifact after
    upload; the result is reported in the GitHub release.