Skip to content

feat: fit Ebbinghaus and exponential forgetting curves to recall@T data#19

Closed
Priyanshu-byte-coder wants to merge 1 commit into
Neal006:mainfrom
Priyanshu-byte-coder:feat/ebbinghaus-curve-fit
Closed

feat: fit Ebbinghaus and exponential forgetting curves to recall@T data#19
Priyanshu-byte-coder wants to merge 1 commit into
Neal006:mainfrom
Priyanshu-byte-coder:feat/ebbinghaus-curve-fit

Conversation

@Priyanshu-byte-coder
Copy link
Copy Markdown
Contributor

Summary

Implements forgetting-curve fitting requested in issue #6.

New function: evaluation/stats.py — fit_forgetting_curve()

Accepts (checkpoints, recalls) and fits two models:

Exponential decay R(t) = a · exp(−k · t)

  • Fitted via log-linear least squares (no extra dependencies)
  • Returns: a, k, half_life = ln(2) / k,

Ebbinghaus forgetting curve R(t) = exp(−t̂ / (S · √(1 + t̂)))

  • t̂ = t / t_max (normalised)
  • S = stability constant; higher = slower forgetting
  • Fitted via fine grid search over S ∈ [0.01, 20] (avoids scipy dependency)
  • Returns: stability, half_life (numerical),

CLI flag: --fit-curves

python main.py --fit-curves
python main.py --seeds 5 --fit-curves

Example output:

FORGETTING CURVE FIT  (Ebbinghaus + Exponential)
-----------------------------------------------------------------
  naive
    Exponential  k=0.004200  half-life=165.0 turns  R²=0.921
    Ebbinghaus   S=2.1000    half-life=142.3 turns  R²=0.944
  rag
    Exponential  k=0.006800  half-life=101.9 turns  R²=0.887
    Ebbinghaus   S=1.3500    half-life=88.5 turns   R²=0.912
  cascading
    Exponential  k=0.003100  half-life=223.6 turns  R²=0.956
    Ebbinghaus   S=2.8200    half-life=195.0 turns  R²=0.971

Half-life and stability give researchers a single interpretable scalar per backend rather than requiring point-by-point curve inspection.

Closes #6

Adds fit_forgetting_curve() to evaluation/stats.py which accepts a list of
(checkpoint, recall) pairs and returns:
  - exponential model: a, k (decay rate), half-life (ln2/k), R²
  - Ebbinghaus model: stability S, half-life (numerical), R²

The exponential model uses log-linear least squares (no extra dependencies).
The Ebbinghaus model uses a fine grid search over S ∈ [0.01, 20] to avoid
introducing a scipy dependency.

Exposed via --fit-curves CLI flag:
  python main.py --fit-curves
  python main.py --seeds 5 --fit-curves

Half-life and stability let researchers compare how quickly each memory backend
'forgets' injected facts as a single interpretable scalar rather than reading
recall curves point-by-point.

Closes Neal006#6
Neal006 pushed a commit that referenced this pull request May 24, 2026
…ta (#19)

Adds fit_forgetting_curve() to evaluation/stats.py — fits both exponential
(log-linear OLS) and Ebbinghaus (grid-search over S) models to recall@T
time-series and reports half-life, stability constant, and R² per backend.

CLI: --fit-curves flag prints the analysis after any benchmark run, works
with both single-seed and multi-seed (uses mean recall).

Merge resolves conflict with PR #18 --scenario arg; both args preserved.

Co-authored-by: Priyanshu-byte-coder
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Neal006
Copy link
Copy Markdown
Owner

Neal006 commented May 24, 2026

Manually squash-merged into main (8a58920) after resolving conflict with PR #18's --scenario argument. Both --fit-curves and --scenario flags are now live on main. Thanks @Priyanshu-byte-coder!

@Neal006 Neal006 closed this May 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fit an Ebbinghaus forgetting curve to Recall@T data

2 participants