fix(behavior): stop scoring frames whose window straddles a bout edge - #104
Merged
Conversation
The rolling window is causal: row i's features summarise frames i-window+1 .. i. For the first window-1 frames of every bout that span still covers the PREVIOUS behavior, so cross-validation was asking the model to name a behavior from features that mostly describe a different one, then counting it wrong when it obliged. Those frames cannot be classified correctly however good the model is. Scoring them measures where the annotator drew the boundary, not the classifier. On a 34-session cohort with ~25-frame bouts at window 8 they are 28% of all scored frames, and excluding them moved measured macro F1 by about +0.02 on a held-out set -- performance that was always there and was being hidden. Only the leading edge is affected; the end of a bout is fine, its window lies wholly inside it. Training is deliberately untouched: a partly-filled window is still a real input the deployed model will meet, and dropping those rows would shrink the training set for no gain. Adds n_rows_scored to the CV result. n_rows_kept counts rows assembled for training, which now differs from what was measured, and one existing test was asserting the two were equal -- that invariant is exactly what changed, so it now asserts support sums to n_rows_scored instead. Note this shifts reported CV numbers upward. Runs measured before and after are not directly comparable, and the change is toward the honest figure.
This was referenced Aug 6, 2026
gbradham
added a commit
that referenced
this pull request
Aug 6, 2026
#104 landed the clean-window scoring rule this branch had copied ahead of it, so both sides defined _clean_window_rows identically and git took one copy. The only real conflict was the helper's name: main still calls it _bout_recall, this branch makes it public bout_metrics so evaluation and cross-validation can share one definition of a detected bout. Kept the rename, which is the point of the branch. n_rows_scored from #104 comes through untouched.
gbradham
added a commit
that referenced
this pull request
Aug 6, 2026
_run_cv_folds has reported n_rows_scored since #104, but the summary written into the model bundle never copied it. A report could therefore only say how many rows were assembled -- n_rows_kept, which counts mirrored copies and window-contaminated frames that train but never score -- leaving the reader to work the real denominator out from the per-class supports. On model11 that was 17,129 scored against 25,005 kept, a 31% difference.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The rolling window is causal: row i's features summarise frames
i-window+1 .. i. For the firstwindow-1frames of every bout, that spanstill covers the previous behaviour. Cross-validation was asking the model
to name a behaviour from features that mostly describe a different one, then
counting it wrong when it obliged.
Those frames cannot be classified correctly however good the model is. Scoring
them measures where the annotator drew the boundary, not the classifier.
Only the leading edge is affected — the end of a bout is fine, its window
lies wholly inside it.
Impact
On a 34-session cohort with ~25-frame bouts at window 8, these are 28% of all
scored frames. Applying the same rule to a held-out evaluation moved macro F1
+0.022 (0.685 → 0.707) and accuracy +0.022 — performance that was always
there and was being hidden by frames the model had no way to get right.
This shifts reported CV numbers upward. Runs measured before and after are
not directly comparable. The change is toward the honest figure, but it is a
behaviour change and worth knowing when comparing against previously recorded
results.
Training is untouched
Deliberately. A partly-filled window is still a real input the deployed model
will meet at a genuine behaviour transition, and dropping those rows would
shrink the training set for no gain. This is a scoring change only.
n_rows_scoredn_rows_keptcounts rows assembled for training, which now differs from whatwas measured. The result dict gains
n_rows_scoredso nobody divides by thewrong denominator.
One existing test asserted
sum(support) == n_rows_kept— that invariant isprecisely what this changes, so it now asserts
== n_rows_scoredplusn_rows_scored <= n_rows_kept. Worth a look, since it changes an existingassertion rather than adding one.
Tests
Ten cases covering the rule directly: leading frames excluded, trailing frames
kept, bouts shorter than the window never scorable, session boundaries not
bridged, frame-number gaps restarting the window,
window=1a no-op,out-of-order rows handled, and the excluded count equalling
window-1across several window sizes.
2813 passed, 1 skipped; ruff clean.