Skip to content

fix(models): version learned PV and load state against its feature vector - #793

Merged
frahlg merged 1 commit into
masterfrom
harvest-version-learned-model-state
Aug 4, 2026
Merged

fix(models): version learned PV and load state against its feature vector#793
frahlg merged 1 commit into
masterfrom
harvest-version-learned-model-state

Conversation

@frahlg

@frahlg frahlg commented Aug 4, 2026

Copy link
Copy Markdown
Member

The problem

Both learned models restored by json.Unmarshal of a bare blob, gated only on
one sanity field — Forgetting > 0 for the PV twin, Alpha > 0 for the load
model — followed by a hand-written migration. Nothing recorded which feature
vector the coefficients had been fitted against.

That is enough to survive a corrupt file. It is not enough to survive a change
to the features, and a change to the features is the failure that matters:

  • add a third time-of-day harmonic, or revive the dead intercept slot, and
    every stored Beta indexes a vector it was never fitted to;
  • change what clearSkyW means — horizontal GHI to plane-of-array, say — and
    Features() returns the same numbers for the same argument while the
    coefficients now describe a different physical quantity;
  • move the bucket a moment maps to and every learned hour-of-week mean shifts
    under the model.

In all three cases the model keeps predicting. It reports a plausible MAE, it
passes every sanity gate it has, and it steers the plan from a fit for a
different world. Nothing in the stored numbers gives it away.

FTW has paid for this class of fault before: 3255deba (local-time bucket
indexing corrupting the learned models across DST) and 41e59efb (models
locking themselves out). Both were found from their effects.

The change

Each persisted model is now wrapped in {schema_version, feature_hash, model}
(go/internal/modelstate). On any mismatch — hash, envelope schema, unreadable
blob — the state is discarded and the model cold starts, logging at Info with
both hashes. A cold start is bounded and self-healing for both models. Silent
wrong coefficients are neither.

How the hash is derived, and why. The brief allowed a hand-maintained
constant as a fallback. That is the failure mode being fixed, so it is a
hybrid, and the automatic half carries the weight:

  • Automatic (featureProbe) — the feature functions are evaluated over a
    fixed grid and the results are hashed. Add a harmonic, reorder a slot, revive
    the dead intercept, change the cloud exponent, drop the UTC coercion in
    HourOfWeek, move HeatingReferenceC: the probe values move, so the hash
    moves, with nobody having to remember anything. Verified by temporarily
    changing 2*h to 2.001*h in Features — the hash moved and the tripwire
    fired, with no constant touched.
  • Declared (featureSemantics) — one short string next to the feature
    definition, naming what the inputs mean. This is the half a probe provably
    cannot derive: Features(clearSkyW, …) returns identical numbers whether the
    caller passes horizontal GHI or plane-of-array irradiance. There is no way in
    Go to fingerprint the meaning of a float64 without wrapping it in a type, a
    refactor that would reach into forecast.go and pvperf — the exact files
    this PR must stay out of. So it is declared, in one obvious place, with a
    comment saying it must change when a caller's input does.

Rounding. Probe values are hashed at twelve significant digits: far finer
than any real feature change, coarse enough to absorb the sub-ULP spread the
same expression can have between an amd64 build host and the arm64 Pi. A state
database stays portable.

Pre-envelope state is adopted, and the path retires itself. Every deployed
site has a bare blob. Discarding those would cost the PV twin an afternoon and
the load model weeks of bucket coverage, for no safety gain today: that state
was fitted against exactly the feature space this build still computes. So each
package freezes the fingerprint that was current when the envelope landed
(legacyFeatureHash) and adopts unversioned state only while FeatureHash()
still equals it. The first feature change breaks that equality and unversioned
state is discarded from then on — no cleanup commit, nobody to remember. The
tests covering that path skip themselves the same way.

Deliberate scope

Tests

go/internal/modelstate covers the envelope: round trip, hash mismatch,
unknown schema version, nine flavours of corrupt blob, and the self-retiring
legacy adoption. Unwrap decodes into a scratch value and copies over only on
success, so a blob damaged halfway through can never leave a model that is part
restored state and part cold start.

Each model package covers the same behaviours end to end through
NewService, plus a pinned-hash tripwire so a feature change cannot ship
without someone noticing that every deployed site will relearn.

Verified the guard bites: removing the hash comparison in Unwrap fails
TestStateFittedAgainstOtherFeaturesColdStarts in both packages and
TestUnwrapDiscardsOnFeatureHashMismatch in modelstate; restoring it passes.

make verify clean.

If TestFeatureHashPinned fails on your branch: that is the tripwire doing
its job. Confirm the feature change is intended, update the literal in the
test, and say in your changeset that the model cold-starts on upgrade. Do
not update legacyFeatureHash in service.go — that constant is frozen,
and moving it would restore pre-envelope coefficients under your new
features, which is the exact fault this PR guards against.

🤖 Generated with Claude Code

…ctor

Both learned models restored by unmarshalling a bare blob, gated only on a
single sanity field, with nothing recording which feature vector the
coefficients were fitted against. Change the harmonic count, the bucket a
moment maps to, or what clear-sky irradiance refers to, and a stale Beta keeps
predicting with a plausible-looking MAE from a fit that describes a different
world.

Each persisted model is now wrapped in {schema_version, feature_hash, model}.
The fingerprint is derived from the feature functions themselves, evaluated
over a fixed probe, so a change to the feature math moves it with nobody
having to remember a constant; a short declared-semantics label next to the
feature definition covers the one thing a probe cannot see, a caller passing
in a differently defined quantity. On mismatch the model logs both hashes at
Info and cold starts, which both models recover from and neither recovers
from silent wrong coefficients.

Pre-envelope state is adopted, but only while the running build still computes
the space it was fitted against: each package freezes the fingerprint that was
current when the envelope landed, and the first feature change retires that
path on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@frahlg
frahlg merged commit bd18d7a into master Aug 4, 2026
13 checks passed
@frahlg
frahlg deleted the harvest-version-learned-model-state branch August 4, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant