Repository navigation
v0.5.0
Added
- WHOOP import (
--source whoop). Reads JSON saved from the WHOOP Developer API v2 (a JSON array of
records, or one{"records": [...]}response): sleep (sleep_hours, asleep time only), recovery
(resting_heart_rate,hrv_msas an RMSSD) and workouts (workout_minutes). Every value is kept as its
own observation with the sourceWHOOP, local days come from each record'stimezone_offset, repeated
records import once, and a bad record is skipped and reported without stopping the rest. Written against
WHOOP's documented v2 sample responses and tested on synthetic data only; the WHOOP app's CSV export is
not supported. Seedocs/importing.mdfor the exact rules and what is left out. - Consistent backups and safe restores (
quantified-self-backup). Copyinghealth.dbis not a backup:
the database runs in WAL mode, so with anything holding it open a copy of the file alone can be empty or
miss recent writes.quantified-self-backup create FILEuses SQLite's online-backup API for a consistent
single-file snapshot, integrity-checks it, owner-only permissions, and never overwrites an existing file.
quantified-self-backup restore FILEvalidates the backup first (integrity, health database, schema not
newer than this release), stages and re-checks a copy, refuses while another program has the database open,
and moves the current database aside as<name>.pre-restore-<time>instead of deleting it. It is a
command-line tool only, not an MCP tool. - Canonical ingestion contract (
ingest.py). The one shape every source adapter's observations
must take: metric, naive local timestamp, value, canonical unit, source, source record id, original
timezone and quality, with a strict validator that normalises known unit spellings (count/minto
bpm), rejects timestamps with a UTC offset, values outside the registry bounds, units that need a
conversion, and unknown fields, and drops repeated record ids. A bad record is rejected on its own
with a reason and the rest of a batch still loads. Nothing uses it yet: adapters are moved onto it in
following changes. - Source coverage matrix.
python -m source_coverage(or--json) measures what the database
actually holds, per importer, source and metric: first and latest day, days covered, gaps, freshness,
the status of the import that produced it, provenance (units, source types, measurement counts) and
how many days analytics resolved to each source. Each entry is classified green, yellow or red with
its reasons listed, and a per-domain view (sleep, HRV, heart rate, activity, body measurements,
workouts) states explicitly that readiness, training load and routes are not modelled. Metrics in
HEALTH_PRIVATE_FIELDSare left out. HRV recorded as SDNN (Apple Health) and as RMSSD (Health
Connect) under one metric is flagged asmixed_statistics. Read-only; no new MCP tool. - Incremental imports with an audit trail. Re-importing a source no longer deletes and rewrites
everything: each (metric, day) is compared with what the importer already stored, unchanged days are left
exactly as they were (ids andimported_atkeep recording when they were first imported), and only new
or changed days are written. Eachimportsrow now recordsrecords_seen,records_added,
records_updated,records_unchangedandrecords_removed(days dropped by--replace), plus the
dataset coverage before and after, andget_import_statusreturns them.quantified-self-init-dbprints
the same counts. Schema migration 9 adds the columns to databases that already have the table. - Import history and data freshness. Every
quantified-self-init-dbrun now
records animportsrow (importer, source file name and SHA-256, status
running/succeeded/failed, rows loaded and skipped, measurements
written, error). Two read-only MCP tools expose it:get_data_statusreports
whether the database isCURRENT,STALE,INCOMPLETE,NO_DATAor
IMPORT_FAILED(latest data date, coverage, gaps, last successful import,
suggested action), andget_import_statuslists recent import runs.
Metrics inHEALTH_PRIVATE_FIELDSare ignored when computing coverage. data_healthon every analytics tool. A single
data-quality state for the window behind the answer:VALID,
VALID_WITH_GAPS,INSUFFICIENT_DATA,STALEorIMPORT_INCOMPLETE, with
every applicable reason, the latest observation and its age, the covered
period, gaps, observation count and the last import. It combines the window's
existing evidence with the import history and dataset status, and is
separate fromclaim(which is unchanged). Results that rest on several
windows (period comparisons, correlations, recent changes, explanations)
report the weakest window and list every window's reasons.- Agent-loop eval (
eval/agent, Phase 1). A real model, the real MCP
server and real tool results in a multi-turn loop, scored per layer (routing,
arguments, execution) with replayable traces, deterministic fixtures defined
relative to today, and a tool coverage matrix built from the live tool list
(python -m eval.agent --coverage --strict). The harness itself is covered
by offline tests that need no API key. Interpretation and final-answer
scoring are not built yet and are reported as not scored.
Changed
- The evidence registry now describes what is actually evaluated —
trend,
window_comparison,correlationandanomalylist only the dimensions that
have evaluators (sample and temporal, plus missingness for the first three),
likebaseline. Previously they also listed robustness, practical,
provenance and measurement_validity, which no evaluator produced, so every
claim resolved those asnot_assessedand could never exceedsuggestive.
Nothing is tolerated to make that go away: those four analyses now carry an
explicitmax_tier: suggestiveceiling whose factor,practical_not_evaluated,
is added toclaim.decision.must_state(a statistically detectable but trivial
effect cannot be calledsupportedwhile practical significance is unchecked).
claim.profile.dimensionsfor those analyses no longer contains the
not_assessedrobustness/practical/provenance entries, and
robustness_not_assessed,practical_not_assessedand
provenance_not_assessedno longer appear inmust_state. A parity test now
fails if a registry dimension has no evaluator or an evaluator's output is not
in the registry.
Fixed
-
A mistyped or unsupported metric is no longer reported as "no data".
read_measurementsand
get_metric_provenanceanswered a name nothing is stored under (for examplesleepinstead of
sleep_hours) with a successful empty result, so a client would tell the user they had no such data.
They now return the same[invalid_metric]errorlog_measurementalready used, listing the supported
metrics and suggesting the closest one. A supported metric with no readings in the window still returns
an empty result, and a metric that has stored readings stays readable even if its aggregation rule was
removed. -
The HTTP transport no longer listens on every network interface. Setting
PORT(the cloud-deploy
mode) used to bind0.0.0.0with no authentication, so anyone who could reach the machine could call
every tool, including the ones that write, clear and export data. It now binds loopback (FastMCP's
default). A deployment that must be reached from outside, such as a container behind a platform
router, now has to opt in withFASTMCP_HOST=0.0.0.0; that is still allowed but logs a warning that
nothing authenticates callers. Stdio use (noPORT) is unchanged. -
read_measurementsandread_workout_sessionsvalidate their inputs. A start/end that was not a
realYYYY-MM-DDdate (last week,2026-3-5), a window ending before it starts, or alimit
outside 1-1000 used to return a successful empty list (or, forlimit=-1and an unparseable
end_date, an unbounded one), which reads as "no data". They now return the same structured
[invalid_range]error asread_health_data. ISO timestamps are still accepted as bounds by
read_measurements; empty strings still mean "not given". -
Re-importing changed days no longer scans the whole metric once per day. Replacing a changed
(metric, day) deleted the importer's old rows with a filter no index could serve, so each changed
day read every row of that metric. The delete now seeks the day through the existing
(metric, timestamp)index and keeps the original filter as an exact check, so the same rows are
deleted. On a 1M-row database with 5% of days changed, the re-import went from about 83s to
about 24s (measured with a throwaway benchmark, not part of the repo). -
A failed import is now visible to
get_data_statusandget_import_statuseven when it fails
while reading the file. Only failures after parsing were recorded inimports, so a truncated or
unreadable export left no trace and the status stayedCURRENT. A read failure is now recorded as a
failed run (statusIMPORT_FAILED, actionrerun_import). The recorded reason is the error class
plus its file position for XML/JSON/CSV/encoding errors and the OS reason for I/O errors; a file the
importer rejects with its own message (e.g. a CSV with no date column) is recorded as rejected, and any
other error by class name only, so a record value or the file's full path never reaches the
model. The importer still raises exactly as before. -
Analytics no longer present a source change as a health change.
daily_metricskeeps one source
per (metric, day), but a baseline, trend, anomaly scan, comparison or correlation read it as one
anonymous series, so a device switch inside a window, a different device in each compared window,
or two devices disagreeing with nosource_priorityall came backdata_health: VALID. They now
report the new statusMIXED_SOURCESwith the reasonssource_changed_within_window,
source_changed_between_windowsorunranked_source_conflict_in_window. A day settled by an
explicitsource_priorityentry is a documented rule and is not flagged. The numbers themselves
are unchanged. -
Wheel packaging and Dockerfile runtime dependencies. The build wheel include list in
pyproject.tomlandCOPYinDockerfilepreviously omittedtools/,db/,data_health.py,
errors.py,privacy.py,schemas.py, andsource_coverage.py, causingModuleNotFoundError
on package installs or container runs. All modules and database schema files are now included. -
GitHub Actions action versions. Updated
.github/workflows/ci.ymlandpublish.ymlfrom
invalid@v7actions to officialactions/checkout@v4andactions/setup-python@v5. -
Imports from different importers are no longer summed together. Day totals an importer wrote
without a source (CSV, Health Connect, and Apple Health's sleep and other daily rows) all fell into one
unattributed group in the daily projection, so importing the same period through two paths doubled
sum metrics such as steps and sleep, and reported the day as a single source. An import now stamps
such rows with the importer's name assource, so the projection resolves between importers with
source_priority(or its documented fallback) andresolutionsays so. Schema migration 10 stamps
existing imported rows the same way and rebuildsdaily_metrics, which corrects days that were
double-counted: those daily values will change. Manual measurements keep no source. -
Apple Health raw observations are now stored in the metric's canonical unit
and validated per reading.daily_metricsis projected frommeasurements,
but the Apple Health adapter wrote raw readings in the device's own unit under
the canonical metric name: a 154 lb weight becameweight_kg = 154and
water_mlwas summed in litres. It also never bounds-checked raw readings, and
a day reported as "Skipping date ..." for an out-of-bounds total was imported
anyway. Pound and litre readings are now converted before storage (the stored
unitiskg/ml), readings in other weight or volume units (stones, fluid
ounces) are skipped and counted instead of being read as kg / ml, out-of-bounds
readings are skipped, and a skipped day imports none of its raw rows. -
NaN / infinity no longer crash an import or reach an aggregate. Apple
Health, Health Connect and CSV imports now skip and report non-finite values
like any other bad record (previouslyint(round(nan))/int(inf)aborted
the whole import with a traceback). A rejected Health Connect reading also no
longer leaves an empty per-day bucket behind.