Lab Insights

Lab (Insights)

The snapshot and recall deliver everything the brain knows. Lab extends that to what the outside world knows — the live product and business metrics that sit in analytics APIs, billing systems, and spreadsheets — without turning dreamcontext into a BI tool. The unit is an insight: a curated metric with a data source, a render type, and a refresh policy. Lab captures what to fetch and how to show it, fetches on demand, caches the rolled-up result in the brain, and surfaces it to agents and the dashboard. It deliberately mirrors the objectives subsystem — markdown-first storage, a pure store, a sync engine, CLI and HTTP routes over one engine — because that pattern is already load-bearing.

Storage

Each insight is a manifest at _dream_context/lab/insights/<slug>.md: frontmatter config (title, render, source, refresh.ttl_minutes, typed tweaks[], an optional binding, credentials_used) plus a ## Meaning prose block that is recall-indexed — so an insight is discoverable by meaning as an insight corpus type, not by slug. The fetched result is cached separately at _dream_context/lab/cache/<slug>.json — the post-rollup series, latest, fetchedAt, granularity, error/errorAt, and a scriptHash. Both the manifest and the cache sync in the brain repo; only _dream_context/lab/credentials.json (mode 0600, written exclusively through the gitignore-first CLI) is excluded. Manifest parsers are lenient — a malformed insight reads as null rather than throwing — so a bad file never crashes the snapshot.

Sync, rollup, and binding

syncInsight resolves the manifest's tweaks into a concrete window, runs the adapter, caps and rolls up the series, writes the cache, and optionally writes a bound objective's Key Result. Two adapters ship: a generic HTTP adapter (endpoint / headers / body templates that expand {{tweak:…}} and {{cred:…}}, a JSON-path extract, GET or POST) and a custom script adapter (a .mjs default async function under lab/scripts/). Ready-made PostHog/Sheets adapters are a v2 item — the generic layer is expressive enough that most sources fit without bespoke code.

Rollup is what keeps Lab "insights, not raw dumps": MAX_POINTS = 62, and capSeries coarsens daily → weekly → monthly until a series is under the cap, with granularity derived from the resolved span (over 180 days monthly, 45–180 weekly, 45 or fewer daily). A TTL guard skips a fresh insight unless --force, and the skip is reported rather than silent. On adapter failure the cache keeps its prior series, records the (redacted) error, and syncAll aggregates a non-empty failed[] so the CLI exits non-zero — there is never a silent half-sync.

Binding is how a metric becomes measured progress. An insight is connected to an objective's Key Result with dreamcontext lab bind <insight> <objective> (or by searching insights inside the dashboard's objective create/detail dialogs) — the bind validates the objective exists, enforces one feeder per objective (a previously bound insight is unbound and reported), and immediately seeds metric.current from the cached latest. From then on, whenever the manifest carries binding.objective with value: latest | series:<name>, a successful sync writes the objective's metric.current through the same updateObjectiveMetric the roadmap uses — but only when the latest value is finite and the metric exists; an empty series leaves the number untouched and warns. The roadmap forecast cascade then reflects a number that was measured, not asserted. Sleep deliberately does not run lab sync (credential exposure, latency, non-determinism); a bound insight feeds its Key Result through its own sync instead.

Three security nets

Credentials are handled structurally, not by convention. writeCredential ensures the canonical brain .gitignore exists first (writing the full template if missing, never a two-line stub), layers the lab credential entries, aborts without writing on any gitignore failure, and chmods the file to 0600; doctor fails loudly if credentials.json ever exists uncovered by gitignore. Every error, log line, and cache string is built from a redacted resolution ({{cred:…}} → ***) with a final redactSecrets net, so a non-2xx body or a thrown script error can never echo a real credential. And because a .mjs script is the first executable artifact the brain repo carries — it runs in-process with your credentials, the same trust level as the repo — Lab records a scriptHash on each successful run and prints a loud change tripwire before executing a script that changed since last time.

The CLI (lab sync | list | show | create | tweak | bind | credentials) and the /api/lab* routes call the same engine, and no route ever returns a credential value. The dashboard's Lab page renders insights grouped by category with hand-rolled SVG (number / line / pie / raw — no chart library), per-insight and sync-all refresh with a loud failed[] summary, and generic tweak editing driven by the manifest's TweakDecl[].

---

Part of the dreamcontext deep dive — Home · README