Architecture

Architecture

Stack

• Next.js 14 (App Router, TypeScript) — UI and the JSON API in one app.
• Postgres 16 + Prisma — storage.
• Playwright (Chromium) — the scraper, run in-process inside the app container.
• Recharts + Tailwind — charts and styling.
• Built on the official mcr.microsoft.com/playwright image so Chromium + its OS deps are present at runtime. poppler-utils (pdftotext) is added for PDF parsing.

Two containers: app (Next.js + scraper + scheduler) and ngrid_postgres.

Weather history is fetched from Open-Meteo (geocoding + archive APIs); the encrypted credential store uses Node's built-in crypto (AES-256-GCM, scrypt-derived key). No paid keys.

Data flow

Browser ─▶ Next.js (UI + /api/*) ─▶ Postgres
                │
                ├─ scraper (Playwright)  ─▶ myaccount.nationalgrid.com   (login, GraphQL, PDFs)
                │       └─ parsePdf (pdftotext) ─▶ per-fuel supply/delivery + current charges
                └─ scheduler (tickOnce) ─▶ predicts next bill, triggers scrapes

The scheduler is not an in-process cron daemon. node-cron broke the Next edge build and the instrumentation hook never fired reliably, so a tiny curl loop in docker-entrypoint.sh hits POST /api/cron/tick hourly (key-guarded via CRON_KEY), and tickOnce() decides whether a scrape is actually due.

Key modules (app/src)

Path	Responsibility
lib/ngrid/auth.ts	Azure AD B2C login; session reuse via Playwright storageState; MFA detection.
lib/ngrid/collect.ts	One browser run: intercept-and-widen every dataset (bills, usage, costs, weather, account) + download new PDFs.
lib/ngrid/parsePdf.ts	Pure parser: turn pdftotext -layout output into per-fuel supply/delivery, usage, current charges, balance forward, amount due.
lib/ngrid/persist.ts	Idempotent upsert of a scrape result into Postgres.
lib/ngrid/verify.ts	Cross-validate stored/API numbers against a fresh parse of each bill PDF.
lib/ngrid/run.ts	Scrape orchestrator: run record, concurrency guard, schedule update, fires notifications.
lib/ngrid/{accounts,bootstrap,firstRun,loginStatus}.ts	Multi-account login store: discovery, env→encrypted-store cutover, first-run setup, needs_reauth status.
lib/ngrid/{preflight,preflightState,secretKey}.ts	Interactive MFA/OTP login (preflight + OTP step) and the auto-generated/persisted secret key.
lib/ngrid/progress.ts	Live scrape-progress phases surfaced to the UI banner.
lib/crypto.ts	Pure AES-256-GCM seal/open for NgLogin passwords (scrypt-derived key).
lib/scheduler.ts	tickOnce() — run a SCHEDULED scrape if any account is due.
lib/prediction.ts	Pure next-bill prediction, the near-the-bill window/back-off cadence, and the next-bill cost estimate.
lib/series.ts	Pure monthly aggregation + rate math (deriveMonthlySeries, trailing12AllIn); folds in degree-days.
lib/weather/*	Pure degree-days (degreeDays.ts); Open-Meteo geocode/archive + monthly rollup + sync.ts.
lib/{csv,archive}.ts	Pure CSV serialization and the zip/tgz bill-PDF bundler.
lib/{notify,notifyFormat}.ts	New-bill notifications: pure message/dedupe/channel-resolution + the webhook/ntfy/SMTP dispatcher.
lib/{range,accountSwitcher}.ts	Month/year range + presets; selected-account resolution.
lib/queries.ts	Thin DB read layer that feeds series.ts.
lib/chartSpec.ts	Declarative chart definitions (series, axes, colors).
lib/prefs.tsx	Client display prefs (localStorage + React context).
app/api/*	overview, series, bills, bills/[date]/pdf, accounts, refresh(+[id]), runs, settings, verify, cron/tick, export(+/pdfs), ng-logins(+/[id]/reauth, /preflight/[id](+/otp)).
components/*	Dashboard, ConfigurableChart, SettingsView, RefreshButton, Modal, AccountSwitcher, NgLoginsSection, MonthRangePicker, RangeControl, ScrapeProgress.
prisma/schema.prisma	Data model.

Data model

• NgLogin — a saved National Grid login: username + the password sealed as ciphertext/iv/authTag (AES-256-GCM), plus status/lastVerifiedAt. One login fans out to several Account rows. The key never lands here — see Accounts and Login.
• Account — accountNumber, region, companyCode, fuelTypes, address, latitude/longitude (geocoded once for weather), and a nullable loginId (null for accounts bootstrapped from env creds). region/company drive portability.
• Bill — statementDate, currentCharges (this period's energy cost, from the PDF — use this for analysis), totalDueAmount (statement Amount Due from the API — may include carryover), pdfPath. Unique (accountId, statementDate).
• Usage — kWh / therms per periodYearMonth.
• Cost — per fuel, kind ∈ SUPPLY | DELIVERY per month (both parsed directly from the PDF; supply and delivery are stored as separate rows). (The Prisma schema comment still says BILL_AMOUNT — that's stale; the code stores DELIVERY.)
• Weather — monthly avg temperature per region, with a source (open-meteo primary, ng fallback). WeatherDaily — per-account daily temps from Open-Meteo's archive (feeds HDD/CDD). See Weather and Degree-Days.
• ScrapeRun — audit trail / job status. ScheduleState — predicted next bill + next check.
• AppSetting — runtime key/value (e.g. schedulerEnabled, the notification watermark).

Rates are computed, not stored (series.ts): supply rate = supply ÷ usage; all-in = (supply+delivery) ÷ usage; the headline cards use a trailing-12-month all-in average.

Design choices worth preserving

• Pure functions for everything numeric → unit-testable (see Testing). This now includes degree-days, the next-bill estimate, CSV/archive serialization, and notification formatting.
• PDF is authoritative → Data Accuracy.
• Schema-as-source-of-truth via prisma db push at startup (no migration files). Because the push is --accept-data-loss with no rollback, a pre-upgrade backup + a CI migration-safety gate protect production data → Backups and Migration-Safety.
• Secrets never hit the DB or git. Passwords are AES-256-GCM ciphertext in NgLogin; the key lives only in env or the root-only session volume → Accounts and Login.
• Region/company auto-detected from the account response → no hardcoding, works across regions.