Feat/global widget enhancements 1

.gitignore

@@ -150,3 +150,13 @@ qa-runs/auth/
 /public/ask-ux-core-dev.js
 /widget/dist/
 /widget/node_modules/
+
+# Personal developer compose files — local-only, never commit.
+docker-compose.dev.yml
+docker-compose.override.yml
+
+# Working / scratch docs — keep out of the repo by default.
+# (Anything tracked under /docs that pre-dates this rule stays tracked;
+#  gitignore only blocks new untracked files. Move docs you want
+#  shared into README/AGENTS/CLAUDE.md or a dedicated published path.)
+/docs/

AGENTS.md

@@ -488,6 +488,19 @@ The `next.config.js` loads env from `.env.{APP_ENV}` (e.g., `.env.local`, `.env.
 
 ---
 
+## Commit Hygiene — never push noise
+
+Before every commit and before every push, audit `git status` and `git diff --cached --stat` and remove anything that doesn't belong in the change set. Specifically:
+
+- **Personal developer files** (`docker-compose.dev.yml`, `docker-compose.override.yml`, local `.env.*`, editor configs, scratch notes) MUST NOT be committed. They live only on the dev machine; the repo has no use for them.
+- **Working / scratch docs** under `/docs` are gitignored. Anything you want shared belongs in `README.md`, `AGENTS.md`, or a CLAUDE.md — not a loose markdown file in `/docs/`.
+- **Build output** (the widget bundle, `.next/`, `dist/`) is gitignored. If it shows up in `git status`, something is wrong with the build script, not the gitignore.
+- **Debug artifacts** (revision constants like `READ_LIB_REVISION = 'v4'`, console.logs left over from a debug session, no-op shim exports kept "for safety") get removed before the PR opens — they age into rot otherwise.
+
+If you find one of these staged or already committed in your branch, drop it with `git rm` (or `git restore --staged`) and add it to `.gitignore` so the next agent can't trip on it. The `.gitignore` is the durable fix; deleting the file alone is not.
+
+---
+
 ## Gotchas
 
 ### Case sensitivity

CLAUDE.md

@@ -0,0 +1,19 @@
+# CLAUDE.md — keepsimple-merged (for Claude Code agents)
+
+This file is loaded by Claude Code at session start. Human-readable agent guidelines live in `AGENTS.md` next to it; this file is the machine-facing version.
+
+## Code search — prefer CodeGraph over Grep
+
+This repo is indexed by **CodeGraph** (MCP server `codegraph`, registered globally). Symbol/structure queries are sub-millisecond there and dramatically cheaper than grep. Reach for it FIRST when you have a name:
+
+- `codegraph_search` — find a symbol by name (kind + location + signature in one shot)
+- `codegraph_callers` / `codegraph_callees` — function-call graph navigation
+- `codegraph_context` — fastest onboarding for "what is this file/feature about?"
+- `codegraph_impact` — blast radius before a rename or refactor
+- `codegraph_files` — what's in a directory + per-file symbol counts
+
+Use **Grep / Glob only when** the query is a *concept* with no symbol name ("where do we handle the Cohere fallback?"), or when a CodeGraph query returned nothing. Index lags writes ~500ms; if you just edited a file, give it a turn before re-querying.
+
+## Everything else
+
+See `AGENTS.md` for repo conventions, build/test commands, and contribution rules.

docs/article-drafts/copilot-not-search.md

@@ -0,0 +1,71 @@
+# I set out to build a search bar, I ended up with a copilot.
+
+This one's more technical than my usual. It starts with something I've wanted for years — a global search on keepsimple.io that could help visitors navigate the hundreds of pieces we've put on the project.
+
+What stopped me, for years, was the same thing that probably stops you from using most website searches — they suck. You use one once, the result is mediocre, and you forget the search bar exists. I didn't want dead weight on the project. A project like keepsimple, full of unique work, deserves a search built for it — one that's actually friendly.
+
+The other constraint was cost. The project is self-funded and I can't stretch the budget. If I mess up with LLM or service APIs, thousands of users — most of them spending more than three minutes per session — will bankrupt me in a week.
+
+So four criteria: high fidelity, friendly, dirt-cheap, unique.
+
+## High fidelity
+
+Tackled this first. Ruled out a pile of mediocre approaches and landed on LightRAG — an open-source project out of universities in Beijing and Hong Kong, and frankly an epic piece of engineering.
+
+Configuration at this point: visitor writes a message → LightRAG takes the question, uses gpt-4o-mini to expand it and match against the graph + vectors, returns ranked snippets with source URLs.
+
+## Friendly
+
+For this one I stepped away from the classic search bar entirely. Built a widget instead, called it Copilot, started prompting. The widget is just the frame — the actual friendliness has to come from the writer.
+
+Configuration at this point: Copilot takes the snippets and asks Claude Sonnet 4.6 to draft the answer in our voice — with OpenAI's gpt-4.1 as fallback.
+
+Total models in play: gpt-4o-mini (LightRAG's brain, index + query), text-embedding-3-small (vectors), Sonnet 4.6 (the writer). No Opus.
+
+## Dirt-cheap
+
+That model list above is the entire paid surface. LightRAG runs locally, the embeddings cost almost nothing, the retrieval pass doesn't bill at all — the only place the meter runs is the final Sonnet call. Cents a day across hundreds of pages and three languages.
+
+## Unique
+
+At this point I had a high-fidelity search that talked back like a regular LLM chatbot. Which was fine, except the "regular" part went directly against my unique criterion. So I made two more moves.
+
+The first move was a second indexing pass. The first pass had fed Copilot the article-style content. The second added a structured snapshot of every landing page on the site — every heading, every CTA, every paragraph, with a short text anchor for each. That gave Copilot two new powers at once. It now knows where the visitor is on the site. And when it recommends something that's also linked on the current page, it can light up the exact element — using the anchors stored on the server plus a live read of the page in the visitor's browser. Headings and CTAs are just text, so the indexing was trivial; the navigation feel is what changed. Copilot now gently nudges visitors toward the exact part of the page they should be clicking.
+
+The second move sits on top of LightRAG, not inside it. I wanted theme-based clusters across keepsimple — small orbits of related material — and a rule that, once we know what a visitor is reading, keeps their suggestions inside that orbit. So between LightRAG's snippets and the writer call, I added a thin server-side step that re-weights what comes back based on the visitor's page. A UX Core reader gets more UX Core. An AI Atlas reader gets more AI. Same logic as a social feed weighted toward your interests: when we know where you are, we keep you there.
+
+These two moves gave Copilot real character, specific to this site, at no extra cost.
+
+That's how I ended up building a thingy that closed my gestalt — a search bar that's none of the things I hated about search bars, and one I actually use.
+
+Next: the orbit logic up close — how the cluster weights actually move, and what makes a cluster a cluster.
+
+Stay safe. Learn and grow with us. Thanks.
+
+Wolf Alexanyan, Armenia, May 2026.
+
+```
+Visitor question + current page
+        ↓
+LightRAG  (gpt-4o-mini)
+  expands the question
+  matches against graph + vectors
+        ↓
+Ranked snippets + source URLs
+        ↓
+Orbit reweighting
+  reorders snippets and pointer cards
+  toward the visitor's current page
+        ↓
+Copilot widget server
+  assembles the prompt
+  attaches page identity + history
+        ↓
+Sonnet 4.6  (gpt-4.1 fallback)
+  drafts the reply in our voice
+  picks 2-3 pointer cards
+        ↓
+Reply + cards + on-page highlights
+        ↓
+Back to visitor
+```

docs/copilot-analytics-spec.md

@@ -0,0 +1,137 @@
+# Copilot analytics — Postgres spec (copilot-events service)
+
+Goal: capture every Copilot session AND every visitor movement end-to-end so we can read transcripts, see who went where, watch dwell-time per page, and catch the anonymous → registered moment. Stored in the **copilot-events** sibling service (Postgres 16, sits next door, HTTP ingest at `COPILOT_EVENTS_URL`). KeepSimple side ships zero Postgres dependencies — it's a thin HTTP client.
+
+This supersedes the deleted `copilot-analytics-strapi-spec.md`. Strapi was wrong for this: at our user base the `copilot-turn` collection would balloon past 100k rows per week, the admin panel would become unreadable, and we'd only have Q&A — no nav, no clicks, no dwell.
+
+---
+
+## 1. Ingest endpoint
+
+The copilot-events service exposes a single ingest endpoint that upserts the session row and appends an event row in one shot.
+
+```
+POST /track
+Authorization: Bearer ${COPILOT_EVENTS_WRITE_TOKEN}
+Content-Type: application/json
+
+{
+  "sid":       string  (required, browser-side session id)
+  "threadId":  string  (required, bumped on every CLEAR)
+  "kind":      string  (required, see event taxonomy below)
+  "env":       string  (required, dev | staging | prod)
+  "ts":        string  (optional, ISO timestamp; defaults to server now())
+  "lang":      string  (optional, en | ru | hy)
+  "pageUrl":   string  (optional, max 500 chars)
+  "pageTitle": string  (optional, max 300 chars)
+  "userAgent": string  (optional, max 500 chars)
+  "firstUrl":  string  (optional, max 500 chars)
+  "payload":   object  (optional, event-specific fields)
+}
+
+→ 204 No Content       on success
+→ 400 {error}          on missing required fields
+→ 401 {error}          on bad token
+→ 500 {error}          on DB insert failure
+```
+
+The service auto-creates the session row on first sighting (reads `lang`, `userAgent`, `firstUrl` from the first event), increments `event_count` on every subsequent event, and has a special case for `kind=auth` with `payload.user` — that one stamps `linked_user` + `linked_at` on the session row.
+
+Read endpoints (`GET /sessions`, `GET /sessions/{sid}/events`) are token-gated on a separate `COPILOT_EVENTS_READ_TOKEN` — not used by the KeepSimple side.
+
+---
+
+## 2. Event taxonomy
+
+| kind             | Fires when                                   | Carries in `payload`                                         |
+| ---------------- | -------------------------------------------- | ------------------------------------------------------------ |
+| `session_start`  | First touch from a new sid                   | —                                                            |
+| `question`       | Visitor sends a Copilot message              | `query` (PII-scrubbed)                                       |
+| `answer`         | Server finishes building the bot reply       | `answer`, `cardsShown`, `mode`                               |
+| `clear`          | Visitor hits CLEAR (rotates thread)          | —                                                            |
+| `card_click`     | Visitor clicks a Copilot card                | `cardClicked: {title, url, tier}`                            |
+| `nav`            | Widget-visible nav chip (internal nav)       | —                                                            |
+| `page_view`      | Every entry into a page (mount + URL change) | —                                                            |
+| `dwell`          | Every exit from a page (in-app + unload)     | `dwellMs`, `pageUrl`, `pageTitle`, `sealed` (true on unload) |
+| `outbound_click` | Click on an anchor to a different origin     | `href`, `anchorText`, `target`                               |
+| `auth`           | NextAuth session detected on this sid        | `user` (email or sub)                                        |
+
+New kinds are forward-compatible — the service accepts any string and stores the rest in JSONB `payload`. No schema migration needed when we add more.
+
+---
+
+## 3. DB schema (mirrored from copilot-events init.sql)
+
+```
+sessions
+  session_id     TEXT PRIMARY KEY
+  env            TEXT NOT NULL
+  lang           TEXT
+  user_agent     TEXT
+  first_url      TEXT
+  started_at     TIMESTAMPTZ NOT NULL
+  last_seen_at   TIMESTAMPTZ NOT NULL
+  linked_user    TEXT
+  linked_at      TIMESTAMPTZ
+  thread_count   INT NOT NULL DEFAULT 1
+  event_count    INT NOT NULL DEFAULT 0
+
+events
+  id            BIGSERIAL PRIMARY KEY
+  session_id    TEXT NOT NULL
+  thread_id     TEXT NOT NULL
+  env           TEXT NOT NULL
+  kind          TEXT NOT NULL
+  ts            TIMESTAMPTZ NOT NULL
+  page_url      TEXT
+  page_title    TEXT
+  payload       JSONB NOT NULL DEFAULT '{}'::jsonb
+  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
+
+indices:
+  events_session_ts_idx  (session_id, ts)
+  events_kind_ts_idx     (kind, ts)
+  events_env_ts_idx      (env, ts)
+  events_payload_gin     USING GIN (payload)
+  sessions_env_started_idx    (env, started_at DESC)
+  sessions_linked_user_idx    (linked_user)
+```
+
+---
+
+## 4. KeepSimple-side wiring
+
+- **Writer**: `src/lib/copilotAnalytics.ts`. Exports `ensureSession`, `logTurn`, `markAuthLink`, `bumpThread`, `copilotAnalyticsEnabled`. Every call is fire-and-forget; failures land in `console.warn`, never bubble to the visitor.
+- **Server-side Q&A fan-out**: `src/pages/api/concierge.ts` calls `logTurn({kind:'question'})` + `logTurn({kind:'answer'})` after the response is built.
+- **Widget event endpoint**: `src/pages/api/copilot/event.ts` receives non-Q&A events (`clear`, `card_click`, `nav`, `page_view`, `dwell`, `outbound_click`, `auth_probe`) from the widget. Reads the `aux_sid` cookie, does the NextAuth-detection / `markAuthLink` dance, then dispatches to `logTurn` / `bumpThread`.
+- **Widget emitter**: `widget/src/api.ts` → `postCopilotEvent(...)`. Uses `navigator.sendBeacon` so card_click and dwell-on-unload survive page navigation.
+- **Page-movement capture**: `widget/src/AskUxCore.tsx` nav `useEffect` fires `page_view` on every page entry, `dwell` on every page exit (in-app or unload), and `outbound_click` on any anchor whose href crosses origin.
+
+---
+
+## 5. Environment variables
+
+| Var                          | Where  | Notes                                                                                          |
+| ---------------------------- | ------ | ---------------------------------------------------------------------------------------------- |
+| `COPILOT_EVENTS_URL`         | server | DEV: `http://127.0.0.1:5046`. Staging + prod: `https://copilot-events.administration.ae`.      |
+| `COPILOT_EVENTS_WRITE_TOKEN` | server | Bearer token for `POST /track`. Same token value across envs (set per-host). Inert when unset. |
+
+Both are SERVER-ONLY — never `NEXT_PUBLIC_*`. The widget never talks to the copilot-events service directly; it always goes through `/api/copilot/event` so the token stays on the server.
+
+Local dev without the sibling container: leave both unset. The writer becomes a no-op and the rest of the widget works as normal.
+
+---
+
+## 6. Reading the data
+
+For now: query Postgres directly (or hit `GET /sessions` and `GET /sessions/{sid}/events` with the read token). A dedicated dashboard / admin UI is a future epic, not v1.
+
+---
+
+## 7. Hard guarantees
+
+- The KeepSimple repo has **zero** Postgres dependencies (no `pg`, no `prisma`, no `DATABASE_URL`). The DB lives in the sibling container.
+- Neither endpoint is queried at Next.js build time, so a copilot-events outage cannot break a deploy.
+- All writes are fire-and-forget — a 5xx, a missing token, or a timeout NEVER blocks the visitor's reply.
+- PII scrub runs before any free-text field hits the wire (`scrubPii` in `src/lib/copilotSafety.ts`).
+- The widget never sees the write-token. It always proxies through `/api/copilot/event`.