Skip to content

feat(agents): proactive "render UI by default" directive for MAF + Copilot agents (genUI Phase 2) - #190

Merged
vjvarada merged 8 commits into
mainfrom
claude/chat-agent-framework-review-qvimev
Jul 23, 2026
Merged

feat(agents): proactive "render UI by default" directive for MAF + Copilot agents (genUI Phase 2)#190
vjvarada merged 8 commits into
mainfrom
claude/chat-agent-framework-review-qvimev

Conversation

@vjvarada

Copy link
Copy Markdown
Contributor

Why

Agents can create interactive generative UI, but they weren't reaching for it — and one whole agent type couldn't be reached by the guidance at all. This closes both gaps. The review confirmed the emit path and frontend render are fully runtime-agnostic (one shared queue.put({CUSTOM, generative_ui}) via resolve_run_queue(session_id), rendered on payload alone), so nothing was wrong with UI creation — the problem was entirely on the prompting side.

Two root causes

  1. No answer-time nudge. The "reach for it eagerly" guidance lived only in the emit_generative_ui tool docstring (a weak signal the model consults after it decides to look at tools) and, at system-prompt level, buried mid-addendum among ~15 other tools. Nothing prompted the model, at answer time, to convert a data / status / comparison / choice reply into UI.
  2. Native MAF agents saw none of it. _build_injected_tools_addendum (which carries the genUI guidance and design.md) is applied only on the GitHub-Copilot _tools branch of _inject_agent_tools. Native MAF agents — e.g. email-assistant, which explicitly scopes in emit_generative_ui — got the tool + its docstring + at most the delegation registry. The agent type most set up to render UI had the weakest guidance.

Fix — _ui_first_directive

A static, byte-stable (KV-cache safe), tool-gated "Rich UI by default" directive delivered to both runtimes:

  • Template-first ordering is the token-thrift + on-brand lever: it names the 11 templates as the cheapest, on-brand-by-construction path (you supply data only), the component tree next, and custom HTML as the costly last resort (with the --cc-* CSS-token hook for when it's unavoidable). This directly answers "don't burn tokens on custom generation" and "use the proper design language, fonts, themes, icons."
  • Placement: leads the Copilot addendum (full + compact) instead of being buried among tool specs.
  • Parity fix: appended to native-MAF agents' instructions — both the default_options and agent.tools shapes — marker-guarded for idempotency, with a compact variant for sub-agents. MAF and Copilot agents now get the identical answer-time rule.
  • Pick/set flows steered to formCard / optionPicker + "hitl":true.

Design-language scope note: the full 16 KB design.md is intentionally not dumped into every MAF prompt — templates are on-brand without it, and that's the path the directive steers to. Injecting design.md for MAF agents that lean on custom-HTML reports is left as an optional follow-up (documented in the spec); the --cc-* hook covers the common custom-HTML case.

Tests

  • tests/unit/test_genui_proactive_directive.py (7 tests): native-MAF instructions injection for both MAF shapes, idempotency (marker guard), compact-vs-full variant selection, byte-stability, template-first ordering.
  • evals/trajectories/test_tool_addendum_drift_trajectory.py: new assertions that the directive leads both addenda, templates precede the custom-HTML escape hatch, and the directive is gated on the tool being present. Also refreshed the drift set to cover the coding-skill tools (run_script/code_task/list_integrations) that were injected but untracked.
  • Full suite green: 1678 unit tests, 120 trajectory evals, CI-blocking lint clean.
  • Spec: ai-company-brain/specs/generative_ui_2.md §7.

🤖 Generated with Claude Code

https://claude.ai/code/session_01K9rzBFbJiEtwSyq8u7sWWt


Generated by Claude Code

claude added 8 commits July 23, 2026 05:05
BreathingCharacter default sprite scale 1.35x -> 1.8x of its box (the frames
carry heavy transparent padding, so the character reads much bigger without
cropping), and boxes now nearly fill their fixed columns: picker cards
52->60px, agents grid tiles 56->62px, agent settings header 76->92px. Card
and column dimensions unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K9rzBFbJiEtwSyq8u7sWWt
…th durable script store

Give every MAF agent the Copilot SDK as a CAPABILITY (not a peer agent),
per the framework review's dual-runtime verdict:

- code_task(task): one-shot, 600s-bounded Copilot coding session in the
  calling agent's own workspace (orchestrator/code_session.py). Manifest-
  first contract: read agent-data/SCRIPTS.md, edit scripts in place under
  agent-data/scripts/, update the manifest. BYOK via the gateway /v1.
- run_script(path, args?): zero-LLM re-execution of a saved .py/.sh
  script — cheap reuse of capability built in previous sessions.

Durability: scripts live under blob-store-backed agent-data/ (Postgres
authoritative), surviving restarts, redeploys, and volume wipes. Both
tools finish with an explicit sweep that mirrors native-written files
(Copilot CLI + script self-writes bypass the write_artifact mirror) into
the store — code_task sweeps even when the session failed.

Execution hygiene: workspace-jailed path resolution (fail closed on
escape), secret-scrubbed subprocess env (allowlist + deny pattern — no
tokens/keys/DB URLs reach script code), wall-clock timeouts, capped
output. Registered open-world in TOOL_ANNOTATIONS; rides the permission
gate like every injected tool.

Wiring: injected into all agent shapes, added to the guaranteed core
tool floor, documented in the full + compact addendum. 24 new unit
tests; spec at ai-company-brain/specs/agent_coding_skill.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K9rzBFbJiEtwSyq8u7sWWt
…ed skill fixes flow through the approval pipeline

The agent's chat workspace IS its persistent git clone, so the coding
skill and the existing self-mutation machinery are one system:

- Coding-session contract now teaches TWO script homes: agent-data/
  scripts/ (workspace scripts, blob-durable, no approval) vs git-tracked
  repo source (skills/*/scripts/, agents.py — the agent's built-in
  skills). Repo skills are edited IN PLACE and committed locally, never
  pushed — the executor's post-run commit scan registers every local
  commit as a pending_commit inbox row and the pre-push hook blocks
  direct pushes, same as failure-driven mutation and loader auto-sync.

- Fail-safe in code_task (_commit_repo_changes): if a session edits
  tracked source and forgets to commit, code_task commits the residue —
  the loader's next _pull_latest stash-drops/hard-resets uncommitted
  tracked changes, so an uncommitted skill fix would be silently
  destroyed. git add -A respects the loader-managed .gitignore, so
  agent-data/, inputs/, outputs/ never leak into a commit.

- Addendum + tool doc tell agents that a misbehaving built-in skill is
  fixed via code_task and goes live after human approval.

- Spec: §7 harmonisation — script taxonomy + the three mutation paths
  (failure-driven, chat-driven, promotion via loader auto-sync) all
  converging on pending_commit → approve → push.

5 new tests (mini-clone fixture): fail-safe commit + clean tree, no
noise commits, non-git workspaces, contract/addendum coverage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K9rzBFbJiEtwSyq8u7sWWt
The sweep compared file mtimes against a time.time() cutoff, but kernel
file timestamps come from a coarse clock that can lag time.time() by a
few ms — a file written immediately after the cutoff capture could carry
an mtime just BEFORE it and be silently skipped (a very fast code_task
write would then miss the blob-store mirror; surfaced as a CI-only flake
in test_code_task_failure_still_sweeps). Slacken the cutoff by 2s;
re-mirroring a file modified moments before the run is harmless because
the mirror is an idempotent write-through.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K9rzBFbJiEtwSyq8u7sWWt
…st_integrations discoverability

Scripts often exist precisely to call an external service the platform
holds credentials for — but the scrubbed subprocess env blocked even the
integrations an agent legitimately declared, while blanket pass-through
would leak the whole secret surface. Apply the platform's existing rule
to the coding skill: an agent's scripts get EXACTLY the integrations
that agent declared in config.json — nothing else.

- acb_skills.integrations.FIELD_TO_ENV: canonical service→env-var
  mapping, now the single source of truth shared by the executor's
  run-scoped env injection and the script env (also fixes google-maps,
  which was missing from the executor's private copy and therefore
  never env-injected). env_var_names() helper.
- Executor publishes each run's resolved integration names (+ warnings)
  into the tool context; sub-agents get their own declared set, not the
  parent's (saved/restored around delegation).
- code_tools._script_env grants the base allowlist PLUS exactly the
  declared services' env vars; gateway master key, DB URL, and
  undeclared integrations stay scrubbed (proven by live subprocess
  test).
- New injected core-floor tool list_integrations(): resolved services +
  env-var NAMES scripts may read + declared-but-unavailable reasons —
  never credential values.
- code_task session prompt names the available vars (names only); the
  harness forbids hard-coding, printing, or persisting credential
  values. Addendum documents the workflow.
- Spec §8; 9 new tests (34 total in test_code_tools.py).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K9rzBFbJiEtwSyq8u7sWWt
…times (genUI Phase 2)

Agents could create generative UI but weren't reaching for it. Two root
causes, both on the prompting side (emit path + frontend render are
already runtime-agnostic):

1. The "reach for it eagerly" guidance lived only in the tool docstring
   (weak signal) and, at system-prompt level, buried mid-addendum. No
   answer-time nudge to convert a data/status/comparison/choice reply
   into UI.
2. Native MAF agents got NONE of it: _build_injected_tools_addendum
   (which carries the genUI guidance + design.md) is applied only on the
   Copilot _tools branch. email-assistant explicitly scopes in
   emit_generative_ui yet had zero system-prompt guidance to use it.

Fix — _ui_first_directive: a static, byte-stable, tool-gated directive
delivered to BOTH runtimes:
- Template-first ordering (11 named templates → tree → custom HTML last)
  is the token-thrift + on-brand-by-construction lever; custom HTML
  carries the --cc-* token hook for the rare escape-hatch case.
- Leads the Copilot addendum (full + compact) instead of being buried.
- Parity fix: appended to native-MAF instructions (default_options and
  agent.tools shapes), marker-guarded for idempotency, compact variant
  for sub-agents.
- Pick/set steered to formCard/optionPicker + "hitl":true.

Also refreshes the addendum-drift eval to cover the coding-skill tools
(run_script/code_task/list_integrations) that were injected-but-untracked.

Tests: test_genui_proactive_directive.py (MAF instructions injection,
idempotency, compact/full, byte-stability) + drift-eval assertions
(directive leads both addenda, template-before-html, tool-gated). Spec
generative_ui_2.md §7. Full suite green (1678 unit, 120 trajectory).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K9rzBFbJiEtwSyq8u7sWWt
@vjvarada
vjvarada merged commit 3e17390 into main Jul 23, 2026
6 checks passed
vjvarada pushed a commit that referenced this pull request Jul 23, 2026
…t (Granola pattern)

The single highest-leverage note-taker UX idea (research §3, spec §4
Tier-1 item 3): the user's attention during the meeting steers a far
better summary than the transcript alone.

- migration 97_note_taker_scratch.sql: meeting.scratch_notes TEXT (the
  user's rough input notes, distinct from the generated meeting_note
  output). Validated on a clean 00->97 chain + idempotency.
- summaries.py: generation reads scratch_notes and threads it into the
  prompt via _scratch_block as EMPHASIS SIGNALS — expand the flagged
  topics from the transcript, fix shorthand, ensure each is addressed,
  but ground every fact in the transcript and never invent from the
  notes. Works for both single-pass and map-reduce.
- meetings.py: PUT /meetings/{id}/scratch; scratch_notes surfaces on the
  meeting detail.
- UI: a 'Jot notes as you go' textarea on the recording screen
  (debounced autosave, flushed on stop) and a 'Your notes' editor on the
  meeting detail (save on blur) — both feed the same field.

Rebased onto current main (picked up #189/#190). 3 new tests for the
scratch block (empty no-op, labelling + grounding guardrail + bound);
full suite 1694 passed; 19 /notes routes; frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
vjvarada pushed a commit that referenced this pull request Jul 23, 2026
…t (Granola pattern)

The single highest-leverage note-taker UX idea (research §3, spec §4
Tier-1 item 3): the user's attention during the meeting steers a far
better summary than the transcript alone.

- migration 97_note_taker_scratch.sql: meeting.scratch_notes TEXT (the
  user's rough input notes, distinct from the generated meeting_note
  output). Validated on a clean 00->97 chain + idempotency.
- summaries.py: generation reads scratch_notes and threads it into the
  prompt via _scratch_block as EMPHASIS SIGNALS — expand the flagged
  topics from the transcript, fix shorthand, ensure each is addressed,
  but ground every fact in the transcript and never invent from the
  notes. Works for both single-pass and map-reduce.
- meetings.py: PUT /meetings/{id}/scratch; scratch_notes surfaces on the
  meeting detail.
- UI: a 'Jot notes as you go' textarea on the recording screen
  (debounced autosave, flushed on stop) and a 'Your notes' editor on the
  meeting detail (save on blur) — both feed the same field.

Rebased onto current main (picked up #189/#190). 3 new tests for the
scratch block (empty no-op, labelling + grounding guardrail + bound);
full suite 1694 passed; 19 /notes routes; frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
vjvarada pushed a commit that referenced this pull request Jul 23, 2026
…t (Granola pattern)

The single highest-leverage note-taker UX idea (research §3, spec §4
Tier-1 item 3): the user's attention during the meeting steers a far
better summary than the transcript alone.

- migration 97_note_taker_scratch.sql: meeting.scratch_notes TEXT (the
  user's rough input notes, distinct from the generated meeting_note
  output). Validated on a clean 00->97 chain + idempotency.
- summaries.py: generation reads scratch_notes and threads it into the
  prompt via _scratch_block as EMPHASIS SIGNALS — expand the flagged
  topics from the transcript, fix shorthand, ensure each is addressed,
  but ground every fact in the transcript and never invent from the
  notes. Works for both single-pass and map-reduce.
- meetings.py: PUT /meetings/{id}/scratch; scratch_notes surfaces on the
  meeting detail.
- UI: a 'Jot notes as you go' textarea on the recording screen
  (debounced autosave, flushed on stop) and a 'Your notes' editor on the
  meeting detail (save on blur) — both feed the same field.

Rebased onto current main (picked up #189/#190). 3 new tests for the
scratch block (empty no-op, labelling + grounding guardrail + bound);
full suite 1694 passed; 19 /notes routes; frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
vjvarada added a commit that referenced this pull request Jul 23, 2026
… + Deepgram named speakers (#184)

* feat(notes): slice 1 — notes generation, SSE progress, transcript↔notes UI

Turns transcripts into grounded, structured meeting notes (spec §3.5):

- templates.py: meeting-notes template as data (sections + instructions)
  → prompt compiler with the Meetily-derived grounding + anti-injection
  rules; markdown renderer for the canonical notes document.
- summaries.py: notes generation on acb_llm (acompletion_with_fallback,
  strict JSON, transcript pinned as DATA). Single-pass for short
  transcripts, map-reduce for long ones (per-chunk cap is logged, never
  silent). Writes meeting.summary_md/json + meeting_note, and extracts
  draft action_item rows with segment_ids grounding (HITL approve→task is
  slice 2). Endpoints: POST /summarize, GET/PUT /note, GET /actions,
  GET /templates. Env-tunable model roles (NOTES_MEETING_SUMMARY_MODEL).
- pipeline.py: auto-enqueues notes generation after transcription, so one
  upload yields transcript → notes with no second action.
- events.py: per-meeting SSE progress stream (honest per-stage status,
  closes on terminal) + a dedicated Next stream proxy (the catch-all
  JSON proxy can't stream).
- meeting detail: two-pane transcript↔notes (markdown via MarkdownMessage),
  live SSE-driven progress, and a confidence-ranked action-items rail.

9 new unit tests (template compiler, chunking, ref collection, markdown
render); full suite 1647 passed. Gateway registers all /notes routes;
frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8

* feat(notes): in-browser recorder — live mic capture → chunked upload (slice 1)

Completes slice 1's capture loop (R1: record a live conversation in-app):

- recordings.py: chunked live-recording endpoints — POST /recordings/start
  (creates the row + empty file, marks the meeting 'recording'),
  POST /recordings/{id}/chunk?seq= (idempotent, gap-checked append; a
  MediaRecorder session's ordered timeslice blobs concatenate to a valid
  container), POST /recordings/{id}/complete (finalize + enqueue
  transcription, which auto-chains into notes generation).
- recorder.ts: MeetingRecorder — getUserMedia (echo-cancel on), feature-
  detected mime (webm/opus, mp4 for Safari), MediaRecorder timeslice, and a
  sequential retry-with-backoff uploader that keeps unacked blobs queued in
  memory (the offline buffer for wifi blips) and flushes before complete.
  AnalyserNode drives a live level meter.
- session/[id]: the 'studio, not form' recording screen (spec §5.1) —
  consent line, canvas waveform, large mono timer, pause/stop, backlog
  indicator, honest mic-permission errors.
- Library 'Record' button (was disabled) now creates an in_person meeting
  and opens the session; 'Upload' demoted to secondary.
- Notes proxy passes application/octet-stream bodies through byte-exact
  (the chunk uploads), alongside the existing multipart passthrough.

Gateway registers all 12 /notes routes; full suite 1647 passed; frontend
tsc clean on the new files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8

* feat(notes): action-item → task HITL loop-closure (slice 2)

Turns draft action items into real GTD tasks — the Meeting→Task
counterpart of the Email→Task capture (spec §3.9):

- actions.py: POST /notes/actions/{id}/approve inserts a LOCAL gtd_items
  task (the store the task manager owns) with an origin={kind:meeting,
  meeting_id, action_item_id, segment_ids} provenance link, and records
  action_item.resulting_task_id for a two-way link. Idempotent (re-approve
  returns the existing task). /reject marks it rejected;
  /meetings/{id}/actions/approve-all approves every draft ≥ a confidence
  threshold. Every approval writes an audit_event.
- Meeting detail action rail is now interactive triage (spec §5.5):
  Approve → Task / Reject per item, a confidence-ranked list, an
  'Approve all ≥80%' bulk button, a success toast, and created items
  deep-link into /tasks. Read-only states for created/rejected.

gtd_items columns (id/user_id/title/description from mig 48, origin from
mig 65) verified. Gateway registers all 15 /notes routes; full suite
1647 passed; frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8

* feat(notes): attendees + follow-up email loop-closure (slice 2)

Closes the Notes→Email arc (spec §3.9):

- migration 96_note_taker_attendees.sql: meeting.attendees JSONB
  [{name,email}] for external attendees (distinct from attendee_ids org
  person refs) — the recap recipient set. Validated on a clean PG16 chain
  run + idempotency; ORM mirror added.
- meetings.py: attendees surface on the meeting detail; PUT
  /meetings/{id}/attendees replaces the list.
- share.py: POST /meetings/{id}/share/email/draft LLM-drafts a recap
  (subject + body) from the notes (pinned as DATA), recipients from
  attendees; degrades to a raw-notes recap if the LLM is unavailable.
  Draft-only — nothing sends server-side.
- UI: inline attendee chip editor on the meeting detail; a 'Follow-up'
  button opens a HITL compose modal (FollowupEmailModal) that fetches the
  draft + the user's email accounts, lets them edit to/subject/body and
  pick a sending account, and sends via the existing /email/send.

Migration chain OK through 96; all 17 /notes routes register; full suite
1647 passed; frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8

* feat(notes): ask-the-meeting — grounded Q&A over a transcript

The first AI-intelligence feature (spec §4 Tier-1 item 2): ask a question
about a meeting and get a transcript-grounded answer with clickable
citations.

- qa.py: POST /notes/meetings/{id}/ask reuses the summarization grounding
  discipline (transcript pinned as DATA, cite segment numbers, ignore
  embedded instructions). No precomputed embeddings needed — a meeting
  transcript is bounded, so it feeds relevant segments directly; when a
  transcript exceeds the pass budget it keyword-ranks segments (+neighbours
  for context) and flags the answer as partial rather than silently
  truncating. New meeting_qa model role (tier-balanced default).
- AskPanel: a Q&A panel in the meeting detail; each answer's segment
  citations are clickable and scroll the transcript to that moment (with a
  highlight flash) + cue the audio — provenance you can touch (§5.3).

4 new unit tests for the keyword/selection logic (incl. the match-wins-
budget-over-its-own-context case). Full suite 1651 passed; all 18 /notes
routes register; frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8

* refactor(notes): route STT through the LiteLLM plumbing + tier-stt model config

STT was calling Groq/Deepgram/OpenAI via its own httpx clients with its own
key-preference resolution — bypassing the platform's model plumbing that
every other app uses. This makes speech-to-text a configured model like
everything else:

- New tier-stt tier: added to infra/litellm/config.yaml (default
  groq/whisper-large-v3-turbo), acb_llm _TIER_ALIAS_MAP/_TIER_DEFAULTS, and
  settings _TIER_LABELS — so it resolves through the same _TIER_MODEL
  machinery as the chat tiers and is editable in Settings → Models (the
  frontend already declared an 'stt' tier). set_tier_model now accepts it.
- acb_stt.LiteLLMSTT: resolves the model via resolve_underlying_model(
  'tier-stt'), loads keys with _ensure_keys_loaded, registers via
  ensure_model_registered, transcribes with litellm.atranscription
  (verbose_json → segments/words), and emits usage via _emit_usage for
  observability — mirroring acompletion_with_fallback. Removed the bespoke
  httpx providers + key-preference registry. NOTES_STT_PROVIDER/
  DEEPGRAM_API_KEY env replaced by an optional NOTES_STT_MODEL override.
- pipeline: transcript_source is now the resolved litellm id
  (e.g. groq/whisper-large-v3-turbo), no double prefix.

Caveat: diarization is provider-dependent (whisper-via-litellm returns
none → capture-channel prior); real diarization arrives with the self-host
tier (slice 3), which becomes just another tier-stt model.

Verified: tier-stt resolves to the configured model via the shared tier
init; 8 rewritten acb_stt tests (litellm response normalization + tier
resolution) pass; full suite 1652 passed; gateway imports clean; no
blocking lint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8

* feat(notes): scratch-notes merge — jot during a meeting, AI expands it (Granola pattern)

The single highest-leverage note-taker UX idea (research §3, spec §4
Tier-1 item 3): the user's attention during the meeting steers a far
better summary than the transcript alone.

- migration 97_note_taker_scratch.sql: meeting.scratch_notes TEXT (the
  user's rough input notes, distinct from the generated meeting_note
  output). Validated on a clean 00->97 chain + idempotency.
- summaries.py: generation reads scratch_notes and threads it into the
  prompt via _scratch_block as EMPHASIS SIGNALS — expand the flagged
  topics from the transcript, fix shorthand, ensure each is addressed,
  but ground every fact in the transcript and never invent from the
  notes. Works for both single-pass and map-reduce.
- meetings.py: PUT /meetings/{id}/scratch; scratch_notes surfaces on the
  meeting detail.
- UI: a 'Jot notes as you go' textarea on the recording screen
  (debounced autosave, flushed on stop) and a 'Your notes' editor on the
  meeting detail (save on blur) — both feed the same field.

Rebased onto current main (picked up #189/#190). 3 new tests for the
scratch block (empty no-op, labelling + grounding guardrail + bound);
full suite 1694 passed; 19 /notes routes; frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8

* feat(notes): org glossary — teach transcription your jargon (§4 Tier-1 item 6)

A per-user vocabulary that biases transcription toward the right spellings
of product/people/customer names and acronyms — fixing the most common
source of transcript errors before it propagates into notes, action items,
and search.

- migration 98_note_taker_glossary.sql: notes_glossary(user_id, term)
  with case-folded per-user uniqueness. Validated on a clean 00->98 chain
  + idempotency; ORM model added.
- glossary.py: GET/POST/DELETE /notes/glossary CRUD (add is idempotent per
  case-folded term) + format_glossary_prompt/glossary_prompt — a bounded
  vocabulary-hint sentence (term/char caps) fed to the STT prompt.
- pipeline.py: loads the meeting owner's glossary and passes it as
  SttOptions(prompt=...) at transcription time; best-effort so it never
  blocks transcription.
- UI: a GlossaryModal (chip add/remove) opened from a Glossary button in
  the /notes library header.

4 new tests for the prompt builder (empty no-op, vocabulary-hint framing,
trimming, length bound). Full suite 1698 passed; 21 /notes routes;
frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8

* feat(notes): selectable STT model + Deepgram named speakers (Settings → Models)

Make the Note Taker's speech-to-text model user-selectable and add named
speakers (diarization) via a paid Deepgram model — all routed through the
existing LiteLLM tier plumbing. Self-hosted STT (slice 3) is deferred.

Settings → Models
- The Tiers tab gains a dedicated "Speech-to-text" section: its own editable
  picker for the `tier-stt` model, filtered to transcription-capable models
  (chat pickers exclude them, and vice versa). STT is no longer a "Coming soon"
  card. Fixes the tier→metadata lookup (tier-fast → fast) so labels/icons render.
- New `ModelInfo.transcription` capability flag drives the split; Deepgram gets
  a provider tile, setup guide, colour/icon, and a "names speakers" badge.

Model catalogue (settings.py + acb_llm.model_limits)
- Add whisper models to Groq/OpenAI, gpt-4o-transcribe, and a Deepgram provider
  (nova-3/nova-2, DEEPGRAM_API_KEY). All tagged transcription:True, audio:True.
- `_is_transcription_model` heuristic flags un-catalogued whisper/nova variants
  a live provider fetch surfaces, keeping them out of the chat pickers.
- `_provider_from_model` recognises deepgram/*; key_store loads DEEPGRAM_API_KEY.

Diarization plumbing (acb_stt)
- `transcribe()` branches params per family: Deepgram gets native
  diarize/punctuate/smart_format; whisper keeps verbose_json + glossary prompt.
- `normalize_transcription` reconstructs speaker-attributed segments from
  Deepgram's word-level speakers (litellm strips them from `words` but leaves
  the raw JSON on `_hidden_params`); whisper's segment path is unchanged.
- `capabilities()` reports diarization based on the resolved model.

Verification: 3 new acb_stt tests (Deepgram hidden-params diarization, param
branching); full unit suite 1700 passed; blocking ruff clean; frontend tsc +
next build clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8

* fix(notes): consolidate slice-2 migrations into 99_note_taker_slice2

Rebased onto current main. main now carries 96_gtd_item_deep_work,
97_gtd_planning_prefs and 98_gtd_day_templates (the last two from the
#198 collision fix), which collide by number with this branch's
96_note_taker_attendees / 97_note_taker_scratch / 98_note_taker_glossary.

Both the migration runner (apply_migrations.sh) and test_migration_prefixes
only recognise TWO-digit prefixes ([0-9][0-9]_), so 99 is the only free
slot and the three can't each take one. They're additive, idempotent, and
order-independent (each depends only on 95_note_taker or base uuid support),
so they collapse cleanly into a single 99_note_taker_slice2.sql. Spec + ORM
comments repointed to the new filename.

Verification: test_migration_prefixes + test_acb_stt pass; full unit suite
1710 passed; blocking ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants