Skip to content

AI Note Taker slices 1–2: notes generation, in-browser recorder, and action-item → task loop-closure - #184

Merged
vjvarada merged 10 commits into
mainfrom
claude/ai-note-taker-architecture-clfh8q
Jul 23, 2026
Merged

AI Note Taker slices 1–2: notes generation, in-browser recorder, and action-item → task loop-closure#184
vjvarada merged 10 commits into
mainfrom
claude/ai-note-taker-architecture-clfh8q

Conversation

@vjvarada

@vjvarada vjvarada commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Builds the AI Note Taker from slice 0's upload→transcript foundation into a working record → notes → tasks / email loop. Rebased onto current main (the earlier migration 94→95 collision was fixed independently by #185).

Slice 1a — notes generation

  • templates.py — meeting-notes template as data → prompt compiler with grounding + anti-injection rules + markdown renderer.
  • summaries.py — notes on acb_llm (strict JSON), single-pass or map-reduce for long transcripts (per-chunk cap logged). Writes summary_md/json + meeting_note + draft action_item rows with segment_ids grounding. POST /summarize, GET/PUT /note, GET /actions, GET /templates.
  • pipeline.py auto-chains notes after transcription; events.py + Next proxy → per-meeting SSE progress.

Slice 1b — the in-browser recorder

  • recordings.py — chunked startchunk?seq= (idempotent, gap-checked append)→complete.
  • recorder.tsMeetingRecorder: getUserMedia, feature-detected mime, a retry-with-backoff uploader (in-memory offline buffer) that flushes before complete; level meter.
  • session/[id] — the "studio" screen: consent line, canvas waveform, mono timer, pause/stop. Library "Record" button live.

Slice 2 — loop-closure into Tasks + Email

  • actions.pyapprove inserts a LOCAL gtd_items task with an origin={kind:meeting,…} link + resulting_task_id (idempotent); reject; approve-all bulk ≥ threshold; every approval audited. Meeting detail action rail is interactive triage with deep-links into /tasks.
  • Attendeesmeeting.attendees JSONB (migration 96_note_taker_attendees.sql), inline chip editor, PUT /attendees.
  • share.pyPOST /share/email/draft LLM-drafts a recap from the notes → a HITL compose modal (editable to/subject/body + account picker) that sends via the existing /email/send. Draft-only server-side.

Verification

  • 9 notes unit tests; full suite 1647 passed. All 17 /notes routes register; frontend tsc clean. Migration chain 00→96 + idempotency validated on a clean PG16+pgvector instance; gtd_items columns verified against migrations 48/65.

Not yet

Share-to-chat (last slice-2 item), then slice 3 — self-hosted STT + open speaker diarization.

🤖 Generated with Claude Code

https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8

@vjvarada vjvarada changed the title fix(notes): renumber migration 94_note_taker.sql → 95 (deploy-blocking collision with 94_email_voice_profile) AI Note Taker slice 1: notes generation + SSE progress + transcript↔notes UI (incl. migration 94→95 fix) Jul 23, 2026
@vjvarada
vjvarada force-pushed the claude/ai-note-taker-architecture-clfh8q branch from d65b10a to 65b0830 Compare July 23, 2026 05:35
@vjvarada vjvarada changed the title AI Note Taker slice 1: notes generation + SSE progress + transcript↔notes UI (incl. migration 94→95 fix) AI Note Taker slice 1: notes generation + SSE progress + transcript↔notes UI Jul 23, 2026
@vjvarada vjvarada changed the title AI Note Taker slice 1: notes generation + SSE progress + transcript↔notes UI AI Note Taker slices 1–2: notes generation, in-browser recorder, and action-item → task loop-closure Jul 23, 2026
@vjvarada
vjvarada force-pushed the claude/ai-note-taker-architecture-clfh8q branch from 28ca691 to 62df225 Compare July 23, 2026 13:53

Copy link
Copy Markdown
Contributor Author

Unit-tests failure is a base-branch migration collision, not this PR

The failing check is test_migration_prefixes::test_migration_numeric_prefixes_are_unique:

duplicate migration number(s): {'93': ['93_rename_reply_to_needs_reply.sql', '93_gtd_planning_prefs.sql']}

This is inherited from main, not introduced here:

I'm not renumbering those four already-merged migrations from other PRs here — that's out of scope for this branch and risks DB drift against what's already applied. This needs a dedicated fix on main (renumber the later of each colliding pair). Once main is green on this check I'll rebase and re-run; this PR's own suite is otherwise green (1700 passed locally, incl. the new Deepgram STT tests).


Generated by Claude Code

@vjvarada
vjvarada force-pushed the claude/ai-note-taker-architecture-clfh8q branch from c042ecb to 9a89740 Compare July 23, 2026 19:17
claude added 10 commits July 23, 2026 19:51
…es UI

Turns transcripts into grounded, structured meeting notes (spec §3.5):

- templates.py: meeting-notes template as data (sections + instructions)
  → prompt compiler with the Meetily-derived grounding + anti-injection
  rules; markdown renderer for the canonical notes document.
- summaries.py: notes generation on acb_llm (acompletion_with_fallback,
  strict JSON, transcript pinned as DATA). Single-pass for short
  transcripts, map-reduce for long ones (per-chunk cap is logged, never
  silent). Writes meeting.summary_md/json + meeting_note, and extracts
  draft action_item rows with segment_ids grounding (HITL approve→task is
  slice 2). Endpoints: POST /summarize, GET/PUT /note, GET /actions,
  GET /templates. Env-tunable model roles (NOTES_MEETING_SUMMARY_MODEL).
- pipeline.py: auto-enqueues notes generation after transcription, so one
  upload yields transcript → notes with no second action.
- events.py: per-meeting SSE progress stream (honest per-stage status,
  closes on terminal) + a dedicated Next stream proxy (the catch-all
  JSON proxy can't stream).
- meeting detail: two-pane transcript↔notes (markdown via MarkdownMessage),
  live SSE-driven progress, and a confidence-ranked action-items rail.

9 new unit tests (template compiler, chunking, ref collection, markdown
render); full suite 1647 passed. Gateway registers all /notes routes;
frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
…(slice 1)

Completes slice 1's capture loop (R1: record a live conversation in-app):

- recordings.py: chunked live-recording endpoints — POST /recordings/start
  (creates the row + empty file, marks the meeting 'recording'),
  POST /recordings/{id}/chunk?seq= (idempotent, gap-checked append; a
  MediaRecorder session's ordered timeslice blobs concatenate to a valid
  container), POST /recordings/{id}/complete (finalize + enqueue
  transcription, which auto-chains into notes generation).
- recorder.ts: MeetingRecorder — getUserMedia (echo-cancel on), feature-
  detected mime (webm/opus, mp4 for Safari), MediaRecorder timeslice, and a
  sequential retry-with-backoff uploader that keeps unacked blobs queued in
  memory (the offline buffer for wifi blips) and flushes before complete.
  AnalyserNode drives a live level meter.
- session/[id]: the 'studio, not form' recording screen (spec §5.1) —
  consent line, canvas waveform, large mono timer, pause/stop, backlog
  indicator, honest mic-permission errors.
- Library 'Record' button (was disabled) now creates an in_person meeting
  and opens the session; 'Upload' demoted to secondary.
- Notes proxy passes application/octet-stream bodies through byte-exact
  (the chunk uploads), alongside the existing multipart passthrough.

Gateway registers all 12 /notes routes; full suite 1647 passed; frontend
tsc clean on the new files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
Turns draft action items into real GTD tasks — the Meeting→Task
counterpart of the Email→Task capture (spec §3.9):

- actions.py: POST /notes/actions/{id}/approve inserts a LOCAL gtd_items
  task (the store the task manager owns) with an origin={kind:meeting,
  meeting_id, action_item_id, segment_ids} provenance link, and records
  action_item.resulting_task_id for a two-way link. Idempotent (re-approve
  returns the existing task). /reject marks it rejected;
  /meetings/{id}/actions/approve-all approves every draft ≥ a confidence
  threshold. Every approval writes an audit_event.
- Meeting detail action rail is now interactive triage (spec §5.5):
  Approve → Task / Reject per item, a confidence-ranked list, an
  'Approve all ≥80%' bulk button, a success toast, and created items
  deep-link into /tasks. Read-only states for created/rejected.

gtd_items columns (id/user_id/title/description from mig 48, origin from
mig 65) verified. Gateway registers all 15 /notes routes; full suite
1647 passed; frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
Closes the Notes→Email arc (spec §3.9):

- migration 96_note_taker_attendees.sql: meeting.attendees JSONB
  [{name,email}] for external attendees (distinct from attendee_ids org
  person refs) — the recap recipient set. Validated on a clean PG16 chain
  run + idempotency; ORM mirror added.
- meetings.py: attendees surface on the meeting detail; PUT
  /meetings/{id}/attendees replaces the list.
- share.py: POST /meetings/{id}/share/email/draft LLM-drafts a recap
  (subject + body) from the notes (pinned as DATA), recipients from
  attendees; degrades to a raw-notes recap if the LLM is unavailable.
  Draft-only — nothing sends server-side.
- UI: inline attendee chip editor on the meeting detail; a 'Follow-up'
  button opens a HITL compose modal (FollowupEmailModal) that fetches the
  draft + the user's email accounts, lets them edit to/subject/body and
  pick a sending account, and sends via the existing /email/send.

Migration chain OK through 96; all 17 /notes routes register; full suite
1647 passed; frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
The first AI-intelligence feature (spec §4 Tier-1 item 2): ask a question
about a meeting and get a transcript-grounded answer with clickable
citations.

- qa.py: POST /notes/meetings/{id}/ask reuses the summarization grounding
  discipline (transcript pinned as DATA, cite segment numbers, ignore
  embedded instructions). No precomputed embeddings needed — a meeting
  transcript is bounded, so it feeds relevant segments directly; when a
  transcript exceeds the pass budget it keyword-ranks segments (+neighbours
  for context) and flags the answer as partial rather than silently
  truncating. New meeting_qa model role (tier-balanced default).
- AskPanel: a Q&A panel in the meeting detail; each answer's segment
  citations are clickable and scroll the transcript to that moment (with a
  highlight flash) + cue the audio — provenance you can touch (§5.3).

4 new unit tests for the keyword/selection logic (incl. the match-wins-
budget-over-its-own-context case). Full suite 1651 passed; all 18 /notes
routes register; frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
…del config

STT was calling Groq/Deepgram/OpenAI via its own httpx clients with its own
key-preference resolution — bypassing the platform's model plumbing that
every other app uses. This makes speech-to-text a configured model like
everything else:

- New tier-stt tier: added to infra/litellm/config.yaml (default
  groq/whisper-large-v3-turbo), acb_llm _TIER_ALIAS_MAP/_TIER_DEFAULTS, and
  settings _TIER_LABELS — so it resolves through the same _TIER_MODEL
  machinery as the chat tiers and is editable in Settings → Models (the
  frontend already declared an 'stt' tier). set_tier_model now accepts it.
- acb_stt.LiteLLMSTT: resolves the model via resolve_underlying_model(
  'tier-stt'), loads keys with _ensure_keys_loaded, registers via
  ensure_model_registered, transcribes with litellm.atranscription
  (verbose_json → segments/words), and emits usage via _emit_usage for
  observability — mirroring acompletion_with_fallback. Removed the bespoke
  httpx providers + key-preference registry. NOTES_STT_PROVIDER/
  DEEPGRAM_API_KEY env replaced by an optional NOTES_STT_MODEL override.
- pipeline: transcript_source is now the resolved litellm id
  (e.g. groq/whisper-large-v3-turbo), no double prefix.

Caveat: diarization is provider-dependent (whisper-via-litellm returns
none → capture-channel prior); real diarization arrives with the self-host
tier (slice 3), which becomes just another tier-stt model.

Verified: tier-stt resolves to the configured model via the shared tier
init; 8 rewritten acb_stt tests (litellm response normalization + tier
resolution) pass; full suite 1652 passed; gateway imports clean; no
blocking lint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
…t (Granola pattern)

The single highest-leverage note-taker UX idea (research §3, spec §4
Tier-1 item 3): the user's attention during the meeting steers a far
better summary than the transcript alone.

- migration 97_note_taker_scratch.sql: meeting.scratch_notes TEXT (the
  user's rough input notes, distinct from the generated meeting_note
  output). Validated on a clean 00->97 chain + idempotency.
- summaries.py: generation reads scratch_notes and threads it into the
  prompt via _scratch_block as EMPHASIS SIGNALS — expand the flagged
  topics from the transcript, fix shorthand, ensure each is addressed,
  but ground every fact in the transcript and never invent from the
  notes. Works for both single-pass and map-reduce.
- meetings.py: PUT /meetings/{id}/scratch; scratch_notes surfaces on the
  meeting detail.
- UI: a 'Jot notes as you go' textarea on the recording screen
  (debounced autosave, flushed on stop) and a 'Your notes' editor on the
  meeting detail (save on blur) — both feed the same field.

Rebased onto current main (picked up #189/#190). 3 new tests for the
scratch block (empty no-op, labelling + grounding guardrail + bound);
full suite 1694 passed; 19 /notes routes; frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
…1 item 6)

A per-user vocabulary that biases transcription toward the right spellings
of product/people/customer names and acronyms — fixing the most common
source of transcript errors before it propagates into notes, action items,
and search.

- migration 98_note_taker_glossary.sql: notes_glossary(user_id, term)
  with case-folded per-user uniqueness. Validated on a clean 00->98 chain
  + idempotency; ORM model added.
- glossary.py: GET/POST/DELETE /notes/glossary CRUD (add is idempotent per
  case-folded term) + format_glossary_prompt/glossary_prompt — a bounded
  vocabulary-hint sentence (term/char caps) fed to the STT prompt.
- pipeline.py: loads the meeting owner's glossary and passes it as
  SttOptions(prompt=...) at transcription time; best-effort so it never
  blocks transcription.
- UI: a GlossaryModal (chip add/remove) opened from a Glossary button in
  the /notes library header.

4 new tests for the prompt builder (empty no-op, vocabulary-hint framing,
trimming, length bound). Full suite 1698 passed; 21 /notes routes;
frontend tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
… → Models)

Make the Note Taker's speech-to-text model user-selectable and add named
speakers (diarization) via a paid Deepgram model — all routed through the
existing LiteLLM tier plumbing. Self-hosted STT (slice 3) is deferred.

Settings → Models
- The Tiers tab gains a dedicated "Speech-to-text" section: its own editable
  picker for the `tier-stt` model, filtered to transcription-capable models
  (chat pickers exclude them, and vice versa). STT is no longer a "Coming soon"
  card. Fixes the tier→metadata lookup (tier-fast → fast) so labels/icons render.
- New `ModelInfo.transcription` capability flag drives the split; Deepgram gets
  a provider tile, setup guide, colour/icon, and a "names speakers" badge.

Model catalogue (settings.py + acb_llm.model_limits)
- Add whisper models to Groq/OpenAI, gpt-4o-transcribe, and a Deepgram provider
  (nova-3/nova-2, DEEPGRAM_API_KEY). All tagged transcription:True, audio:True.
- `_is_transcription_model` heuristic flags un-catalogued whisper/nova variants
  a live provider fetch surfaces, keeping them out of the chat pickers.
- `_provider_from_model` recognises deepgram/*; key_store loads DEEPGRAM_API_KEY.

Diarization plumbing (acb_stt)
- `transcribe()` branches params per family: Deepgram gets native
  diarize/punctuate/smart_format; whisper keeps verbose_json + glossary prompt.
- `normalize_transcription` reconstructs speaker-attributed segments from
  Deepgram's word-level speakers (litellm strips them from `words` but leaves
  the raw JSON on `_hidden_params`); whisper's segment path is unchanged.
- `capabilities()` reports diarization based on the resolved model.

Verification: 3 new acb_stt tests (Deepgram hidden-params diarization, param
branching); full unit suite 1700 passed; blocking ruff clean; frontend tsc +
next build clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
Rebased onto current main. main now carries 96_gtd_item_deep_work,
97_gtd_planning_prefs and 98_gtd_day_templates (the last two from the
#198 collision fix), which collide by number with this branch's
96_note_taker_attendees / 97_note_taker_scratch / 98_note_taker_glossary.

Both the migration runner (apply_migrations.sh) and test_migration_prefixes
only recognise TWO-digit prefixes ([0-9][0-9]_), so 99 is the only free
slot and the three can't each take one. They're additive, idempotent, and
order-independent (each depends only on 95_note_taker or base uuid support),
so they collapse cleanly into a single 99_note_taker_slice2.sql. Spec + ORM
comments repointed to the new filename.

Verification: test_migration_prefixes + test_acb_stt pass; full unit suite
1710 passed; blocking ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
@vjvarada
vjvarada force-pushed the claude/ai-note-taker-architecture-clfh8q branch from 9a89740 to 71890a8 Compare July 23, 2026 19:52
@vjvarada
vjvarada merged commit 3dcfd28 into main Jul 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants