AI Note Taker slices 1–2: notes generation, in-browser recorder, and action-item → task loop-closure - #184
Merged
Merged
Conversation
vjvarada
force-pushed
the
claude/ai-note-taker-architecture-clfh8q
branch
from
July 23, 2026 05:35
d65b10a to
65b0830
Compare
vjvarada
force-pushed
the
claude/ai-note-taker-architecture-clfh8q
branch
from
July 23, 2026 13:53
28ca691 to
62df225
Compare
Contributor
Author
Unit-tests failure is a base-branch migration collision, not this PRThe failing check is This is inherited from
I'm not renumbering those four already-merged migrations from other PRs here — that's out of scope for this branch and risks DB drift against what's already applied. This needs a dedicated fix on Generated by Claude Code |
vjvarada
force-pushed
the
claude/ai-note-taker-architecture-clfh8q
branch
from
July 23, 2026 19:17
c042ecb to
9a89740
Compare
…es UI Turns transcripts into grounded, structured meeting notes (spec §3.5): - templates.py: meeting-notes template as data (sections + instructions) → prompt compiler with the Meetily-derived grounding + anti-injection rules; markdown renderer for the canonical notes document. - summaries.py: notes generation on acb_llm (acompletion_with_fallback, strict JSON, transcript pinned as DATA). Single-pass for short transcripts, map-reduce for long ones (per-chunk cap is logged, never silent). Writes meeting.summary_md/json + meeting_note, and extracts draft action_item rows with segment_ids grounding (HITL approve→task is slice 2). Endpoints: POST /summarize, GET/PUT /note, GET /actions, GET /templates. Env-tunable model roles (NOTES_MEETING_SUMMARY_MODEL). - pipeline.py: auto-enqueues notes generation after transcription, so one upload yields transcript → notes with no second action. - events.py: per-meeting SSE progress stream (honest per-stage status, closes on terminal) + a dedicated Next stream proxy (the catch-all JSON proxy can't stream). - meeting detail: two-pane transcript↔notes (markdown via MarkdownMessage), live SSE-driven progress, and a confidence-ranked action-items rail. 9 new unit tests (template compiler, chunking, ref collection, markdown render); full suite 1647 passed. Gateway registers all /notes routes; frontend tsc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
…(slice 1)
Completes slice 1's capture loop (R1: record a live conversation in-app):
- recordings.py: chunked live-recording endpoints — POST /recordings/start
(creates the row + empty file, marks the meeting 'recording'),
POST /recordings/{id}/chunk?seq= (idempotent, gap-checked append; a
MediaRecorder session's ordered timeslice blobs concatenate to a valid
container), POST /recordings/{id}/complete (finalize + enqueue
transcription, which auto-chains into notes generation).
- recorder.ts: MeetingRecorder — getUserMedia (echo-cancel on), feature-
detected mime (webm/opus, mp4 for Safari), MediaRecorder timeslice, and a
sequential retry-with-backoff uploader that keeps unacked blobs queued in
memory (the offline buffer for wifi blips) and flushes before complete.
AnalyserNode drives a live level meter.
- session/[id]: the 'studio, not form' recording screen (spec §5.1) —
consent line, canvas waveform, large mono timer, pause/stop, backlog
indicator, honest mic-permission errors.
- Library 'Record' button (was disabled) now creates an in_person meeting
and opens the session; 'Upload' demoted to secondary.
- Notes proxy passes application/octet-stream bodies through byte-exact
(the chunk uploads), alongside the existing multipart passthrough.
Gateway registers all 12 /notes routes; full suite 1647 passed; frontend
tsc clean on the new files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
Turns draft action items into real GTD tasks — the Meeting→Task
counterpart of the Email→Task capture (spec §3.9):
- actions.py: POST /notes/actions/{id}/approve inserts a LOCAL gtd_items
task (the store the task manager owns) with an origin={kind:meeting,
meeting_id, action_item_id, segment_ids} provenance link, and records
action_item.resulting_task_id for a two-way link. Idempotent (re-approve
returns the existing task). /reject marks it rejected;
/meetings/{id}/actions/approve-all approves every draft ≥ a confidence
threshold. Every approval writes an audit_event.
- Meeting detail action rail is now interactive triage (spec §5.5):
Approve → Task / Reject per item, a confidence-ranked list, an
'Approve all ≥80%' bulk button, a success toast, and created items
deep-link into /tasks. Read-only states for created/rejected.
gtd_items columns (id/user_id/title/description from mig 48, origin from
mig 65) verified. Gateway registers all 15 /notes routes; full suite
1647 passed; frontend tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
Closes the Notes→Email arc (spec §3.9):
- migration 96_note_taker_attendees.sql: meeting.attendees JSONB
[{name,email}] for external attendees (distinct from attendee_ids org
person refs) — the recap recipient set. Validated on a clean PG16 chain
run + idempotency; ORM mirror added.
- meetings.py: attendees surface on the meeting detail; PUT
/meetings/{id}/attendees replaces the list.
- share.py: POST /meetings/{id}/share/email/draft LLM-drafts a recap
(subject + body) from the notes (pinned as DATA), recipients from
attendees; degrades to a raw-notes recap if the LLM is unavailable.
Draft-only — nothing sends server-side.
- UI: inline attendee chip editor on the meeting detail; a 'Follow-up'
button opens a HITL compose modal (FollowupEmailModal) that fetches the
draft + the user's email accounts, lets them edit to/subject/body and
pick a sending account, and sends via the existing /email/send.
Migration chain OK through 96; all 17 /notes routes register; full suite
1647 passed; frontend tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
The first AI-intelligence feature (spec §4 Tier-1 item 2): ask a question
about a meeting and get a transcript-grounded answer with clickable
citations.
- qa.py: POST /notes/meetings/{id}/ask reuses the summarization grounding
discipline (transcript pinned as DATA, cite segment numbers, ignore
embedded instructions). No precomputed embeddings needed — a meeting
transcript is bounded, so it feeds relevant segments directly; when a
transcript exceeds the pass budget it keyword-ranks segments (+neighbours
for context) and flags the answer as partial rather than silently
truncating. New meeting_qa model role (tier-balanced default).
- AskPanel: a Q&A panel in the meeting detail; each answer's segment
citations are clickable and scroll the transcript to that moment (with a
highlight flash) + cue the audio — provenance you can touch (§5.3).
4 new unit tests for the keyword/selection logic (incl. the match-wins-
budget-over-its-own-context case). Full suite 1651 passed; all 18 /notes
routes register; frontend tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
…del config STT was calling Groq/Deepgram/OpenAI via its own httpx clients with its own key-preference resolution — bypassing the platform's model plumbing that every other app uses. This makes speech-to-text a configured model like everything else: - New tier-stt tier: added to infra/litellm/config.yaml (default groq/whisper-large-v3-turbo), acb_llm _TIER_ALIAS_MAP/_TIER_DEFAULTS, and settings _TIER_LABELS — so it resolves through the same _TIER_MODEL machinery as the chat tiers and is editable in Settings → Models (the frontend already declared an 'stt' tier). set_tier_model now accepts it. - acb_stt.LiteLLMSTT: resolves the model via resolve_underlying_model( 'tier-stt'), loads keys with _ensure_keys_loaded, registers via ensure_model_registered, transcribes with litellm.atranscription (verbose_json → segments/words), and emits usage via _emit_usage for observability — mirroring acompletion_with_fallback. Removed the bespoke httpx providers + key-preference registry. NOTES_STT_PROVIDER/ DEEPGRAM_API_KEY env replaced by an optional NOTES_STT_MODEL override. - pipeline: transcript_source is now the resolved litellm id (e.g. groq/whisper-large-v3-turbo), no double prefix. Caveat: diarization is provider-dependent (whisper-via-litellm returns none → capture-channel prior); real diarization arrives with the self-host tier (slice 3), which becomes just another tier-stt model. Verified: tier-stt resolves to the configured model via the shared tier init; 8 rewritten acb_stt tests (litellm response normalization + tier resolution) pass; full suite 1652 passed; gateway imports clean; no blocking lint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
…t (Granola pattern)
The single highest-leverage note-taker UX idea (research §3, spec §4
Tier-1 item 3): the user's attention during the meeting steers a far
better summary than the transcript alone.
- migration 97_note_taker_scratch.sql: meeting.scratch_notes TEXT (the
user's rough input notes, distinct from the generated meeting_note
output). Validated on a clean 00->97 chain + idempotency.
- summaries.py: generation reads scratch_notes and threads it into the
prompt via _scratch_block as EMPHASIS SIGNALS — expand the flagged
topics from the transcript, fix shorthand, ensure each is addressed,
but ground every fact in the transcript and never invent from the
notes. Works for both single-pass and map-reduce.
- meetings.py: PUT /meetings/{id}/scratch; scratch_notes surfaces on the
meeting detail.
- UI: a 'Jot notes as you go' textarea on the recording screen
(debounced autosave, flushed on stop) and a 'Your notes' editor on the
meeting detail (save on blur) — both feed the same field.
Rebased onto current main (picked up #189/#190). 3 new tests for the
scratch block (empty no-op, labelling + grounding guardrail + bound);
full suite 1694 passed; 19 /notes routes; frontend tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
…1 item 6) A per-user vocabulary that biases transcription toward the right spellings of product/people/customer names and acronyms — fixing the most common source of transcript errors before it propagates into notes, action items, and search. - migration 98_note_taker_glossary.sql: notes_glossary(user_id, term) with case-folded per-user uniqueness. Validated on a clean 00->98 chain + idempotency; ORM model added. - glossary.py: GET/POST/DELETE /notes/glossary CRUD (add is idempotent per case-folded term) + format_glossary_prompt/glossary_prompt — a bounded vocabulary-hint sentence (term/char caps) fed to the STT prompt. - pipeline.py: loads the meeting owner's glossary and passes it as SttOptions(prompt=...) at transcription time; best-effort so it never blocks transcription. - UI: a GlossaryModal (chip add/remove) opened from a Glossary button in the /notes library header. 4 new tests for the prompt builder (empty no-op, vocabulary-hint framing, trimming, length bound). Full suite 1698 passed; 21 /notes routes; frontend tsc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
… → Models) Make the Note Taker's speech-to-text model user-selectable and add named speakers (diarization) via a paid Deepgram model — all routed through the existing LiteLLM tier plumbing. Self-hosted STT (slice 3) is deferred. Settings → Models - The Tiers tab gains a dedicated "Speech-to-text" section: its own editable picker for the `tier-stt` model, filtered to transcription-capable models (chat pickers exclude them, and vice versa). STT is no longer a "Coming soon" card. Fixes the tier→metadata lookup (tier-fast → fast) so labels/icons render. - New `ModelInfo.transcription` capability flag drives the split; Deepgram gets a provider tile, setup guide, colour/icon, and a "names speakers" badge. Model catalogue (settings.py + acb_llm.model_limits) - Add whisper models to Groq/OpenAI, gpt-4o-transcribe, and a Deepgram provider (nova-3/nova-2, DEEPGRAM_API_KEY). All tagged transcription:True, audio:True. - `_is_transcription_model` heuristic flags un-catalogued whisper/nova variants a live provider fetch surfaces, keeping them out of the chat pickers. - `_provider_from_model` recognises deepgram/*; key_store loads DEEPGRAM_API_KEY. Diarization plumbing (acb_stt) - `transcribe()` branches params per family: Deepgram gets native diarize/punctuate/smart_format; whisper keeps verbose_json + glossary prompt. - `normalize_transcription` reconstructs speaker-attributed segments from Deepgram's word-level speakers (litellm strips them from `words` but leaves the raw JSON on `_hidden_params`); whisper's segment path is unchanged. - `capabilities()` reports diarization based on the resolved model. Verification: 3 new acb_stt tests (Deepgram hidden-params diarization, param branching); full unit suite 1700 passed; blocking ruff clean; frontend tsc + next build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
Rebased onto current main. main now carries 96_gtd_item_deep_work, 97_gtd_planning_prefs and 98_gtd_day_templates (the last two from the #198 collision fix), which collide by number with this branch's 96_note_taker_attendees / 97_note_taker_scratch / 98_note_taker_glossary. Both the migration runner (apply_migrations.sh) and test_migration_prefixes only recognise TWO-digit prefixes ([0-9][0-9]_), so 99 is the only free slot and the three can't each take one. They're additive, idempotent, and order-independent (each depends only on 95_note_taker or base uuid support), so they collapse cleanly into a single 99_note_taker_slice2.sql. Spec + ORM comments repointed to the new filename. Verification: test_migration_prefixes + test_acb_stt pass; full unit suite 1710 passed; blocking ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8
vjvarada
force-pushed
the
claude/ai-note-taker-architecture-clfh8q
branch
from
July 23, 2026 19:52
9a89740 to
71890a8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds the AI Note Taker from slice 0's upload→transcript foundation into a working record → notes → tasks / email loop. Rebased onto current
main(the earlier migration 94→95 collision was fixed independently by #185).Slice 1a — notes generation
templates.py— meeting-notes template as data → prompt compiler with grounding + anti-injection rules + markdown renderer.summaries.py— notes onacb_llm(strict JSON), single-pass or map-reduce for long transcripts (per-chunk cap logged). Writessummary_md/json+meeting_note+ draftaction_itemrows withsegment_idsgrounding.POST /summarize,GET/PUT /note,GET /actions,GET /templates.pipeline.pyauto-chains notes after transcription;events.py+ Next proxy → per-meeting SSE progress.Slice 1b — the in-browser recorder
recordings.py— chunkedstart→chunk?seq=(idempotent, gap-checked append)→complete.recorder.ts—MeetingRecorder: getUserMedia, feature-detected mime, a retry-with-backoff uploader (in-memory offline buffer) that flushes before complete; level meter.session/[id]— the "studio" screen: consent line, canvas waveform, mono timer, pause/stop. Library "Record" button live.Slice 2 — loop-closure into Tasks + Email
actions.py—approveinserts a LOCALgtd_itemstask with anorigin={kind:meeting,…}link +resulting_task_id(idempotent);reject;approve-allbulk ≥ threshold; every approval audited. Meeting detail action rail is interactive triage with deep-links into/tasks.meeting.attendeesJSONB (migration96_note_taker_attendees.sql), inline chip editor,PUT /attendees.share.py—POST /share/email/draftLLM-drafts a recap from the notes → a HITL compose modal (editable to/subject/body + account picker) that sends via the existing/email/send. Draft-only server-side.Verification
/notesroutes register; frontendtscclean. Migration chain00→96+ idempotency validated on a clean PG16+pgvector instance;gtd_itemscolumns verified against migrations 48/65.Not yet
Share-to-chat (last slice-2 item), then slice 3 — self-hosted STT + open speaker diarization.
🤖 Generated with Claude Code
https://claude.ai/code/session_016xdbRumGveUBFRsPh3Duu8