ResearchMate

Reading papers is still a solo grind: PDF in one window, notes somewhere else, Google tabs everywhere, ChatGPT on the side, Anki later, Zotero maybe, and your own half-broken memory trying to hold the thread.

ResearchMate turns one paper into a living workspace. You open the paper, and suddenly you are not alone: a small lab appears around it.

Instead of one chatbot, ResearchMate gives you a lab bench of grounded helpers: one explains, one challenges, one builds your wiki, one creates practice, one tracks what you forget, and one connects the paper to your longer research direction.

Every paragraph can become a Feynman explanation, a flashcard, a quiz, a wiki node, a debate, a related-paper search, or a research idea.

The language model (Gemma 4 on Cerebras) is the fast, grounded voice.

Cognee is the lab notebook that makes it a partner instead of a document Q&A box, a hybrid graph+vector memory that every agent reads before it acts and writes after you work.

Feature Gallery

Project Start Window

Project-first workflow: create a named project, reopen an existing one, and add or remove papers over time.
Material gating: ResearchMate does not study from empty space; papers are visibly the source of truth.
Control room: project status, memory health, citation graph artifacts, settings, and activity updates are surfaced from the start window.
Editable behavior: project preferences, learner profile, and internal agent skills can be adjusted through ResearchMate Settings.

Paper-Centered Workspace

Read uploaded papers inside the app.
Select text to ground Wiki and Chat.
Add margin notes beside the paper.
Send detected figures or regions into study tools.

Knowledge Graph As Table Of Contents

Auto-generates a concept map from uploaded material.
Uses honest states: ongoing or completed, not fake mastery scores.
Uses Cognee memory to reshape future study.
Commit runs the idea-observer, writes memory, and lets the Graph Curator add explored concepts.

Citation Graph

Shows uploaded project papers as graph nodes.
Connects papers with verified paper-to-paper citation/reference edges.
Provides context for related-paper recommendations.

Wiki

Turns selected terms into grounded explanation cards.
Supports active recall prompts.
Can attach grounded visuals through the visual service.

Chat

Answers from project material and selected context.
Supports /quiz <topic> and /flashcards <topic> as inline tools.
Mic dictation is wired for transcription-only input: record audio, insert the transcript into the draft, then edit/send manually.
Routes paper recommendation requests through project, citation, and memory context.
Optional Net Support adds explicit web citations when configured.

Pair Buddy

Runs Socratic pair-reading conversations.
Feynman mode lets the student teach a concept back.
Style feedback is stored as learner preference memory.
Voice Buddy is visible as a future shell; dictation infrastructure exists, but full voice-out and hands-free STT -> LM -> TTS loop are deferred.

Sandboxed Visuals

Generates visuals lazily.
Declines when there is no meaningful grounding.
Runs generated HTML in a sandboxed iframe.
Repairs runtime errors through /sandbox/repair.

Local Memory Layers

Cognee memory lives locally under ~/.studybuddy/cognee/.
Embeddings run locally through fastembed / sentence-transformers.
Memory failures are surfaced as degraded/unavailable instead of silently becoming plain RAG.
research-memory-reviewer is loaded as Cognee skill content for project memory review.

Signoz Observability Layers

Built With GPT-5.6 And Codex

GPT-5.6 in Codex was ResearchMate's primary development partner.

During development, GPT-5.6 in Codex worked inside the repository to inspect the full-stack system, challenge design assumptions, turn decisions into implementation plans, edit code and tests, diagnose failures, and keep the architecture documentation aligned with the shipped product.

ResearchMate was developed through a combination of specification-driven development, custom agent harnesses, and goal-bounded loop engineering.

Before implementation, major features were expressed as specifications defining their purpose, architectural boundaries, interfaces, failure conditions, and acceptance criteria. This was particularly important for systems such as layered memory, citation exploration, PDF-region analysis, visualization, and agent orchestration, where a locally correct implementation could still violate a wider product invariant.

Codex, powered by GPT-5.6, was then used as an engineering agent rather than only as a code-completion tool. Work was divided into bounded objectives and delegated through task-specific subagents and harnesses. Depending on the feature, these agents could inspect the existing architecture, propose a plan, implement changes, run targeted tests, inspect failures, and perform another revision.

This created a controlled engineering loop:

establish a concrete goal and repository context;
constrain the work using the relevant specification;
implement the smallest coherent change;
verify it through tests, static checks, runtime evidence, or UI behavior;
review the result against the original acceptance criteria;
either revise with the new evidence or terminate the loop.

Model reasoning effort was allocated according to the risk and ambiguity of the task. Light reasoning was used for narrow execution work whose design had already been established. Medium reasoning was used for debugging, test failures, refactoring, and code review. High reasoning was reserved for system architecture, specification design, failure analysis, and decisions involving several interacting subsystems.

The custom subagents were able to implement their coding, exploratory tasks with little supervision. Their role was to pursue a tightly defined goal and return evidence. Architectural direction, scope control, acceptance criteria, security boundaries, and the final decision to retain or reject an implementation remained human-controlled.

This workflow made it possible to develop several complex systems in parallel without treating generated code as correct by default. The important output of each loop was therefore not merely a patch, but a patch accompanied by enough evidence to justify moving the feature to its next lifecycle state.

A Few Key decisions made with Codex

Decision	Why it mattered
Replace disposable sessions with persistent projects	Research accumulates; papers, annotations, history, and memory need a durable scope.
Make memory load-bearing	The product test became: day 10 should behave differently from day 1. Project observations belong in Chroma; only durable cross-project learner signals belong in Cognee.
Replace mastery scores with an idea-observer	Source-cited qualitative observations and explicit `ongoing` / `completed` states are more honest than fabricated numeric understanding.
Make SQLite/FTS5 canonical and Chroma the dense index	Stable page and bounding-box evidence can support citations, graph anchors, deletion, and rebuilds without trusting model prose or vector metadata as truth.
Keep external papers transient until an explicit add	Recommendations cannot silently become project truth; acquisition must be verified and pass through normal ingestion.
Use strict structured outputs at agent boundaries	Independently developed agents compose through validated contracts; malformed output fails visibly and can be retried.
Surface memory degradation without blocking study	Cognee is important enough to monitor, but an unavailable memory layer should not destroy the grounded paper-reading workflow.

Human judgment set the absolute product direction and decided trade-offs.

Codex supplied rapid codebase analysis, implementation leverage, adversarial review, and repeatable verification, the difference between using AI to generate snippets and using it as an engineering collaborator across project development lifecycle.

Project Motivation

Reading research is a long-horizon, accumulating activity but the tools for it have no memory and no shape.

Chatbots forget. A pure-RAG tutor (or a raw LLM) knows only what the current document says. Reload the same paper next week and it re-explains what you've mastered, re-asks what you've answered, and never notices you've been stuck on the same idea for three sessions. There is no you in the loop.
Material arrives over time, but tools are one-shot. Real study grows a corpus incrementally - add a chapter today, a problem set next week, connect them later. Session-based tools treat every upload as a disposable, isolated event with no through-line.
LLMs hallucinate, and "it rendered" isn't "it's right." A model will happily fabricate a citation, a formula, or a visualization that runs cleanly and teaches something false.
"Understanding" gets faked with numbers. A monotone score bolted onto a graph measures activity, not whether an idea actually holds up against the source.

The Solution

ResearchMate is a grounded, memory-load-bearing research partner.

Your material groups into resumable projects; a knowledge graph gives it shape; every answer is sourced from your content; and a local hybrid graph+vector memory of how you read, struggle, and explore underlies every agent - so the lab rearranges itself around this learner and this research direction, not just the document open right now.

flowchart LR
  U["Upload your material<br/>(grouped into Projects)"] --> G["Auto-extracted<br/>Knowledge Graph"]
  G --> S["Workspace: Wiki · Chat<br/>Pair Buddy · Citation Graph"]
  S --> C["Commit"]
  C --> M[("Cognee memory<br/>day 10 &gt; day 1")]
  M -.->|"read before every action<br/>(Brain · Tutor · Evaluator · Curator)"| G

Grounded-only. Agents organises and rephrases your material and cites every claim ([Source: file, chunk N]); it never invents facts from its weights (except explicitly-cited web results in Net mode).
Memory changes behavior, not just prompts. The Brain restructures the curriculum on your history, the Tutor adapts per-concept, the Evaluator judges your ideas against your trajectory, and the Graph Curator spawns nodes for what you explored off-path (detailed below).
Ideas judged against sources, students never scored. Two honest node states (ongoing / completed) and a cited-prose idea-observer replace fabricated mastery numbers.
Hallucination-resistant by construction. Verbatim, self-checking anchors and a plan→QC→code→QC visual loop make fabricated content unusable, not merely discouraged.

Cognee: the memory that makes it a study partner

The single most important thing about ResearchMate is that it doesn't start from zero every time.

RAG alone (ChromaDB, here) only knows what the uploaded material says.

Reload the same paper next week and a pure-RAG tutor re-teaches what you've mastered, re-asks what you've answered, and never notices you've been stuck on the same idea across three sessions.

Cognee is the layer that closes that gap, it remembers how this student thinks, persisted locally as a hybrid knowledge graph + vectors, and it is load-bearing, not a bolt-on: pull Cognee out and the app behaves differently, because memory shapes what the agents actually do.

Concretely, memory changes agent behavior at four points:

The Brain Agent branches the curriculum structurally on memory - it doesn't just read it as context. Before a single node is drawn, query_prior_knowledge() recalls what you've engaged before, and the tree is restructured: ground you've mastered is collapsed or omitted, a scaffold node is inserted where your history reveals a gap, and sections are ordered from known → new. Nodes carry a memory_tag (new / review / scaffold) so the reshaping is visible in the graph itself - a returning student literally gets a different tree.
The Evaluator is a memory reader, not just a writer. At Commit it judges each idea against your trajectory - "still conflates X as in an earlier session", "moved from what-is to why questions here" - not the current session in isolation. It's an idea-observer: it writes grounded, cited prose observations about your ideas (cognee.remember()), never a fabricated score.
The Tutor adapts per-concept from your cross-document history. Opening a lesson pulls that concept's memory and adapts - skimming what you have, leaning into what you've struggled with - instead of teaching every student the same words.
A dedicated Graph Curator reads Cognee to tell genuine engagement (an idea you actually developed) from a passing mention, and spawns new, distinctly-coloured "exploration" nodes for what you discovered off the planned path.

flowchart TB
  subgraph READ["READ - before acting: query_prior_knowledge() → recall(INSIGHTS)"]
    B["Brain Agent<br/>branches the curriculum structurally"]
    T["Tutor<br/>adapts each lesson per-concept"]
    E["Evaluator<br/>judges ideas against your trajectory"]
  end
  B & T & E --> J["Interaction Journal"]
  J -->|"Commit"| OBS["Idea-Observer<br/>cited prose observations (no scores)"]
  J -->|"Commit"| CUR["Graph Curator<br/>spawns exploration nodes"]
  OBS -->|"remember() + improve()"| COG[("Cognee<br/>~/.studybuddy/cognee")]
  CUR --> COG
  COG -.->|"feeds the next READ"| READ

And it never leaves the machine.

Cognee's data root is pinned to ~/.studybuddy/cognee/ (LanceDB for vectors, SQLite for relational metadata), and even its embedding model runs locally (fastembed / sentence-transformers) so a student's learning history is only ever on disk and in the grounded LLM calls that read and write it.

Without Cognee, ResearchMate is a well-built single-session RAG tutor. With it, day 10 is structurally smarter than day 1 - which is the whole point of a study partner.

Cognee memory internals — bootstrap, write/flush, and the day-10-vs-day-1 mechanism

Bootstrap (app/main.py lifespan, app/services/cognee_bootstrap.py): Cerebras is aliased as an OpenAI-compatible LiteLLM provider (llm_model="openai/gemma-4-31b", Cerebras's own base URL), llm_instructor_mode="json_schema_mode" is forced (Cerebras doesn't reliably support instructor's default TOOLS-mode extraction, so without this Gemma can emit markdown-fenced JSON that fails Cognee's Pydantic parse), embeddings are forced local (fastembed, sentence-transformers/all-MiniLM-L6-v2, 384-dim — Cerebras has no embeddings endpoint), and the data/system root is pinned to ~/.studybuddy/cognee/.

Two write destinations, both gated by MemoryPromotionGate (app/services/memory_promotion.py) so nothing becomes durable memory just because the model said it — assistant output alone can never create persistent memory:

Project memory what you're investigating in this project. Project-filtered ChromaDB (project_memory collection), written via stage_project_observation(); requires evidence-backed statements.
Student profile — durable, cross-project traits. Cognee's research_profile dataset, written via stage_profile_observation() → cognee.remember() (cheap — writes only to Cognee's session cache, no graph rebuild yet); non-explicit signals require recurrence across ≥2 projects before promotion.

Flush: flush_project() → cognee.improve(dataset="research_profile", session_ids=[project_id]) is the expensive step that actually runs Cognee's add+cognify pipeline and commits the session cache into the permanent graph. This only runs at Commit (COMMIT_PROJECT), in the background, not on every turn.

Read: query_prior_knowledge() (app/services/student_memory.py) is the single boundary every agent goes through — cognee.recall(query_type=SearchType.GRAPH_COMPLETION | SearchType.TEMPORAL, only_context=True, ...), only_context=True skipping Cognee's own answer-synthesis since callers just want raw context. A DatabaseNotCreatedError / "No data found" is treated as a healthy empty state (fresh install, no Commit yet), not a failure.

Four consumers, four structural effects — this is the actual mechanism behind "day 10 feels different from day 1":

Agent	Reads	Structural effect
Brain Agent	prior knowledge before building the tree	tags each section `new` / `review` / `scaffold`; collapses mastered ground, inserts scaffold nodes for revealed gaps, orders known→new
Evaluator (idea-observer)	cross-session trajectory at Commit	judges ideas against history ("still conflates X as in an earlier session", "moved from what-is to why here") instead of scoring the session in isolation
Tutor	per-concept memory on lesson open	adapts what a lesson leans into vs. skims, per concept, per student
Graph Curator	prior context at `COMMIT_PROJECT`	filters "genuine engagement" from "passing mention" before spawning `origin="exploration"` nodes

Pull Cognee out and the Brain Agent produces an identical tree on day 1 and day 10 — memory reshapes what gets built and what gets judged against, not just prompt tone.

Privacy

All student data stays local at ~/.studybuddy/ -> nothing goes to the cloud
ChromaDB vector store, session memory, annotations, and summaries are all file-based
The only outbound calls are to the Cerebras API (inference) and optionally Tavily (web search)

🛠 Prerequisites

Tool	Version	Install
Python	3.12+	python.org
uv	latest	`pip install uv` or docs.astral.sh/uv
Node.js	20+	nodejs.org

Quick Start

# 1. Clone
git clone https://github.com/solusops/StudyBuddy.git
cd StudyBuddy

# 2. Install dependencies
npm install
# Installs root + frontend Node workspaces and runs `uv sync --directory backend`.

# 3. API keys
cp backend/.env.example backend/.env
# Edit backend/.env:
#   CEREBRAS_API_KEY=csk-...       (required)
#   TAVILY_API_KEY=tvly-...        (optional -> enables Net Support mode)
#   YOUTUBE_API_KEY=...            (optional -> enables Deep Dive)

# 4. Launch
npm run dev

Open http://localhost:5173. The "Start Studying" button activates once the backend finishes loading (~10–15s on first run while the embedding model warms up).

Sample data and a 5-minute demo

No private or licensed sample paper is bundled. For the demo, use any open-access research PDF you are permitted to download (an arXiv paper works well). A single text-based PDF is enough; scanned PDFs without a text layer may produce weaker evidence extraction.

Create a project such as Attention Paper Demo and add the PDF.
Build the knowledge graph, open one concept, and select a paragraph in the reader.
Ask Chat to explain the selection, then try /quiz <topic> or /flashcards <topic>.
Teach the concept back in Pair Buddy/Feynman mode.
Open Citation Graph. With two related uploaded papers, ResearchMate can verify and display paper-to-paper relationships.
Click Commit, reopen the project, and continue studying to demonstrate persistent project context and learner memory.

The first ingestion and first memory operation are slower because local embedding and Cognee assets initialize on demand. Net Support, YouTube Deep Dive, and paper-provider lookups require internet access and their optional keys; the core uploaded-paper workflow does not require sample data beyond the PDF itself.

🖥 Running

Command	What it does
`npm run dev`	Starts Vite (`localhost:5173`) + uvicorn (`127.0.0.1:8765`) concurrently
`npm run dev:docker`	Same dev servers, bound to `0.0.0.0` for containers
`npm run dev:electron`	Full Electron desktop shell + Vite
`npm run setup:node`	Installs only Node dependencies, skipping backend postinstall
`npm run setup:backend`	Runs `uv sync --directory backend`
`cd backend && uv run uvicorn app.main:app --reload --host 127.0.0.1 --port 8765`	Backend only
`cd frontend && npm run dev`	Frontend only (browser, no Electron IPC)

Voice Dictation And STT Benchmarks

Current voice support is dictation-only. The Chat mic records audio, prepares a 16 kHz mono WAV in the browser, sends it through TRANSCRIBE_AUDIO, and inserts the returned text into the current draft. It does not auto-send, interpret voice commands, or trigger an agent response.

The backend STT boundary is TranscriptionService.transcribe(...) -> STTResult, backed today by pywhispercpp / whisper.cpp. The current default is:

Setting	Value
Backend	`whisper.cpp`
Wrapper	`pywhispercpp`
Model	`base.en`
Model file	`ggml-base.en-q5_1.bin`
Quantization	`q5_1`
Language	`en`
Task	`transcribe`
Translate	`false`

Override the model file with STT_MODEL_PATH when comparing variants:

cd backend
$env:STT_MODEL_PATH = "$env:LOCALAPPDATA\pywhispercpp\pywhispercpp\models\ggml-base.en.bin"
.\.venv\Scripts\python.exe -m app.benchmarks.stt_offline_benchmark --vad none,silero

Docker

For a reproducible browser-based dev run:

cp backend/.env.example backend/.env
# Edit backend/.env with your API keys.
docker compose up --build

Then open http://localhost:5173. The container runs the same Vite frontend and FastAPI backend as npm run dev; the Electron desktop shell still runs natively with npm run dev:electron.

Images are built on pull requests and pushed to GitHub Container Registry on pushes to the default branch by .github/workflows/docker.yml.

How To Use

Open ResearchMate.
Create a project or reopen an existing one.
Add the papers ResearchMate is allowed to use.
Build the project graph.
Open a concept from the graph, then start studying.
Read in the PDF workspace and select text when you want grounded help.
Use Wiki for quick drill-downs.
Use Chat for questions, /quiz <topic>, /flashcards <topic>, and paper recommendations.
Use Pair Buddy for Socratic/Feynman-style explanation practice.
Open Citation Graph to inspect paper-to-paper relationships.
Commit when you want ResearchMate to write observations into memory and update the project.

System Design Patterns

Beyond the Cerebras-shaped decomposition described above, a handful of deliberate patterns recur throughout the codebase:

Content-addressed document identity

Two hash-based identifiers drive caching and dedup: a file_id (SHA-256 of one file's bytes) and a document_id (an order-independent combined hash of the whole file set in a session -> sorted per-file hashes, joined, re-hashed). Uploading the same set of papers again, in any order, resolves to the same document_id, which means the curriculum tree is replayed from ~/.studybuddy/graphs/doc_{document_id}.json instead of regenerated -> a real latency and cost win for anyone re-studying the same material, with a defense-in-depth validity check (structural soundness + full document coverage) before a cached tree is trusted, falling back to regeneration if it isn't.

Multi-stage curriculum generation with cross-paper merge detection

The curriculum pipeline is three explicit stages, not one call: derive_root_and_sections() (coarse root + sections, each tagged with which source document(s) it draws from) → expand_section() (one parallel call per section, fed only that section's relevant document excerpts, aware of sibling sections so it doesn't duplicate their topics) → cleanup_curriculum() (sees every node's source filenames and explicitly looks for the same concept described differently across papers, merging those into one node tagged is_merged with a merge_summary explaining what's shared vs. distinct). Every downstream consumer of a merged node -> lessons, chat, Feynman -> gets told it's a merge so it attributes claims to the correct paper instead of blending two treatments into one voice.

Ideas judged against sources, never students scored

The graph holds two node states - ongoing and completed (explicit, via a "Mark Completed" button) - not a numeric mastery score. Evaluation was deliberately re-founded away from grading a human and toward pressuring-testing ideas against the literature: the Evaluator is an idea-observer that writes short, cited, prose observations about your ideas and how they held up (cognee.remember()), judged against your cross-session trajectory. There's no fabricated score to protect, and no rubric lookup table - the durable output is a grounded memory the graph refines over time, which is exactly what makes the Cognee layer load-bearing rather than decorative.

Lazy generation + self-healing sandbox

Visuals are never generated eagerly -> LEARN_NODE returns lesson text only; the (often expensive) HTML5 visual is generated on-demand only when the student opens the Visual tab. Once generated, it goes through a server-side syntax pre-flight (compile(script, '<visual>', 'exec')) before ever reaching the sandboxed iframe (sandbox="allow-scripts", srcdoc only, no external src -> fully self-contained), and a client-side runtime error triggers an automatic repair round-trip instead of surfacing a broken visualization to the student.

Structured output as the connective tissue

Every agent-to-agent and agent-to-frontend boundary is a strict=True JSON-schema Pydantic model, never free-text parsing. CerebrasClient._build_schema() inlines $refs and forces additionalProperties: false at every nesting level (Cerebras's strict mode rejects $ref), so a dozen independently-developed agents can compose without any of them needing a bespoke text parser for another agent's output.

Defense-in-depth over trust-the-first-answer

Several pipelines validate a generated artifact and fall back rather than assuming success: a cached curriculum graph is checked for structural validity and full document coverage before replay; a curriculum "cleanup" pass that would drop content is rejected in favor of the pre-cleanup tree; a blank/generic curriculum root label triggers a second, differently-framed LLM attempt before ever falling back to (much weaker) filename-derived naming.

🧱 Architecture

Three processes at runtime, inside one Electron shell - a React renderer, a FastAPI backend hosting every agent + the RAG store, and two persistence layers (ChromaDB for content, Cognee for load-bearing memory). The only cloud call is grounded Gemma 4 inference on Cerebras; web sources are optional and per-turn.

flowchart TB
  subgraph SHELL["🖥  Electron Desktop Shell"]
    direction TB
    FE["React + Vite Renderer - :5173<br/>Project Start Window · PDF Reader · Knowledge Graph · Wiki · Chat · Pair Buddy · Citation Graph"]
    subgraph BE["FastAPI Backend - :8765"]
      WS["WebSocket dispatch (BUILD_GRAPH · CHAT_TURN · COMMIT_PROJECT · …)"]
      AG["Agent roster<br/>Brain · Tutor · Evaluator(idea-observer) · Graph Curator<br/>Net Research · Draft · Wiki · Senses · Modality Router"]
      RAG[("ChromaDB - content RAG (per-project)")]
      MEM["StudentMemoryService"]
    end
  end
  FE <-->|"WebSocket + REST"| BE
  AG --> CB["LLM<br/>strict JSON schema + streaming"]
  MEM --> COG[("Cognee (local)<br/>LanceDB + SQLite · ~/.studybuddy/cognee")]
  AG -.->|"optional, per-turn"| NET["🌐 Tavily · arXiv · YouTube"]

Detailed component map (auto-generated)

┌────────────────────────────────────────────────────────────────┐
│                        Electron Shell                          │
│  ┌────────────────────────┬────────────────────────────────┐   │
│  │     React Frontend     │       FastAPI Backend          │   │
│  │     (Vite, port 5173)  │       (uvicorn, port 8765)     │   │
│  │                        │                                │   │
│  │  PDFReader ◄──────────►│  WebSocket /ws/{project_id}    │   │
│  │  InfiniteWiki          │  ├─ BrainAgent (curriculum)    │   │
│  │  ChatTool              │  ├─ TutorAgent (lessons)       │   │
│  │  FlashcardTool         │  ├─ NetResearchAgent (chat)    │   │
│  │  QuizTool              │  ├─ StudyBuddyAgent (Feynman)  │   │
│  │  PairBuddyTool         │  ├─ ReportAgent (canvas)       │   │
│  │  ReportView            │  ├─ WikiAgent / InfinityWiki   │   │
│  │  VisualSandbox         │  ├─ EvaluatorAgent             │   │
│  │  KnowledgeGraph        │  ├─ SensesAgent (vision)       │   │
│  │  EvaluationView        │  └─ ModalityRouter             │   │
│  │                        │                                │   │
│  │  Zustand Stores ──────►│  REST Routers                  │   │
│  │  (session, context,    │  ├─ /library   (per-session    │   │
│  │   interaction, graph)  │  │              upload+history)│   │
│  │                        │  ├─ /session   (create/commit/ │   │
│  │                        │  │              clear/trajectory)  │
│  │                        │  ├─ /regions   (figure detect)  │  │
│  │                        │  ├─ /annotations (margin notes) │  │
│  │                        │  ├─ /sandbox   (visual repair)  │  │
│  │                        │  ├─ /review    (Cognee recall)  │  │
│  │                        │  └─ /api       (health, keys)   │  │
│  │                        │                                 │   │
│  │                        │  Services                       │   │
│  │                        │  ├─ ChromaDB (per-session RAG)  │   │
│  │                        │  ├─ StudentMemoryService(Cognee)│   │
│  │                        │  ├─ MemoryService (report/traj.)│   │
│  │                        │  ├─ project_files/project_commit│   │
│  │                        │  ├─ LayoutService (PyMuPDF)     │   │
│  │                        │  ├─ OutputCache / JournalService│   │
│  │                        │  └─ AnnotationService           │   │
│  └────────────────────────┴───────────────────────────── ───┘   │
│                              │                                  │
│              ┌───────────────┼────────────────┐                 │
│    ┌─────────▼──────────┐    │    ┌────────────▼───────────┐    │
│    │          LLM       │    │    │   Cognee (local)       │    │
│    │                    │    │    │   LanceDB + SQLite     │    │
│    │                    │    │    │   ~/.studybuddy/cognee/│    │
│    └────────────────────┘    │    └────────────────────────┘    │
│                              │ (optional)                       │
│                    ┌─────────▼──────────┐                       │
│                    │  Tavily / YouTube  │                       │
│                    │  / OpenAlex        │                       │
│                    │  (Net Support,     │                       │
│                    │   Deep Dive,       │                       │
│                    │   Further Reading) │                       │
│                    └────────────────────┘                       │
└──────────────────────────────────────────────────────────────────┘

RAG & evidence architecture (structured-evidence v3)

Full detail: docs/architecture.md (authoritative, verified against code).

Three separate knowledge channels, never flattened into one blob — ContextComposer is the only code allowed to combine them, and it returns them as a labeled EvidenceBundle (source_evidence / project_memory / student_memory / citations / diagnostics):

Source evidence — what uploaded papers and bounded regions actually say. SQLite is canonical (EvidenceStore, FTS5 lexical index); ChromaDB (paper_evidence) is a searchable dense index, never the source of truth.
Project memory — what the student is investigating/deciding/noting inside one project. Project-filtered Chroma (project_memory).
Student memory — durable cross-project personalization. Cognee's research_profile dataset (see the Cognee internals section above).

Ingestion: AcademicDocumentParser (PyMuPDF4LLM + PyMuPDF) reconstructs reading order and section paths instead of splitting every N characters, producing three retrieval granularities — leaf evidence (paragraphs/formulas/tables/figures/captions), local windows (coherent adjacent evidence), and section cards (broad/graph-construction retrieval). Every EvidenceUnit gets a stable, deterministic evidence_id (hash of project_id + document_id + page + bbox + element_type + normalized_content), so re-parsing unchanged content is idempotent and graphs/memory/selections/citations can all anchor to the same durable ID.

Non-negotiable invariants (from docs/architecture.md):

Every evidence/project-memory operation is scoped by project_id; the same PDF in two projects produces different evidence IDs.
Citations are materialized from canonical EvidenceUnit records, never reconstructed from model prose or Chroma metadata alone.
Explicit text selections, bounding boxes, and evidence IDs outrank semantic search.
Assistant output alone cannot create persistent memory — project observations require verified evidence provenance; profile inference requires an explicit signal or cross-project recurrence.
Cognee/telemetry failures are visible but nonfatal to the study request; canonical evidence corruption or unsafe deletion is fail-closed.
Removing a paper or project cleans derived evidence before deleting the preserved source.

Citation graph & paper discovery architecture

Full detail: docs/citation-graph-and-paper-workflow-architecture.md.

Three deliberately separate surfaces, never conflated:

Durable project citation graph — uploaded project papers (paper:{file_id}, status="owned") and provider-verified citation edges between them. A durable edge requires provider-verified reference and matching reference text in the source PDF — never inferred from title similarity, search rank, or prose.
Transient exploration graph — external papers (external:{candidate_id}, identity resolved in priority order DOI → arXiv ID → Semantic Scholar ID → OpenAlex ID → normalized title/authors/year) surfaced while browsing a paper or author. Cached, but never project membership.
Paper recommendations — discovered/ranked external candidates from natural language (Chat/Pair Buddy). A recommendation becomes a project paper only after an explicit Download succeeds through the normal acquisition + ingestion path — never added to the durable graph just because it was displayed.

PaperMetadataService is the provider-neutral boundary merging OpenAlex, Semantic Scholar, and arXiv into one PaperMetadata record (identifiers, citation counts/references, destinations, open-access facts, field-level provenance/confidence) — no raw provider response shape leaks past it. Google Scholar is a fallback destination for a human to inspect, never a citation authority or PDF source.

WebSocket event reference

All messages are { "type": str, "data": dict }. Dispatch lives in backend/app/websockets/handlers.py.

Event	What it does
`BUILD_GRAPH`	Streams the curriculum graph: root+sections, then parallel per-section expansion (`GRAPH_NODE_ADDED`/`GRAPH_EDGE_ADDED`) → `GRAPH_BUILD_DONE`
`LEARN_NODE`	RAG fetch scoped to the node's source paper(s) → streamed lesson (`LESSON_TOKEN`* → `LESSON_DONE`)
`CHAT_TURN`	RAG fetch → optional Net Support web research → streamed answer (`CHAT_TOKEN`* → `CHAT_DONE`)
`CONTEXT_CARD_REQUEST`	Infinite Wiki: resolves a page for the selected term → `WIKI_PAGE` → `WIKI_DONE`
`WIKI_DEEPDIVE_REQUEST` / `WIKI_DEEPDIVE_SUMMARIZE`	Deep Dive: YouTube search + transcript summarization
`FLASHCARDS_REQUEST` / `QUIZ_REQUEST`	Generate from question/content chunks → `FLASHCARDS_READY` / `QUIZ_READY`
`FLASHCARD_GRADE` / `QUIZ_SUBMIT`	Journal append (+ `QUIZ_FEEDBACK` for quiz)
`STUDY_BUDDY_INIT` / `STUDY_BUDDY_TURN` / `STUDY_BUDDY_AUDIO`	Pair Buddy/Feynman turns (text or transcribed audio)
`TRANSCRIBE_AUDIO` / `TRANSCRIBE_AUDIO_CHUNK`	Dictation-only STT: transcript is inserted into the draft, never auto-sent
`TTS_START` / `TTS_CANCEL` / `TTS_PLAYBACK_STATUS`	Voice output playback control
`REPORT_COMPILE` / `REPORT_CLOSE`	Report Canvas: process/pool annotations into a synthesized report; flush the per-PDF cluster on close
`COMMIT_PROJECT`	"Push": idea-observer evaluation (trajectory-aware) → `NODE_ASSESSMENT`; Graph Curator spawns exploration nodes (`GRAPH_NODE_ADDED`); project snapshot committed; Cognee push/flush + persona/citation-graph/intelligence refresh run in the background → `EVALUATION_DONE`
`CLOSE_PROJECT`	Final wrap: re-evaluates the session, writes the Markdown summary export → `SESSION_COMPLETE`
`UPDATE_NODE_STATUS`	Student manually marks a node `ongoing`/`completed`
`STYLE_FEEDBACK`	Explicit learner style preference → written to profile memory
`CACHE_CLEAR`	Dev: clears the deterministic output cache

📂 Project Structure

Click to expand

StudyBuddy/
├── electron/                  # Electron main process, backend spawn, preload IPC bridge
├── backend/
│   ├── app/
│   │   ├── agents/            # AI agent layer
│   │   │   ├── brain_agent.py         # Project graph extraction, memory-shaped structure, titles
│   │   │   ├── tutor_agent.py         # Grounded lessons, inline quiz/flashcards, visual generation
│   │   │   ├── net_research_agent.py  # Chat query decomposition + per-entity web research
│   │   │   ├── study_buddy_agent.py   # Pair Buddy / Feynman-style Socratic turns
│   │   │   ├── report_agent.py        # Deferred Report surface: note→insight/report synthesis
│   │   │   ├── wiki_agent.py          # Wiki card generation
│   │   │   ├── infinity_wiki_agent.py # Deep Dive: YouTube search + transcript summarization
│   │   │   ├── evaluator_agent.py     # Idea-observer: cited prose observations, trajectory-aware
│   │   │   ├── graph_curator.py        # Spawns "exploration" nodes for genuinely-developed topics
│   │   │   ├── draft_agent.py          # Draft Review backend: claim verification against project corpus
│   │   │   ├── senses_agent.py        # Vision model (figure/table description)
│   │   │   ├── modality_router.py     # Routes concepts/reports to visual type
│   │   │   ├── cerebras_client.py     # Cerebras SDK wrapper (structured + streaming)
│   │   │   └── cerebras_errors.py     # Error classification & rate-limit handling
│   │   ├── memory_skills/      # Cognee SKILL.md content (research-memory-reviewer)
│   │   ├── rag/               # ChromaDB vector store, embeddings, chunker
│   │   ├── schemas/           # Pydantic contracts (project, graph, journal, annotations)
│   │   ├── services/          # Business logic
│   │   │   ├── project_service.py     # Project registry and metadata
│   │   │   ├── project_files.py       # Per-project paper add/remove and upload isolation
│   │   │   ├── project_commit.py      # Project snapshot commit (history + resume)
│   │   │   ├── student_memory.py      # Cognee-backed cross-session memory (load-bearing)
│   │   │   ├── memory_service.py      # Disk-only report clusters + idea trajectory (NOT Cognee)
│   │   │   ├── journal_service.py     # In-memory per-project interaction journal
│   │   │   ├── citation_graph.py      # Uploaded-paper citation graph storage and refresh
│   │   │   ├── recommendation_service.py # Paper recommendations from project/citation/memory context
│   │   │   ├── scholar_service.py     # OpenAlex paper lookup
│   │   │   ├── youtube_service.py     # YouTube search + transcript fetch
│   │   │   ├── transcription_service.py # Local whisper.cpp speech-to-text
│   │   │   ├── voice_activity_service.py # Optional backend VAD preprocessing
│   │   │   ├── voice_benchmark_logger.py # JSONL voice benchmark log paths/helpers
│   │   │   ├── transcript_diagnostics.py # Repetition-loop diagnostics for STT output
│   │   │   ├── layout_service.py      # PyMuPDF page segmentation (figures/tables)
│   │   │   ├── output_cache.py        # Deterministic cache for LLM outputs
│   │   │   ├── annotation_service.py  # Margin-note CRUD + persistence
│   │   │   ├── graph_state.py         # Graph state manager (ongoing/completed node state)
│   │   │   ├── visualization_service.py # Grounded visual generation boundary
│   │   │   ├── agent_control.py       # Editable project/learner/skill settings
│   │   │   └── summary_writer.py      # Commit summary Markdown export
│   │   ├── routers/           # FastAPI REST endpoints
│   │   │   ├── projects.py         # Project CRUD, add/remove files, project detail
│   │   │   ├── library.py          # Project history/resume and legacy library endpoints
│   │   │   ├── session.py          # Session create / ingest-status / trajectory / clear
│   │   │   ├── regions.py          # Figure/table segmentation
│   │   │   ├── annotations.py      # CRUD for margin notes
│   │   │   ├── sandbox.py          # Visual self-repair endpoint
│   │   │   ├── review.py           # Cognee-backed review query
│   │   │   ├── citation_graph.py   # Citation Graph API
│   │   │   ├── chat.py             # Paper recommendation endpoint
│   │   │   ├── draft.py            # Draft verification API
│   │   │   ├── control_room.py     # Memory/artifact/status inspection
│   │   │   ├── agent_control.py    # Editable ResearchMate settings API
│   │   │   └── health.py           # Readiness, memory graph, provenance, export
│   │   └── websockets/
│   │       └── handlers.py         # Central event dispatch (BUILD_GRAPH, CHAT_TURN, etc.)
│   └── tests/                 # pytest suite
├── frontend/
│   └── src/
│       ├── components/
│       │   ├── graph/              # KnowledgeGraph, ConceptNode, CitationGraphView
│       │   ├── panel/              # ScientificFigurePanel, InfiniteWiki, VisualSandbox, future/grayed panels
│       │   ├── reader/              # PDFReader, HighlightLayer, RegionLayer, MarginGutter
│       │   ├── study-tools/         # ChatTool, inline quiz/flashcards, PairBuddyTool, VoiceTool shell
│       │   ├── overlay/             # FloatingToolbar (cursor mode switcher)
│       │   └── init/                # ProjectStartWindow, ResearchMate Settings, ResumeCard
│       ├── hooks/                  # useWebSocket
│       ├── lib/                    # fileSystem (Electron IPC + browser fallback), clearSession
│       ├── pages/                  # TreePage (graph, Citation Graph, Commit), ManualPage (reader workspace)
│       ├── store/                  # Zustand: graphStore, sessionStore, contextStore, interactionStore
│       └── types/                  # Shared TypeScript interfaces
└── package.json               # Root orchestration (concurrently, electron-forge)

Running Tests

# Full backend suite
cd backend
uv run pytest tests/ -v

# Single test file
uv run pytest tests/test_tutor_lesson.py -v

# Frontend tests
cd frontend
npx vitest run

🔑 Environment Variables

Variable	Required	Description
`CEREBRAS_API_KEY`	✅	Cerebras Cloud API key for Gemma 4 inference
`TAVILY_API_KEY`	❌	Enables "Net Support" knowledge mode (web search fallback)
`YOUTUBE_API_KEY`	❌	Enables Deep Dive video search
`ALLOWED_ORIGINS`	❌	CORS origins (defaults to `http://localhost:5173`)
`STT_MODEL_PATH`	❌	Explicit whisper.cpp model file path. Defaults to pywhispercpp's local `ggml-base.en-q5_1.bin` path
`STT_VAD_ENGINE`	❌	Live backend VAD engine for dictation: `none` (default), `ten`, or `silero`
`VOICE_LOG_TRANSCRIPTS`	❌	When `false`, benchmark rows keep transcript length but omit full transcript text
`VITE_BROWSER_VAD_ENABLED`	❌	Frontend browser VAD UX toggle. Defaults to enabled; it never trims uploaded audio
`DEPLOYMENT_ENV`	❌	`desktop` (default) uncaps concurrency (e.g. 50 concurrent Cerebras calls) for local hardware. `demo` throttles internal semaphores severely (e.g. max 5 concurrent Cerebras calls) to survive constrained hosts like Hugging Face Spaces without OOM/429s

Local data layout (~/.studybuddy/)

Everything below lives under ~/.studybuddy/ — nothing leaves the machine except Cerebras inference calls and, optionally, Tavily/YouTube/scholar-provider lookups.

Path	Owner	What
`projects/registry/`	`project_service.py`	Project metadata registry
`projects/uploads/{project_id}/`	`project_files.py`	Per-project uploaded files, isolated per project
`projects/intelligence/`	`project_intelligence.py`	Auto-updated project brief, refreshed on Commit
`projects/activity/`	`project_activity.py`	Project activity feed (Control Room)
`pdfs/{file_id}.pdf`	`projects.py` / `library.py`	Content-addressed PDF cache, keyed by file SHA-256
`graphs/{project_id}.json`	`graph_state.py`	Live graph state (`ongoing`/`completed` nodes)
`sessions/{project_id}.json`	`project_commit.py`	Commit snapshot / Project History
`evidence/`	`evidence_store.py`	Canonical `EvidenceStore` (SQLite + FTS5)
`regions/{project_id}/{document_id}.json`	`evidence_ingestion.py`	Detected/manual figure & table regions
`annotations/{document_id}.json`	`annotation_service.py`	Margin notes
`chroma/`	`chromadb_client.py`	ChromaDB (`paper_evidence`, `project_memory` collections)
`cognee/`	`student_memory.py`	Cognee LanceDB + SQLite (`research_profile` dataset)
`memory_integrity/`	`student_memory.py`, `memory_promotion.py`	Promotion-decision ledger, pending memory staging, temporal ledger
`citation_graphs/`	`citation_graph.py`	Durable per-project citation graph
`citation_exploration/`	`citation_exploration_store.py`	Transient paper/author exploration cache
`recommendations/`, `recommendation_clarifications/`	`recommendation_service.py`	Paper recommendation + clarification cache
`paper_metadata/`	`paper_metadata.py`	Resolved provider metadata cache (OpenAlex/Semantic Scholar/arXiv)
`cache/`	`output_cache.py`	Deterministic LLM output cache
`agent_control/`	`agent_control.py`	Editable project/learner/skill settings (persona engine)
`summaries/`	`summary_writer.py`	Commit summary Markdown export
`visual_lessons/`	`visual_lesson_store.py`	`/visualize` composition/chart engine cache
`models/`, `voice/`	`tts_service.py`, `voice_selection.py`	Local STT/TTS model files and voice defaults
`observability/`	`observability/config.py`	SigNoz/OTel artifact export root

Research Companion Mode

Roadmap

ResearchMate is moving toward a full long-horizon research companion: Draft Review, Personal Graph, Viva, richer paper recommendation, and full Voice Buddy are planned or partially scaffolded.

The current v1 focuses on the working core: project-based paper reading, grounded Wiki and Chat, Pair Buddy, Citation Graph, Commit, Cognee-backed memory, and sandboxed visuals.

Tech Stack

Layer	Technologies
Desktop shell	Electron, Electron Forge
Frontend	React 19, TypeScript, Vite, Zustand
UI and reading surfaces	React PDF, React Flow (`@xyflow/react`), lucide-react, KaTeX, Plotly, Three.js, Mermaid
Backend API	Python 3.11, FastAPI, Uvicorn, Pydantic, WebSockets
Agent and LLM layer	Cerebras Cloud SDK, Gemma 4, structured Pydantic outputs, streaming completions
Retrieval and document ingestion	ChromaDB, LangChain text splitters, PyPDF, python-docx, PyMuPDF, pymupdf4llm
Long-term memory	Cognee 1.2.2, LanceDB, fastembed local embeddings
Web and media integrations	Tavily, OpenAlex-style scholar lookup, YouTube Data API, youtube-transcript-api
Speech and multimodal support	pywhispercpp / whisper.cpp, TEN VAD, Silero VAD, Gemma 4 vision path
Testing and quality	pytest, pytest-asyncio, respx, Vitest, Testing Library, oxlint
Local-first storage	`~/.studybuddy/` project registry, uploads, annotations, graphs, summaries, Cognee memory, and activity logs

Cerebras API & structured output contracts

Model ID always pinned explicitly: gemma-4-31b (never omitted — the SDK may otherwise resolve a retired default). Vision calls (SensesAgent) must also use gemma-4-31b, not llama-4-scout.
Context: 32K MCL (message context limit) / 65K MSL.
Structured outputs: response_format.type = "json_schema", strict: true, additionalProperties: false at every nesting level. CerebrasClient._build_schema() inlines $refs (Cerebras strict mode rejects $ref) and fails fast locally, naming the offending field, if a schema uses a bare dict[str, X] (strict mode requires explicit properties; reshape to list[SomeKeyValueModel] instead).
Image input is multimodal image_url content, base64 data URIs only — hosted URLs aren't supported.
Reasoning is off by default; opt in per call with reasoning_effort: "low"|"medium"|"high".
CerebrasClient.structured_complete() catches both json.JSONDecodeError and pydantic.ValidationError (Cerebras can return truncated/EOF'd JSON) and retries once.
Every agent-to-agent and agent-to-frontend boundary is one of these strict Pydantic contracts, never free-text parsing — a dozen independently-developed agents compose without any of them needing a bespoke parser for another agent's output.

License

Apache 2.0

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
assets		assets
backend		backend
electron		electron
frontend		frontend
.dockerignore		.dockerignore
.gitattributes		.gitattributes
.gitignore		.gitignore
.npmrc		.npmrc
AGENTS.md		AGENTS.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
package-lock.json		package-lock.json
package.json		package.json
vercel.json		vercel.json

Latest commit

History

Folders and files

Repository files navigation