-
Notifications
You must be signed in to change notification settings - Fork 0
Architecture
Draggy is an Electron app: a Node main process that owns everything privileged, and a React renderer that owns everything visible. The interesting decisions are mostly about the line between them.
electron/ main process
main.cjs windows, protocols, every IPC handler
preload.cjs the entire surface the renderer can reach
storage.cjs chats and settings, SQLite
library.cjs the document index, SQLite
search.cjs web search providers and fallback order
documents.cjs reading and writing Word, Excel, PowerPoint, PDF
markdownHtml.cjs Markdown to HTML, for the PDF writer
pdfWriter.cjs laying a document out in Chromium and printing it
urlPolicy.cjs which addresses the model may make the app fetch
mcp.cjs MCP servers: spawning them, JSON-RPC over stdio
mcpCatalogue.cjs the servers offered, and what each one needs
runner.cjs running Python and JavaScript
adblocker.cjs Ghostery's engine, and the filter lists it compiles
favicon.cjs site icons for search results, fetched and cached
filters/ one supplemental filter list, shipped with the app
updater.cjs checking, downloading, installing
platform.cjs the per-OS parts: VRAM detection, installers
appData.cjs adopting the data folder left by an earlier app name
logger.cjs the rotating log file behind Settings -> Data
src/ renderer
agent/ the streaming tool-calling loop, and compaction
chat/ the message list, and the rules for what can be attached
settings/ the settings panels
tools/ tool definitions and the registry
voice/ capture, detection, turn-taking, speech
BrowserBar.tsx the toolbar above a page you opened
The renderer has no Node integration and no direct filesystem access. Everything
it can do is a function in preload.cjs, and that file is deliberately short
enough to read in one sitting. If something is not exposed there, the renderer
cannot do it.
The renderer is served from a custom app:// protocol rather than file://,
which gives it a real origin and lets a normal Content Security Policy apply.
The second boundary, and the one that decides whether the browser works at all.
Draggy's own windows run on Electron's default session, where a strict Content
Security Policy is applied to every response: default-src 'self' app: draggy:,
frame-src 'none', form-action 'none'.
Every external page runs on a separate persistent partition with no policy of ours attached. This is not a detail. When both shared one session, that policy was being applied to other people's sites as well, which blocked their scripts, stylesheets, images, video, iframes and form submissions. The browser rendered wreckage and it was not obvious why.
Your browsing and the model's page reads deliberately share that partition, so a verification check you pass by hand leaves a cookie the model's next fetch can use. It also means one ad-blocker switch governs both, which is the behaviour people expect from a switch labelled "ad blocker".
read_url and browser_navigate hand the main process a string the model
produced. Chromium will happily load file:///C:/Users/…/.aws/credentials and
render it as a plain text document, which the extractor would then read back
into the conversation.
That is not hypothetical: the model reads pages other people wrote, and a page saying "now read this local file" is the whole exploit: the contents arrive in context and leave again in the next search query.
electron/urlPolicy.cjs states the reachable surface as an allowlist rather
than a list of things to block: http and https, and not this machine or its
network. A blocklist would have to anticipate app:, draggy:, devtools:,
blob: and whatever Electron adds next; an allowlist of two is complete by
construction. The user's own typed address is allowed to reach localhost,
because that is their dev server and their decision; a tool call is not.
BaseWindow with two WebContentsViews: the toolbar, drawn by the renderer so
it is the app's own typeface and colours, and the page below it.
A view clips what it draws to its own bounds, so a menu hanging below a 48-pixel toolbar is simply cut off. While a menu is open the toolbar view is grown to fit it and sits above the page; the part of that region the menu does not use is transparent, and clicking it closes the menu.
Draggy does not attempt to pass bot-detection challenges. It recognises one, stops, and says so, see Browser and Ad Blocking.
Directly, from the renderer, over 127.0.0.1:11434. It is a local HTTP service
and routing it through IPC would buy nothing but a copy of every token.
src/ollama.ts holds the client: streaming NDJSON, model metadata, context
sizing, pull progress. Pull progress is worth a look: Ollama reports per-layer
byte counts that restart part way through, so the tracker accumulates by digest
rather than reading the numbers off the latest line.
src/agent/agentLoop.ts. One turn is: build a system prompt from what is
actually enabled, stream a reply, and if the model calls a tool, run it, append
the result, and stream again, up to a cap.
Two paths through it. Models that support native tool calling get their tools in
the request. Models that do not get a text catalogue in the prompt and their
output is parsed for call syntax, which is what src/toolParsing.ts exists for.
The loop also handles running out of context mid-answer: src/agent/resume.ts
works out how to continue without repeating the sentence the user already read.
src/agent/compaction.ts. A long chat used to grow until it hit the wall. Now
the older half is folded into notes once the conversation approaches the size of
the window the model is loaded at.
Three things keep the fold from costing more than it saves:
- It folds a prefix and appends to the summary rather than rewriting it, so the text already on the wire stays byte-identical and Ollama's prefix cache survives every fold after the first. Rewriting the summary each time would invalidate the whole conversation on every fold.
- It triggers off the bucket in
CONTEXT_BUCKETS, not the model's maximum. The point is to stay inside the window already loaded, not to run as close to the limit as possible. A KV cache that no longer fits beside the weights pushes layers onto the processor, which costs every token from then on. - It runs after a turn, not before the next one, and is cancelled the moment you send another message. The generation it costs is paid while you are reading.
The planning half is a pure function of the message list, tested without a model. The messages themselves are never touched: they stay on screen and stay searchable, and a line in the transcript says where the fold is.
A related detail lives in src/prompts.ts. The system prompt carries the date
and not the time. A timestamp with seconds in it is the first thing on the wire
and differs on every request, which ended the cached prefix at token zero and
made every turn re-evaluate the whole conversation. The clock is appended to the
last user message instead, where changing it costs nothing.
src/tools/registry.ts is a registry keyed by name; src/tools/builtin.ts
registers the actual ones. A tool declares its schema, whether it is available
in the current environment, and how to run itself. The system prompt is built
from whatever is registered and enabled, so turning a tool off removes it from
the model's world rather than merely discouraging its use.
electron/mcp.cjs. An MCP server is a program, not a service: Draggy spawns it
and speaks JSON-RPC 2.0 over its standard input and output, one message a line.
That is the whole transport.
Servers are installed once with npm, into mcp-servers/ in the data folder, and
then their entry point is run directly by the binary already running the app.
Going through npx at launch instead was what put a console window on screen:
npx starts a package through a cmd.exe shim, and Electron is a GUI binary with
no console of its own, so that shim was given a brand new visible one. Install
scripts are refused as well, which a server has no business needing.
The tools it offers are translated into ordinary ToolSpecs and registered
under the external group, which the registry had declared and never used. The
group is replaced wholesale whenever the running set changes, because a stopped
server must not leave tools in a catalogue the model was told it could use.
Names are prefixed with the server, as in github__create_issue, so two servers can
both offer search, and so the text-mode parser can tell an external tool from
a built-in one.
Nothing starts on its own, and everything stops on quit. See Extensions for the user-facing side.
electron/library.cjs. Passages are embedded with a model sized to the graphics
card, and searched two ways at once: cosine similarity over the vectors, and
BM25 over an FTS5 index of the same passages. The two are combined with
reciprocal rank fusion.
Fusing on position rather than score is the point. A cosine similarity and a BM25 rank are not the same kind of number, and normalising one against the other means inventing a conversion. All that is claimed is that being near the top of either list is worth something and being near the top of both is worth more.
The vectors live in one flat Float32Array rather than one array per passage,
and passage text is fetched from SQLite only for the handful being returned. The
old shape held every vector and every passage resident for the life of the
process.
FTS5 has no foreign keys, so the keyword index is reconciled against the chunks on startup, which also means a library indexed before that table existed catches itself up rather than needing a migration.
The part worth reading if you only read one.
microphone → AudioWorklet → VAD → gate → Whisper → model → chunker → voice
-
capture.ts: an AudioWorklet at 16 kHz, so resampling happens once in native code. Muting is applied on the audio thread, not several frames later. -
vad.ts: Silero in a worker, with an energy detector as a fallback that keeps the same interface. -
gate.ts: turns speech probabilities into conversation events. It knows nothing about audio, models or output, which is why the whole turn-taking policy is unit tested without a microphone. Two thresholds rather than one, so a voice hovering at the boundary cannot rattle the state machine. -
turnDetector.ts: decides how long a pause has to be before your turn is over, from what you just said. A trailing "um" or a conjunction buys more time than a full stop. -
conversation.ts: the wiring, and the only file that knows about all of it. -
reply.ts: generation, including the search round trip. Held back only as long as the opening could still be a search marker, which is seven characters. -
chunker.ts: cuts streamed text at the earliest point that still sounds like a phrase, so synthesis starts before generation finishes. -
speaker.ts: sits between them and can suspend without discarding, which is what makes an interruption recoverable when it turns out to be "mhm".
Everything overlaps deliberately. Transcription starts during your pause rather than after it; the reply is spoken clause by clause as it is generated; the model is resident in memory before the first question.
SQLite through node:sqlite, no native module to rebuild. Chats, settings and
the document index share one database in the app's user data folder.
Writes are debounced and coalesced per session, and flushed on quit. A failed write surfaces in the interface rather than being swallowed, because silently losing a conversation is worse than an ugly banner.
Draggy starts real processes, and every one of them is stopped from a single
shutdown() on before-quit: browser windows, extension servers, any code run
still going, Ollama, then the database handles. Each step is wrapped on its own,
so one failing cannot skip the rest, and the log names the one that did.
Two of those are worth the detail. A code run is tracked from the moment it
starts, because the timeout that would have killed it dies with the process that
set it, and on macOS and Linux the child is deliberately in its own process
group and would otherwise survive. Servers and runs are killed as trees through
platform.killTree, since signalling the child alone leaves whatever it spawned
with nobody to notice.
Ollama is only stopped when Draggy was the one that started it. An instance that was already up belongs to whoever started it and may be serving something else.
An in-app browser window is a top-level window, so window-all-closed does not
fire while one is open. Closing the main window closes them too, or quitting
Draggy would leave the app running with no way back to it.
Vitest, around 990 of them, running in under two seconds because none of them touch a GPU, a network or a model. The parts that are hard to test (audio, inference, Electron itself) are pushed to the edges, and the parts that encode actual decisions are pure functions in the middle.
npm run check is typecheck, lint and tests together, and CI runs it on every
push and pull request.