Building a local-first Deepnote in one autonomous run — a test of roadmap-level resolution in an agentic dev harness

[muunkky]

Building a local-first Deepnote in one autonomous run — a test of roadmap-level resolution in an agentic dev harness

Deepnote's open-source toolchain already runs a .deepnote project end-to-end against a local
Python kernel — but only headlessly. deepnote run notebook.deepnote executes the whole
project and prints outputs to your terminal. The one way to get the interactive Deepnote
experience — render the notebook, edit a cell, hit run, watch a plot appear, change an input and
have dependents re-run — is deepnote open, which uploads your project to Deepnote Cloud.

There is no local, interactive, in-browser way to read and run a .deepnote project against the
open runtime. That's the feature this milestone builds. But the feature is also the payload for
an experiment — see "The experiment" below.

$ cd ~/work/quarterly-analysis
$ deepnote serve quarterly.deepnote
Resolving Python interpreter... /Users/sam/.venvs/analysis/bin/python
Starting deepnote-toolkit server... ready
Deepnote UI running at http://localhost:9876  (Ctrl-C to stop)

A browser opens. You see your notebooks in a left rail and the active one rendered top to
bottom — markdown as prose, code with its last-saved plot or dataframe, a SQL block with its
result table, a big-number tile, an input at its current value. Nothing has been uploaded.
Nothing left localhost.

What it is, in three layers

1. A headless runtime server (@deepnote/runtime-server) — runtime-core's existing
   execution engine exposed over a documented HTTP + WebSocket API: open a project, list
   notebooks/blocks, run block(s), stream block-start / output / block-done events live,
   and save back to disk. No UI required to use it — it's scriptable on its own.
2. A one-command launch — a new deepnote serve CLI command, a natural sibling of
   deepnote run, that boots that server over a local project and (optionally) opens a browser.
3. A viewer SPA — a React app that renders every block type and every Jupyter output MIME
   type the format supports, the browser counterpart to the terminal output renderer.

Why it's useful even though Deepnote builds this closed-source

The hosted Cloud UI is excellent — and closed. This is the complement, not a competitor:

• Local-first and offline. The full open → view → run → save loop works with the network
  disabled; zero outbound requests beyond localhost. That's the whole point for anyone who
  can't upload — regulated industries, client-confidential data, air-gapped environments — or
  who just wants a fast local loop.
• Open, self-hostable, inspectable. It runs against the open runtime you already have, with
  no account and no upload path.
• Upstream-contributable. The server + deepnote serve are a natural extension of the
  existing CLI, and they're the piece offered back to the project. (More on the boundary below.)

Why this is genuinely hard

The ambition isn't "wrap an engine in a web server." The real surface is parity with a closed
product plus the concurrency and durability problems a live, stateful, single-kernel session
forces. Among the load-bearing pieces:

• Execution/output parity with deepnote run. The API must stream byte-identical Jupyter
  IOutputs to what the headless CLI produces — across 100% of executable block types — and
  preserve the CLI's typed failure categories (missing-kernel / kernel-launch /
  kernel-died / in-block) rather than flattening them to strings.
• Run serialization over WebSocket. The server fronts one kernel via one sequential
  engine, but a UI is inherently concurrent (mash Run, drag a slider, fan out a reactive chain).
  Issue two runs naively and you interleave two output streams onto one handler and corrupt
  per-block attribution. The fix is a single-concurrency FIFO run queue with a structurally
  guaranteed terminal event for every run — including the common case where a block fails and
  the engine stops early — so no consumer ever hangs waiting on blocks that will never run.
• Save you can trust. Save is the single most data-loss-prone operation in the product, so
  the bar is precise: semantic round-trip + idempotence, not byte-equality (the serializer
  canonicalizes YAML, so the first save reformats and is stable forever after). Atomic
  write-temp-then-rename, plus external-change detection that refuses to clobber a file edited out
  from under you. This gate ships before any editing UI exists.
• No-kernel project viewing. Opening a project is pure deserialization — so the viewer can
  render a notebook's persisted state with no kernel running at all, and a missing kernel surfaces
  as a typed capability flag on run, not as an opaque boot crash.
• A wedge that slices clean. The whole thing has to decompose into a code-only diff with
  zero frontend footprint, sliceable by directory off upstream main — which drives the package
  layout (a publishable server package; the SPA in a fork-only apps/ tier importing only a
  Node-free types entry) and introduces the first frontend toolchain this monorepo has ever
  had, isolated so it can't red the backend's gate.

The honest upstream boundary

Deepnote's epic deepnote/deepnote#162
explicitly assigns the interactive notebook UI to the VS Code extension repo, and the Cloud UI is
closed-source. So the split is deliberate: the server + deepnote serve wedge sits squarely
inside deepnote#162's stated CLI/runtime ownership and is the upstream-contributable piece; the SPA
overlaps an assigned surface and stays a fork-only showcase of what the open backend makes
possible. Backend-first, so the shareable part can be offered early — never pushed unsolicited.

---

The experiment

Here's the actual claim being tested, and it's a bigger one than "I used AI to build a feature."

The hypothesis: a single autonomous execution of the gitban development harness can take a
milestone from a one-line roadmap entry all the way to a reviewed PR — producing every artifact
along the way — driving itself through the whole lifecycle, with a human stepping in only for the
tasks that genuinely require one.

That means one run generates and executes the whole lifecycle:

• the roadmap decomposition (milestone → sprints → projects → features → cards),
• the PRD, the ADRs, the design docs, and the test plans,
• and then the engineering itself — the server, the CLI command, the SPA, the tests — built
  card by card by an autonomous dispatcher,
• finally sliced into a PR.

The planning is not a human deliverable that an agent then implements. The planning, the
architecture decisions, the test strategy, the UI, and the code are all outputs of the same run.
My role is to unblock the things an agent can't self-serve — credentials, external approvals, the
occasional human-required gate — not to write the artifacts. The point I'm trying to demonstrate
is roadmap-level resolution: that the harness can hold and resolve a unit of work as large and
as deep as an entire milestone — not a function, not a file, not a ticket, but five sequenced
sprints of interdependent backend and frontend — and drive it coherently to completion on its own.

---

The full M3 roadmap

The milestone decomposes into five sequenced sprints → nine projects → thirty-two features. Each sprint depends on the one before it. Sprint 1 — the server + deepnote serve wedge — is the only upstream-contributable tier; Sprints 2–5 are the fork-only browser SPA showcase (per the deepnote#162 boundary above). Everything below was generated by the harness from the one-line milestone entry.

The master spec is PRD-003 · Local Deepnote UI. Each sprint links its own design doc / ADRs below; all of it lives on the milestone/m3-local-ui branch under docs/.

Sprint 1 — Headless runtime server + one-command launch  ·  the upstream wedge · no dependencies

📄 Design: m3-s1 · server API and serve  ·  📐 ADR-007 · server/SPA package layout

Expose runtime-core's existing headless execution over a stable, documented HTTP + WebSocket API — open a project, list notebooks/blocks, run block(s), stream events, save — with no UI required, plus a deepnote serve command that boots it over a local .deepnote project. Reuses ExecutionEngine + interpreter/kernel resolution exactly as run.ts composes them. This is the piece offered upstream on deepnote#162.

• HTTP + WebSocket server over runtime-core — a new server package wrapping ExecutionEngine; carries the save-safety gate (ships before any editing UI exists).
  • Server package scaffold — new packages/* server package, located so it slices clean of the SPA; HTTP for request/response, WebSocket for the execution event stream, zero frontend dependency.
  • Open project + list notebooks/blocks API — GET /api/project returns project metadata + the notebook/block tree, reusing deserializeDeepnoteFile and kernel resolution as run.ts does.
  • Execute block(s) + stream events over WS — run via ExecutionEngine; deliver every block-start / block-done / output event over the WS in order, preserving the run.ts failure-category discriminants (missing-kernel / kernel-launch / kernel-died / in-block).
  • Save API (semantic round-trip + idempotence) — POST /api/project/save via serializeDeepnoteFile; deep-equal round-trip + idempotent no-op save — byte-equality is explicitly not the bar.
  • Server integration tests — open, run, stream outputs identical to deepnote run, and save with semantic round-trip fidelity; the documented launch criteria for the wedge.
• deepnote serve / deepnote ui command — the CLI that boots the server over a local project, a natural sibling of deepnote run.
  • deepnote serve command — canonical command; port selection with fall-back to the next free port and reported URL, --port / --no-open, clean startup/ready/stop logging, Ctrl-C shutdown.
  • deepnote ui browser-opening alias — thin alias that defaults --open.
  • SQL / integration parity with run — local SQL/integration support to the extent run.ts already has it; any network-backed feature off by default to preserve the local-first guarantee.
• Slice the wedge as a contrib diff + showcase post — the wedge's delivery.
  • Cut the clean contrib/* diff off upstream/main — code only, no .gitban / .claude / docs; the exact diff we'd open against deepnote/deepnote once invited.
  • Post the milestone update to the fork showcase thread — link the contrib diff and the process diff, label upstream-ready vs fork-only (the No way to share selected env between VSC extension and MCP deepnote/deepnote#288 pattern: showcase and ask before pushing).

Sprint 2 — Open & view a notebook locally  ·  read-only viewer SPA · depends on S1

📄 Design: m3-s2 · viewer  ·  📐 ADR-006 · SPA framework & bundler

The read-only viewer. Introduces the first-ever UI framework + browser bundler to the monorepo, an app shell that loads a project over the S1 API, and read-only DOM renderers for every block type plus Jupyter output — the browser counterpart to the terminal output renderer. No execution, no editing yet: it renders the project's persisted state in a Cloud-like view.

• SPA foundation: framework, bundler, app shell
  • UI framework + bundler introduced to the monorepo — the workspace has zero frontend tooling today (Biome / Prettier / vitest / tsc only); touches CI, lockfile, lint/format config, and the spell-check dictionary, kept isolated so it can't leak into the cleanly-sliceable backend.
  • App shell + routing — a left-hand notebook list and the active notebook rendered top-to-bottom, with routing between notebooks.
  • Project load over the S1 API into app state — fetch GET /api/project and load the notebook/block tree into application state, ready to render.
• Read-only DOM block renderers — the SPA's real surface area; rich outputs are where Cloud-likeness lives.
  • Code / markdown / text renderers — code (syntax highlighting + last-saved outputs), markdown (formatted prose), text.
  • SQL renderer — the query plus its persisted result table.
  • Visualization / big-number / image renderers — vega/plotly natively or fall back to the persisted image, big-number tiles, image blocks.
  • Input / button / separator renderers — the eight input kinds (text / textarea / checkbox / select / slider / date / date-range / file) showing current values, buttons, separators.
  • Jupyter IOutput MIME renderer — DOM rendering of stream / display_data / execute_result / error; the browser counterpart to the terminal output renderer, shared by all output-bearing blocks.
  • Unknown-type safe fallback — raw content + type label rather than crashing the notebook view.

Sprint 3 — Run blocks with live streamed output  ·  live execution UI · depends on S1, S2

📐 ADR-005 · browser↔kernel transport (proxy vs direct)

Run-this-block / run-all wired to the S1 WebSocket server; kernel outputs stream live into the S2 renderers; execution state and counts tracked; kernel-failure banners surfaced.

• Live execution wired to the WS server
  • Run-this-block / run-all controls — UI controls (Run / Shift+Enter, run-all) dispatching execution requests to the server.
  • WebSocket output streaming into renderers — consume the WS event stream and stream outputs live into the renderers in place, no reload; targets < 2 s in-place output on a warm kernel (measured — load-bearing for the transport ADR).
  • Execution state + counts — per-block running/idle indicators and an execution count that increments per run.
  • Kernel-failure banners — actionable banners driven by the run.ts failure categories (e.g. "deepnote-toolkit not installed — pip install deepnote-toolkit[server]") rather than a blank cell; block exceptions render their traceback in place and mark the block failed.

Sprint 4 — Edit blocks & save safely  ·  editing + persistence · depends on S3

Per-block editors for the editable types plus add/delete/reorder, and safe save back to .deepnote. Save is the highest-stakes operation in the product; faithful means semantic round-trip + idempotence, not byte-equality.

• Per-type block editors + structural edits
  • Code / SQL editor — per-type source editors with language-appropriate affordances.
  • Markdown / text editor — per-type source editors.
  • Input config / value editor — editing both the configuration and current value across the eight input kinds.
  • Add / delete / reorder blocks — structural editing of the notebook.
• Safe save back to .deepnote
  • Serialize + save round-trip — save via serializeDeepnoteFile so re-deserializing is deep-equal to the saved project, with an idempotent no-op save; reuses the S1 save-API gate.
  • Atomic write + external-change detection — write-temp-then-rename (no partial/corrupt file on interruption), plus detection of external edits so a file changed on disk since open is warned about, not clobbered.

Sprint 5 — Reactive re-execution  ·  reactivity · depends on S4

Auto re-run of downstream blocks on edit/input change via getDownstreamBlocksForBlocksIds, in dependency order — mirroring Cloud's signature reactive mode against local compute.

• Reactive downstream re-execution
  • DAG-driven downstream re-run in dependency order — compute downstream blocks on edit and re-run them in order against a persistent kernel session (net-new: run.ts does validation-only analysis today); a branch whose block raised an exception is halted.
  • Input-change propagation with commit/debounce — re-run an input's downstream blocks on commit/debounce (slider release / short settle), not continuously per intermediate value — a long dependent chain re-running on every drag increment against a Python kernel is the footgun to avoid.
  • Python-only degradation notice — on a non-Python kernel, degrade exactly as run.ts does (in-order, no dependency analysis) with a visible "reactivity disabled" notice rather than silently mis-ordering.

---

How to follow

• This Discussion — the narrative: the why, the trade-offs, what each gate caught.
• Issues — one per sprint: the design doc + ADRs + cards going in, the result + diff coming out.
• PRs — the diffs themselves, each labeled upstream-ready or fork-only showcase.

Everything — roadmap, PRD, ADRs, design docs, the board — lives on the
milestone/m3-local-ui branch.
This is a dry run on my own fork; nothing goes to deepnote/deepnote until the diff is something
I'd be proud to put in front of a maintainer. Feedback genuinely welcome.

[muunkky]

Sprint LUI1WEDGE · m3/s1 — Headless runtime server + one-command local launch
Status ✅ Complete · 13 cards, every one adversarially reviewed (executor → review → router → close-out) and merged · verified against a real kernel · upstream-ready slice cut as contrib/m3-serve · 9 follow-ups filed to backlog
Shipped @deepnote/runtime-server (kernel-free open · serialized run-over-WS · atomic save) · deepnote serve / deepnote ui · SQL/integration env parity with deepnote run

---

m3/s1 — a headless runtime server and a one-command local launch

This sprint landed the first slice of running a Deepnote project locally: a standalone runtime server plus deepnote serve / deepnote ui. The brief was deliberately narrow — make a .deepnote project openable, runnable, and saveable over a local HTTP/WebSocket surface — and the governing constraint was that it had to land as a clean addition to the codebase, not a change to it. Here's what was built, the decisions behind it, and how they came out in the implementation.

The shape: a new package behind a one-way arrow

The server is a new @deepnote/runtime-server package, not new surface inside cli or runtime-core. The dependency arrow is one-way and enforced — cli → runtime-server, never the reverse, and runtime-server never imports cli (ADR-007). Concretely, the only edits to existing files are the new CLI command registration and one dependency line; there is zero behavior change in runtime-core. Everything the server does, it does by reusing existing runtime-core primitives. That invariant is what makes the slice safe to take: it's additive, and the blast radius on existing deepnote run behavior is by construction nil. An always-on test enforces the no-cli-import boundary through the TypeScript AST so it can't silently regress.

Opening a project without starting a kernel

GET /api/project deserializes a .deepnote file and returns the full notebook/block tree with its persisted outputs without starting a Python kernel (KD-6). The reasoning: you shouldn't pay for a kernel just to read a notebook — a viewer needs to paint the project immediately, and only execution needs a runtime. In implementation that meant splitting the session lifecycle into loadProject() (pure deserialization + capability resolution, no kernel) and a lazy startEngine() (first run only). Capabilities (kernelLanguage, reactivity) are resolved from the existing interpreter/kernel-selection logic; a missing or mis-installed kernel surfaces as a capability flag, not an open failure. The open path is verified by deep-equaling its payload against deserializeDeepnoteFile on the same bytes, so the served tree is provably identical to the file.

One kernel, many callers: serialize at the server

A local UI can fire runs concurrently, but there is exactly one kernel behind it. ADR-005 is explicit that a single shared kernel must be serialized, so rather than expose that hazard the server fronts execution with a single-concurrency FIFO run queue. Runs stream over WS /api/stream, every event tagged with a runId, and the queue is the only caller of engine.runProject — enforced structurally so no code path can issue an un-serialized run. Two guarantees fell out of that design and are tested directly: every run emits exactly one terminal event (so a client can never hang on a "pending forever" run, including when a block raises), and cross-block back-pressure is handled by gating on the socket draining rather than buffering without bound.

Save that can't silently lose work

POST /api/project/save writes atomically — temp file then rename, in the same directory — and refuses to clobber: it SHA-256s the on-disk bytes at open and re-checks at save, returning 409 with the current on-disk content if they diverged (KD-7). Fidelity is defined as semantic, not byte-level, because the serializer re-canonicalizes — so the bar the tests hold is that deserialize(serialize(project)) deep-equals the project and that a re-save is idempotent. The result is a hard guarantee that a save never corrupts or silently overwrites a user's file.

Integration parity without duplicating logic or inverting the arrow

A SQL/integration block has to resolve the same credentials and environment that deepnote run resolves, or the two execution paths quietly diverge. Those integration-env helpers were cli-private, and the server couldn't reach them without inverting the ADR-007 arrow — so they were lifted into a shared home in runtime-core (KD-3), with cli re-exporting them. It's a pure relocation: run's behavior is unchanged, and both paths now resolve integration env through one implementation rather than two. Local-first is preserved — no outbound request fires unless a token is explicitly supplied.

Verified against a real kernel — and what that turned up

The parity claim — "the server runs your project the way deepnote run does" — is checked against an actual kernel, not mocks: an integration suite boots the server over a fixture, runs it, and deep-equals the streamed outputs against deepnote run --output json on the same file.

That suite surfaced a real bug, and a notable one: it lives in existing shared code, not the new package. A hard kernel crash mid-run makes the Jupyter server auto-restart the kernel — the status goes busy → autorestarting → idle and never reports dead — so the abandoned execution was being surfaced as an ordinary in-block error by both the new server and deepnote run. The fix, treating restarting/autorestarting during an active execute as a kernel death, went into the shared kernel-client, so it corrects the run-failure semantics for the existing CLI at the same time it does for the server.

How it's packaged for review

The whole slice is packages/ only — a new server library plus a thin CLI command — with no SPA, board, or planning artifacts in the diff. It builds, typechecks, and tests standalone off main, and an import-form integrity check keeps any frontend or apps/ reference out of the boundary. It is structured to read as exactly the PR we would open: a self-contained, additive capability that respects the existing architecture — and, on the way in, strengthens an existing code path.