Skip to content

feat(runtimed-py): introduce Python bindings for daemon client#348

Merged
rgbkrk merged 8 commits intomainfrom
quill/montpellier-v4
Mar 1, 2026
Merged

feat(runtimed-py): introduce Python bindings for daemon client#348
rgbkrk merged 8 commits intomainfrom
quill/montpellier-v4

Conversation

@rgbkrk
Copy link
Copy Markdown
Member

@rgbkrk rgbkrk commented Feb 28, 2026

Summary

Introduces the runtimed Python package - PyO3 bindings that enable programmatic notebook execution from Python. This is the foundation for agent tooling, MCP integrations, and automation workflows.

What's included:

  • PyO3 bindings exposing daemon operations to Python
  • Document-first execution model (cells synced via automerge)
  • Multi-client support (multiple sessions can share a notebook)
  • Comprehensive integration test suite (24 tests)
  • CI workflow for automated testing

New Components

Rust Crate: crates/runtimed-py/

Class Purpose
DaemonClient Low-level daemon ops: ping, status, list_rooms, flush_pool, shutdown
Session High-level notebook execution with kernel lifecycle
Cell Cell from automerge document (id, type, source, execution_count)
ExecutionResult Execution result with outputs, success flag, stdout/stderr helpers
Output Single output (stream, display_data, execute_result, error)
RuntimedError Exception type for daemon errors

Python Package: python/runtimed/

python/runtimed/
├── pyproject.toml           # Maturin build config
├── src/runtimed/
│   ├── __init__.py          # Public API exports
│   ├── _binary.py           # runt binary discovery
│   ├── _sidecar.py          # Sidecar launcher for rich output
│   └── _ipython_bridge.py   # IOPub bridge for terminal IPython
└── tests/
    ├── test_daemon_integration.py  # 24 integration tests
    ├── test_sidecar.py             # Sidecar unit tests
    ├── test_ipython_bridge.py      # Bridge unit tests
    └── conftest.py                 # Pytest config

API Examples

Session (code execution)

import runtimed

# Basic execution
with runtimed.Session(notebook_id="my-notebook") as session:
    session.start_kernel()
    result = session.run("print('hello')")
    print(result.stdout)  # "hello\n"

# Document-first pattern
session = runtimed.Session()
session.connect()
session.start_kernel()

# Create cell in automerge doc (syncs to all clients)
cell_id = session.create_cell("x = 42")

# Execute reads from document, not passed code
result = session.execute_cell(cell_id)

# Document operations
session.set_source(cell_id, "x = 100")  # Update cell
cell = session.get_cell(cell_id)         # Get cell info
cells = session.get_cells()              # List all cells
session.delete_cell(cell_id)             # Remove cell

DaemonClient (low-level ops)

client = runtimed.DaemonClient()
client.ping()        # True if daemon responding
stats = client.status()  # Pool stats
rooms = client.list_rooms()  # Active notebook rooms

Document-First Execution

The bindings implement the architectural principle that execution reads from the automerge document rather than receiving code directly:

  1. session.create_cell(source) - Creates cell in automerge doc
  2. session.execute_cell(cell_id) - Daemon reads source from doc
  3. All connected clients see the same code being executed

This enables multi-client scenarios where two Python sessions (or a Python session + the notebook app) can share a notebook and see each other's changes.

Integration Tests

24 tests covering:

  • Basic connectivity (2 tests) - Session connect, repr
  • Document-first execution (8 tests) - Cell CRUD, execute reads from doc
  • Multi-client sync (4 tests) - Two sessions sharing a notebook
  • Kernel lifecycle (3 tests) - Start, interrupt, shutdown
  • Output types (4 tests) - stdout, stderr, display_data, errors
  • Error handling (3 tests) - Execute without kernel, missing cells, syntax errors

Test plan

  • All 24 integration tests pass locally
  • CI workflow added (runtimed-py-integration job)
  • Multi-client automerge sync verified
  • Document-first execution pattern verified

@rgbkrk rgbkrk force-pushed the quill/montpellier-v4 branch 2 times, most recently from 3ebe9dd to 0411e8f Compare February 28, 2026 23:28
@rgbkrk rgbkrk marked this pull request as ready for review February 28, 2026 23:42
@rgbkrk rgbkrk changed the title feat(runtimed-py): add PyO3 bindings for daemon client feat(runtimed-py): document-first execution with daemon integration tests Feb 28, 2026
@rgbkrk rgbkrk changed the title feat(runtimed-py): document-first execution with daemon integration tests feat(runtimed-py): introduce Python bindings for daemon client Feb 28, 2026
return Path(os.environ["RUNTIMED_BINARY"])

# Check relative to this repo
repo_root = Path(__file__).parent.parent.parent.parent.parent
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use the CONDUCTOR_WORKSPACE_PATH here as a better fallback?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! We'll use that in CI too.

Add Python bindings for runtimed daemon operations:

- DaemonClient: pool status, ping, list rooms, flush, shutdown
- Session: connect to notebook room, start kernel, execute code
- ExecutionResult/Output: structured output types

Uses PyO3 0.28 with pyo3-async-runtimes for tokio integration.

Note: Current Session.execute() uses QueueCell shortcut which
bypasses automerge doc. See design gap notes for proper
doc-based execution flow.
- Add *.so to root gitignore for Python extension modules
- Remove contributing/architecture.md (lives on architectural-principles branch)
Switch runtimed-py bindings to use ExecuteCell instead of deprecated
QueueCell. The daemon now reads cell source from the automerge document,
ensuring all connected clients see the same code being executed.

Changes:
- Add Cell class to expose cell info to Python
- Add document operations: create_cell, set_source, get_cell, get_cells, delete_cell
- Add execute_cell() using ExecuteCell (reads from doc)
- Add run() convenience method (create + execute)
- Fix sync task exit bug by storing sync_rx in SessionState
- Add timeout to recv_frame_any() to prevent blocking
- Add biased select! to prioritize commands over polling
Add comprehensive integration tests for the document-first execution
pattern. Tests cover:
- Basic connectivity to daemon
- Document operations (create/update/get/delete cells)
- Cell execution via ExecuteCell (reading from automerge doc)
- Multi-client synchronization (two sessions sharing a notebook)
- Kernel lifecycle (start/interrupt/shutdown)
- Output types (stdout/stderr/display_data)
- Error handling

Supports two modes:
- Dev mode: uses existing daemon via `cargo xtask dev-daemon`
- CI mode: spawns isolated daemon with log capture

24 tests covering the core document-first architecture.
Add a new job to the build workflow that runs the runtimed-py
integration tests. The job:
- Builds runtimed binary and runtimed-py Python bindings
- Runs 24 integration tests in CI mode (spawns isolated daemon)
- Uploads test logs as artifacts for debugging

Tests verify document-first execution, multi-client sync, kernel
lifecycle, output capture, and error handling.
Comprehensive guide for the runtimed Python package covering:
- Session API for code execution
- DaemonClient for low-level operations
- Document-first execution pattern
- Multi-client scenarios
- Result types (ExecutionResult, Output, Cell)
- Sidecar launcher for rich output
1. Socket path mismatch (Issue #1): Session.connect() now respects
   RUNTIMED_SOCKET_PATH env var, and test fixtures set it when spawning
   daemon in CI mode.

2. Nested timeouts (Issue #2): Remove outer 10ms timeout wrapping
   recv_frame_any() since it already has internal 100ms timeout.
   This prevents repeatedly canceling mid-read.

3. sync_rx not drained (Issue #3): Use try_send() instead of send().await
   for changes channel. If receiver isn't keeping up, skip the update
   rather than blocking. Python bindings keep sync_rx alive but don't
   consume it.

4. Parse failure semantics (Issue #4): When output_type is "error" but
   parsing fails, create an error Output to preserve success=false
   semantics.

5. CONDUCTOR_WORKSPACE_PATH (Safia's comment): Use env var as preferred
   repo root fallback before walking up parent directories.
@rgbkrk rgbkrk force-pushed the quill/montpellier-v4 branch from 3679986 to f4454c6 Compare March 1, 2026 00:11
@rgbkrk rgbkrk merged commit b7ee30a into main Mar 1, 2026
12 checks passed
@rgbkrk rgbkrk deleted the quill/montpellier-v4 branch March 1, 2026 00:32
@rgbkrk rgbkrk added daemon runtimed daemon, kernel management, sync server editor CodeMirror, syntax highlighting, editor behavior mcp MCP server and agent integrations labels Mar 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

daemon runtimed daemon, kernel management, sync server editor CodeMirror, syntax highlighting, editor behavior mcp MCP server and agent integrations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants