Turn long-running notebooks into durable, observable jobs—without turning them into pipelines.
Run the notebook you already have. Follow it from your phone. Recover where the work failed.
Runwatch is a reliability layer for Jupyter notebooks. It combines cell-by-cell
nbclient execution with durable checkpoints, live resource monitoring, notifications,
and a mobile-friendly dashboard. When a cell fails, the run pauses with its state and
monitors intact, ready for you—or your coding agent—to repair and continue.
Keep notebook ergonomics. Gain the confidence of an operational job.
Install from PyPI · Run your first notebook · Run from VS Code · Explore resource monitoring · See failure recovery · Read the docs
- Recover instead of rerun. Runwatch checkpoints every settled cell, preserves a live kernel after failure, and supports resume or clean replay after a process or kernel restart.
- Walk away without losing visibility. Follow cells, outputs, tracebacks, progress, metrics, logs, and external resources from a responsive dashboard on desktop or mobile.
- Get an ETA that adapts to the work. Runwatch mirrors existing
tqdmbars without changing the loop, adds an uncertainty range, and recalibrates when throughput shifts. - See the whole workload, not just the notebook. Built-in adapters watch SageMaker Processing, S3, CloudWatch, host and kernel metrics, NVIDIA GPUs, local files, logs, and linked localhost dashboards.
- Know when something changes. Durable webhook and
ntfydelivery can notify you with periodic status reminders, notebook section transitions, failures, completion, and refreshed Cloudflare sharing links. - Keep the execution host awake. An optional native IOKit assertion on macOS or logind inhibitor on Linux prevents idle system sleep while the run is live.
- Share access intentionally. Keep the dashboard on localhost, expose it to a
trusted LAN, or use a self-healing
cloudflaredquick tunnel with pairing-token authentication. - Work equally well with humans and coding agents. The editable run-owned notebook, structured context command, and explicit resume/restart workflow use ordinary files and CLI commands—no agent lock-in.
- Get the result back where you expect it. Settled sources, execution counts, and outputs are written atomically to the notebook you launched, with conflict detection to protect outside edits.
- Avoid a trail of abandoned state. Successful and cancelled runs clean themselves
up after the dashboard closes, while incomplete notification delivery and
--keep-runpreserve state when it still matters.
The dashboard is deliberately observation-first. Its one remote mutation is narrowly scoped: stopping an owned, stoppable resource also cancels the run. Runwatch records actions, source hashes, kernel epochs, adapter cursors, observations, logs, notifications, and bounded events in durable SQLite state so recovery does not depend on one long-lived process.
Python 3.10 or newer is required. Runwatch supports local POSIX filesystems on Linux and macOS; those are the platforms exercised in CI. Windows and shared/network filesystems are not currently supported execution targets.
Install the published package from PyPI in the virtual environment used by your notebook project:
python -m venv .venv
source .venv/bin/activate
python -m pip install 'runwatch-notebook[supervisor]'The supervisor extra is the recommended installation for people running notebooks.
It includes the CLI, notebook runner, dashboard, notifications, and built-in AWS and
local resource adapters. The selected notebook kernel must use this environment, or
another environment that can import runwatch, when cells emit resource events.
The smaller, Pydantic-only base package is intended for kernels or integrations that only emit generic Runwatch protocol events:
python -m pip install runwatch-notebookInstall optional NVIDIA monitoring support with:
python -m pip install 'runwatch-notebook[supervisor,gpu]'To develop Runwatch itself from a source checkout:
uv sync --extra supervisor --extra test --extra dev --extra docs
uv run pytest tests
uv run ruff check src testsThird-party supervisor adapters can extend Runwatch without adding provider-specific code to this package. See Third-party resource adapters for the entry-point contract and adapter API.
runwatch init-config runwatch.yaml
runwatch validate notebook.ipynb --config runwatch.yaml
runwatch execute notebook.ipynb --config runwatch.yamlinit-config writes an annotated starter file. validate checks nbformat structure,
kernel availability, paths, sharing prerequisites, configured adapters, and terminal
conditions without starting a kernel, dashboard, or provider resource. Resources
emitted dynamically by cells cannot be predicted during preflight.
Runwatch prints the run directory, editable notebook, and pairing URL. With Cloudflare
sharing, that pairing URL is the authenticated localhost dashboard; the current public
link and QR live on that page.
By default the dashboard remains available for 90 seconds after the run reaches a
terminal state, then closes automatically. If the run succeeded or was cancelled,
Runwatch removes that run directory and removes the empty .runwatch/runs and
.runwatch parents. When
notifications are configured, cleanup first waits up to
notifications.terminal_drain_timeout_seconds for terminal event routing and delivery
attempts. If the outbox is still nonterminal, the run is retained and the CLI
prints a reason plus runwatch open RUN_DIR; opening it restarts notification delivery
without rerunning the notebook. The recovery controller conservatively retains the run
when that dashboard closes because it did not itself observe normal notebook
finalization. After confirming delivery, remove the retained state explicitly if it is
no longer needed. Set server.linger_seconds: 0 to close immediately, set it to null
to keep the dashboard open until Ctrl+C, or use --keep-run to retain successful or
cancelled state after the original execution's dashboard closes.
For a no-AWS replay with live progress, local metrics, file monitoring, log tailing, and final notebook results, use the repository's Runwatch fake session:
web_artifacts_fake_sessions/runwatch/run.shEvery run contains:
.runwatch/runs/<timestamp>-<notebook>-<id>/
├── access-token.txt
├── input.ipynb # immutable original snapshot
├── source.ipynb # agent/human editable nbformat document
├── executed.partial.ipynb # runner-owned rolling checkpoint
├── executed.ipynb # final executed notebook
├── writeback-state.json # conflict guard for the user-owned notebook
├── run-manifest.json
└── runwatch.sqlite3
Each per-run directory is restricted to the current user (0700) and new run-state
files are created with mode 0600. Retained state is nevertheless sensitive: it can
contain notebook source and output, tracebacks, local paths, resource identifiers and
logs, notification destinations and payloads, actions, and the dashboard bearer token.
The mode of the original notebook passed to execute is preserved during write-back.
Runwatch versions its persisted and wire contracts independently. New run manifests and SQLite databases use schema version 3; configuration and kernel resource events use schema version 2; S3 progress manifests use schema version 1. Legacy schema-version-2 run directories reopen conservatively. Runwatch intentionally does not migrate 0.1 run directories.
Turn the active notebook into a Runwatch job from VS Code's task picker. The setup
below is project-local: it uses the project's .venv, reads notification secrets from
an ignored .env, prepares a matching Jupyter kernel, and runs whichever saved
notebook is active in the editor.
First create the project's virtual environment and install Runwatch from PyPI together
with ipykernel and the dotenv command. No Runwatch source checkout is required:
python -m venv .venv
source .venv/bin/activate
python -m pip install 'runwatch-notebook[supervisor]' ipykernel 'python-dotenv[cli]'Create .vscode/runwatch.yaml:
notebook:
kernel_name: runwatch-workspace
host:
prevent_system_sleep: true
notifications:
ntfy_base_url: ${RUNWATCH_NTFY_BASE_URL}
ntfy_topic: ${RUNWATCH_NTFY_TOPIC}
ntfy_on_section_start: trueKeep the ntfy destination in the project's .env and make sure that file is ignored
by Git:
RUNWATCH_NTFY_BASE_URL=https://ntfy.sh
RUNWATCH_NTFY_TOPIC=your-private-topicThen add the following tasks to .vscode/tasks.json (merge them into its existing
tasks array if the project already has one):
{
"version": "2.0.0",
"tasks": [
{
"label": "notebook: prepare Runwatch kernel",
"type": "process",
"command": "${workspaceFolder}/.venv/bin/python",
"args": [
"-m",
"ipykernel",
"install",
"--prefix",
"${workspaceFolder}/.venv",
"--name",
"runwatch-workspace",
"--display-name",
"Runwatch workspace (.venv)"
],
"problemMatcher": []
},
{
"label": "notebook: run active notebook",
"type": "process",
"command": "${workspaceFolder}/.venv/bin/dotenv",
"args": [
"--file",
"${workspaceFolder}/.env",
"run",
"--",
"${workspaceFolder}/.venv/bin/runwatch",
"execute",
"${file}",
"--config",
"${workspaceFolder}/.vscode/runwatch.yaml",
"--share",
"cloudflared"
],
"options": {
"cwd": "${workspaceFolder}",
"env": {
"PATH": "${workspaceFolder}/.venv/bin:${env:PATH}",
"JUPYTER_PATH": "${workspaceFolder}/.venv/share/jupyter:${env:JUPYTER_PATH}"
}
},
"dependsOn": "notebook: prepare Runwatch kernel",
"dependsOrder": "sequence",
"problemMatcher": [],
"presentation": {
"reveal": "always",
"panel": "dedicated",
"clear": true
}
}
]
}Save and focus the .ipynb you want to run, open Tasks: Run Task from the Command
Palette, and select notebook: run active notebook. The terminal shows the Runwatch
dashboard pairing URL while the task keeps running. This example uses a Cloudflare
quick tunnel, so cloudflared must be available on PATH; change the final share value
to lan for a trusted local network or none for a local-only dashboard.
The task configuration enables sleep inhibition and ntfy section announcements. If
notifications are not wanted, remove the notifications block from
.vscode/runwatch.yaml and invoke runwatch directly instead of through dotenv.
For an unattended run on macOS or Linux, opt into native idle-sleep inhibition:
host:
prevent_system_sleep: truemacOS uses an IOKit NoIdleSleepAssertion; Linux holds a logind inhibitor through
systemd-inhibit. The inhibitor starts before the notebook runtime and remains active
through execution, blocking-resource waits, failure pauses, recovery, cancellation, and
finalization. It is released before the post-run dashboard linger begins and during
abnormal supervisor cleanup. Display sleep is unaffected.
Linux hosts need systemd-inhibit and a reachable logind service. Runwatch fails the
start if the explicitly requested inhibitor cannot be acquired; runwatch validate
checks that the platform backend can be constructed before execution. Windows is not a
supported execution target.
After every settled cell attempt, Runwatch atomically writes the executed notebook
state back to the notebook passed to execute. This includes repaired source,
execution counts, outputs, and tracebacks. If that notebook changes outside Runwatch,
write-back stops rather than overwriting the external edit, and the run-owned partial
checkpoint is retained.
Rolling checkpoints serialize an immutable notebook generation on the event-loop
thread before filesystem I/O. Requests that arrive during a write remain pending, and
transient checkpoint failures are journaled and retried with bounded backoff until the
worker recovers. Checkpoint and write-back publication use a temporary file, file
fsync, atomic replacement, and parent-directory fsync where the local filesystem
supports them.
Original-notebook conflict detection is a best-effort portable compare-and-replace. It
checks a content and metadata fingerprint again immediately before os.replace, which
detects external saves during preparation and makes the remaining race very small. No
portable filesystem primitive can make the final comparison and rename indivisible, so
another writer in that last window can still be overwritten. Avoid editing the original
notebook while Runwatch is executing; edit the run-owned source.ipynb for recovery.
When a cell fails or reaches its configured timeout, Runwatch persists its outputs and
traceback, pauses the notebook, and keeps the kernel and resource monitors alive. A
timed-out kernel is interrupted and synchronized before live resume is allowed. Edit
source.ipynb with normal nbformat:
from pathlib import Path
import nbformat
path = Path(".runwatch/runs/.../source.ipynb")
notebook = nbformat.read(path, as_version=4)
notebook.cells[6].source = "result = repaired_input()"
nbformat.write(notebook, path)If only the failed or future cells changed, resume in the live kernel:
runwatch resume .runwatch/runs/...If imported source or an already-executed cell changed, create a new kernel and replay:
runwatch restart .runwatch/runs/...Replay starts at cell zero. An explicit override may start later, but the operator owns reconstruction of any missing kernel state:
runwatch restart .runwatch/runs/... --from-cell 4N is zero-based and must identify a cell in the current source.ipynb.
If the Runwatch process is no longer live, resume first journals the recovery action,
reopens the persisted run, restores resource monitors and cursors, starts a new kernel
epoch, and replays from cell zero. A crash during recovery leaves an action that the
next invocation can safely recover.
Cancellation is bounded rather than relying on one cooperative interrupt. Runwatch
persists cancelling, then advances through kernel interrupt, graceful shutdown,
provisioner terminate, and provisioner kill using the four notebook.cancel_*_grace_seconds
settings. Stage failures are journaled, repeated cancellation requests are idempotent,
and an execution client that still does not return is detached so the run can settle as
cancelled instead of hanging indefinitely.
For an agent-oriented dossier:
runwatch context .runwatch/runs/... --format markdown
runwatch context .runwatch/runs/... --jsonResources are emitted as structured Jupyter MIME output immediately after they are created or selected. Runwatch injects the active run, cell, attempt, and kernel epoch.
from runwatch import aws
sagemaker.create_processing_job(**request)
aws.emit_owned_sagemaker_processing_job(
request["ProcessingJobName"],
region="us-east-1",
logical_key="feature-build",
output_prefixes=["s3://bucket/run/output/"],
)SageMaker Processing is blocking but borrowed and observation-only by default. Use
emit_owned_sagemaker_processing_job only for a job the current run created and may
stop; that explicit helper claims exclusive ownership and enables provider stop during
run cancellation. Existing calls that explicitly pass stop_on_cancel=True continue
to opt into exclusive ownership. Set ownership="exclusive" with
stop_on_cancel=False when manual stop should be available without joining the
cancellation cascade.
aws.emit_s3_prefix(
"s3://bucket/run/output/",
expected_count=400,
completion_marker="_SUCCESS",
blocking=True,
full_rescan_seconds=300,
)A blocking prefix must define an expected count or completion marker. Truncated scans
continue from a persisted key on the next poll instead of restarting at page one. After
the initial reconciliation, polls scan only keys after the last observed key and perform
a full reconciliation every full_rescan_seconds (five minutes by default). Incremental
counts therefore assume append-only keys between reconciliations; the dashboard exposes
the scan mode, reconciliation time, and lower-bound warning. Set
full_rescan_seconds=0 when every poll must be a full exact scan.
aws.emit_s3_manifest("s3://bucket/run/progress.json", blocking=True)The manifest is a small JSON document:
{
"schema_version": 1,
"status": "running",
"completed": 180,
"total": 400,
"message": "Building features",
"metrics": {"rows": 9000000}
}status must be running, completed, or failed; metrics must be finite scalar
values and cannot override reserved manifest fields. Manifests are capped at 1 MiB.
aws.emit_cloudwatch_metric(
namespace="MyPipeline",
metric_name="ChannelsProcessed",
dimensions={"RunId": "abc"},
)
aws.emit_cloudwatch_logs(
log_group="/aws/my-pipeline",
stream_prefix="run-abc/",
)CloudWatch metrics and logs are always nonblocking. Metric cards retain the full current lookback window, while observation history stores only new or revised timestamp samples and bounds its deduplication cursor to 1,440 entries. Log stream discovery rotates across bounded pages when more streams exist than can be displayed at once, shares each poll's line budget across busy streams, and prunes tokens after a complete discovery rotation. Terminal SageMaker jobs drain paginated logs until caught up or report an explicit truncation.
from runwatch import local
local.emit_system_metrics(include_host=True, include_kernel=True, gpu="all")
local.emit_file_count(
"artifacts/parts",
pattern="*.parquet",
expected_count=400,
blocking=True,
)
local.emit_line_count(
"logs/inference.log",
expected_lines=10000,
tail_lines=100,
blocking=True,
)
local.emit_dashboard(
"http://127.0.0.1:8501",
name="Training dashboard",
health_path="/_stcore/health",
logical_key="training-dashboard",
)CPU and memory use psutil. NVIDIA metrics use optional NVML support and degrade to an
nvidia_available=false metric when unavailable. File-count scans stream metadata into
exact count, total-byte, and newest-modification-time aggregates without retaining one
stat object per file. That settlement signature is an aggregate, not content identity;
content edits that preserve all three values may not reset settlement. Line monitoring
is optimized for append-only logs: it detects replacement, truncation, changes near the
committed offset, and non-growing files whose mtime changes, while keeping partial-line
memory bounded. An earlier in-place edit combined with later file growth can remain
undetected if it does not touch the offset fingerprint. Conversely, a metadata-only mtime
change on a non-growing file is conservatively treated as a rewrite and may replay the
bounded log tail. line_count includes records terminated by LF or CRLF; a CR is counted
once a following byte proves it is not the start of CRLF. A final unterminated fragment,
including a trailing lone CR, is excluded until completed and is reported through
partial_line_pending, partial_line_buffered_bytes, and partial_line_truncated.
emit_dashboard registers an already-running local web application. Runwatch monitors
its health and adds an Open Training dashboard action to the resource card. With
--share lan, that action opens a separate authenticated LAN reverse proxy; with
--share cloudflared, it opens a separate authenticated quick tunnel. HTTP streaming,
redirects, cookies, WebSockets, and SSE pass through at the proxy root, which avoids
breaking applications that do not support a configurable path prefix.
Only explicit localhost and loopback IP URLs are accepted. Linked dashboards are
external, nonblocking, and observation-only: Runwatch neither starts nor stops the
application. The registration survives Runwatch recovery, while its proxy port and
tunnel URL are recreated for the current process and are never persisted in dashboard
state. Stop the linked application through its own controls or local process manager.
from runwatch import emit_progress
emit_progress(180, total=400, unit="partitions", message="Building features")Python kernels also mirror tqdm, tqdm.auto, and tqdm.notebook bars into the
same dashboard progress area without notebook changes:
from tqdm.auto import tqdm
for item in tqdm(items, desc="Building features", unit="items"):
process(item)Runwatch preserves tqdm's normal notebook output and emits structured updates at most
twice per second by default. Updates reuse one hidden notebook display per bar, so a
long loop does not append an output for every refresh. The dashboard scopes progress
to the current cell and prefers the outermost bar when bars are nested. It keeps tqdm's
recent-rate ETA and adds a Bayesian median finish time with an 80%
posterior-predictive range
using the current run's elapsed time and completed items. It compares a shared-rate
model with a two-rate model, including the posterior probability that the new regime is
materially slower. When that probability remains high, the estimate recalibrates from
the inferred cache boundary instead of treating cache hits as real-work throughput.
It then keeps testing recent windows against the current regime in both directions,
requiring 99.99% posterior probability before rebasing. Large rate shifts accumulate
that evidence quickly, while smaller shifts require longer observations. Set
notebook.capture_tqdm: false to disable automatic capture, or adjust
notebook.tqdm_min_interval_seconds. Progress created in a separate process is outside
the notebook kernel and is not captured automatically.
Runwatch can announce notebook section transitions through ntfy:
notifications:
ntfy_base_url: https://ntfy.sh
ntfy_topic: your-private-topic
ntfy_on_section_start: trueAfter a non-empty code cell finishes, Runwatch inspects the Markdown cells before the next non-empty code cell. If they contain headings, it emits a durable section-start event and sends the nearest heading to ntfy immediately before that next code cell begins. The notification is ntfy-only even when generic webhooks are also configured. The first code cell is not announced because no preceding code cell has completed.
Heading text leaves the machine when this option is enabled. Keep secrets out of Markdown headings and use a private ntfy topic.
The dashboard shows notebook state, reported progress, outputs, tracebacks, resource metrics and charts, log tails, and the durable event journal.
The dashboard header provides an Open notebook action beside the ntfy action. It
opens a separate, authenticated, read-only rendering of the saved notebook in a new
tab. While the notebook is executing, the view uses executed.partial.ipynb; after
notebook cells finish, it uses the final output; before the first checkpoint, it renders
the saved source. The page does not update automatically: the browser's normal refresh
control (including pull-to-refresh on mobile) loads the newest durable save. The page
reports the snapshot timestamp and settled-cell count. On mobile, scrolling down in the
notebook collapses that metadata header; selecting its compact summary expands it again.
This full-notebook view is intentionally different from the bounded dashboard timeline: it can contain all notebook source and saved outputs. Runwatch removes active and navigation-capable HTML, omits JavaScript-only outputs, and isolates the rendering in a sandboxed frame with scripts and network access disabled. The child retains only its same-origin identity so the trusted wrapper can observe scroll position for the compact mobile header. Treat access to it as access to the notebook itself.
The browser API is an explicit presentation model rather than a dump of SQLite state.
It allowlists display-safe run, cell, resource, metric, and event fields and omits
notebook source, resolved configuration, notification endpoints, provider cursors and
raw responses, controller tokens, and dedicated internal-path fields. Output,
traceback, log, and chart payloads are bounded, but their user-generated text can still
contain sensitive values. SSE carries only sequence, timestamp, and event type as an
invalidation signal; the browser then refreshes the sanitized snapshot. Dashboard
pages, snapshots, SSE, and authenticated redirects use Cache-Control: no-store.
Registered localhost dashboards appear as normal resource cards. Their Open action is available only while the authenticated share is ready and uses the same pairing session as the Runwatch dashboard.
Only active, exclusive resources whose adapter supports stop show a Stop button. The
confirmation lists the selected job and every other resource affected by cancellation.
Confirming the action interrupts the notebook, stops all eligible owned resources, and
finishes the run as cancelled. The provider resource is revalidated and accepts the
stop request before notebook cancellation begins; a stale or superseded resource has no
cancellation side effect.
The equivalent local command is:
runwatch resource stop RUN_DIR RESOURCE_IDrunwatch status RUN_DIR [--json]
runwatch validate NOTEBOOK [--config PATH] [--json]
runwatch events RUN_DIR [--follow] [--json]
runwatch open RUN_DIR
runwatch notifications rotate RUN_DIR --config PATH
runwatch notifications purge RUN_DIR --yes
runwatch version
open serves persisted state without starting notebook execution and retries durable
notification delivery. It conservatively retains the run when the dashboard closes:
only the controller that observed normal notebook finalization may authorize automatic
run cleanup.
status, context, and events emit bounded presentation schema version 1 in JSON
mode. They expose recovery-relevant lifecycle fields and safe summaries, not the raw
SQLite snapshot, resolved notification configuration, controller credentials, or raw
event payloads.
Notification credential maintenance is offline and lock-fenced. notifications rotate reads only the notification settings from --config, atomically records them
as desired state in the run manifest, then rewrites every persisted delivery for the
same webhook/ntfy topology. It also rearms failed deliveries, clears legacy transport
errors, and sanitizes pre-presentation intents. To change topology, purge the old
outbox first; a subsequent rotate may enable the new topology because no old delivery
rows remain. notifications purge --yes disables routing without replay, removes the
outbox, scrubs notification diagnostics, and best-effort compacts SQLite.
Runwatch validates resource lifecycle semantics at every entry point—not just in the
convenience emitters. Metrics, logs, and system monitors are always nonblocking;
conditional S3 and local-file monitors require a concrete terminal condition; and
stop_on_cancel requires an exclusive adapter that implements stop.
Runwatch requires CurveZMQ encryption for the local Jupyter manager-to-kernel
channels and fails kernel startup rather than falling back to plaintext TCP. The
kernel must advertise Curve support; IPython kernels require ipykernel>=7.3.
Localhost is the default. For a trusted LAN:
runwatch execute notebook.ipynb --share lanFor a temporary public tunnel, install cloudflared and use --share cloudflared.
Runwatch prints only the local loopback pairing URL. Open it to get the current public
Cloudflare link and QR. Runwatch checks that public route throughout the run and, after
repeated failures, replaces only the tunnel: the notebook kernel and local dashboard
keep running while the link and QR update. If ntfy is configured, the replacement
pairing URL is also sent as the notification's clickable target.
The pairing URL is a bearer credential. Configure ntfy only with a private topic you trust to receive it. Otherwise keep the URL private. See the security guide.
Runwatch does not launch or communicate with agents. A repository-local skill under
agent_skills/runwatch/ teaches Codex or Claude to inspect runwatch context, edit
source.ipynb, choose resume versus restart, and verify the resulting state.
Install or update that skill for Codex, VS Code Copilot, and Claude Code with:
uv run python scripts/install_skills.py update --target allThe installer copies the skill to the personal skill directories for each selected
agent. Pass runwatch instead of update for a non-replacing first installation, or
use --target codex, --target copilot, or --target claude for one destination.
MIT. See LICENSE.