livebook_test

mix test for Livebooks - Keep your Livebook examples honest.

livebook_test brings mix test-style workflows to Livebook notebooks. It discovers .livemd files, converts them to executable Elixir scripts, runs them, and reports failures - locally and in CI/CD.

Why?

Livebook notebooks are great for examples, tutorials, and interactive documentation. But they drift:

• A dependency update breaks your example notebook
• A refactoring breaks the API shown in a tutorial
• Published notebooks reference old package versions

livebook_test catches these problems before your users do.

Installation

Add to your Mix project:

def deps do
  [
    {:livebook_test, "~> 0.1.0", only: [:dev, :test], runtime: false}
  ]
end

Then fetch dependencies:

mix deps.get

Quick Start

# Run all discovered notebooks
mix livebook.test

# Run with verbose output
mix livebook.test --verbose

# Test against local checkout instead of Hex
mix livebook.test --mode local

# Run a specific notebook
mix livebook.test --path examples/basic.livemd

Example Notebooks

Notebook	Description
examples/basic.livemd	Simple arithmetic and IO - passes
examples/mix_install.livemd	Uses Mix.install with Jason - passes
examples/broken/broken.livemd	Intentionally failing cells - use to verify failure reporting
livebooks/local_dep.livemd	Uses Mix.install with local dependency patching

Configuration

Configure in config/config.exs:

config :livebook_test,
  paths: ["livebooks/**/*.livemd", "examples/**/*.livemd"],
  exclude: ["**/broken/**/*.livemd"],
  dependency_mode: :remote,
  timeout: 60_000,
  local_deps: [],
  verbose: false

Options

Option	Default	Description
paths	["livebooks/**/*.livemd", "examples/**/*.livemd"]	Glob patterns for notebook discovery
exclude	["**/broken/**/*.livemd"]	Glob patterns to exclude from discovery
dependency_mode	:remote	:remote leaves deps unchanged, :local rewrites to path deps
timeout	60_000	Per-notebook timeout in milliseconds
local_deps	[]	Keyword list mapping dependency names to local paths
verbose	false	Show per-notebook details

Local Dependency Testing

The killer feature: notebooks that use Mix.install can be automatically patched to use your local checkout.

Problem

Your example notebook says:

Mix.install([
  {:my_lib, "~> 0.5"}
])

But you want CI to test against the current checkout, not the published Hex version.

Solution

# config/config.exs
config :livebook_test,
  dependency_mode: :local,
  local_deps: [
    my_lib: "."
  ]

Now {:my_lib, "~> 0.5"} becomes {:my_lib, path: "."}.

Or via CLI:

mix livebook.test --mode local

CI/CD Integration

Add to your GitHub Actions workflow:

- name: Test Livebooks
  run: mix livebook.test

With local deps:

- name: Test Livebooks (local)
  run: mix livebook.test --mode local

The task exits with code 0 on success, 1 on failure, and 2 if no notebooks are discovered, perfect for CI gates.

CLI Options

mix livebook.test [options]

Options:
  --path PATTERN    Glob pattern for discovery (repeatable)
  --exclude PATTERN Glob pattern to exclude from discovery (repeatable)
  --mode MODE       Dependency mode: local or remote
  --timeout SECS    Per-notebook timeout in seconds
  --verbose         Show per-notebook details

Programmatic API

# Run with defaults
LivebookTest.run()

# Run with options
LivebookTest.run(paths: ["examples/**/*.livemd"], mode: :local, timeout: 120_000)

# Run and print report, returns exit code
LivebookTest.run_and_report(verbose: true)

Pipeline

Notebooks flow through a pipeline:

1. Discovery - Find .livemd files via glob patterns
2. Export - Convert to .exs scripts using Livebook.live_markdown_to_elixir/1
3. Patch - Optionally rewrite Mix.install deps to local paths
4. Run - Execute each script as an isolated subprocess
5. Report - Summarize results with pass/fail counts and timing

Example Output

3 notebooks
3 passed
0 failed
Total time: 2.1s

All notebooks passed! ✅

With failures:

3 notebooks
2 passed
1 failed
Total time: 5.3s

Failed notebooks:
--------------------

  examples/broken/broken.livemd
  exit: 1
    stderr:
    ** (RuntimeError) Intentional failure for testing

Architecture

Module	Responsibility
LivebookTest	Public entry point, orchestration
LivebookTest.Config	Configuration resolution
LivebookTest.Discovery	Notebook file discovery
LivebookTest.Exporter	.livemd → .exs conversion
LivebookTest.DependencyPatcher	Mix.install dependency rewriting
LivebookTest.Runner	Script execution and result collection
LivebookTest.Report	Summary formatting and exit codes

Roadmap

Version	Feature
v0.2.0	Snapshot testing
v0.3.0	Parallel execution
v0.4.0	JUnit output
v0.5.0	GitHub annotations
v0.6.0	Notebook metadata and tags
v0.7.0	Coverage reporting
v0.8.0	HTML reports
v0.9.0	Distributed notebook execution
v1.0.0	Stable public API

License

MIT