Plottery

Plottery is an in-notebook graphical interface for Matplotlib (MPL). Instead of a static image, with Plottery plt.show() produces a rich graphical interface that lets you add plot elements, drag and resize shapes, tweak plot arguments with live preview, and prompt an LLM to make changes. All changes are written to the code live—the code is the ground truth—and code editing is supported at any time.

Naming note: internally the code still uses the old name SNP ("Sketch-n-Plot").

Install instructions

Setup Python packages.

Create a virtual environment (recommended).

python3 -m venv .venv
source .venv/bin/activate

Install Python packages.

pip install -r requirements.txt

Install node packages.

npm install

The JupyterLab / Notebook v7 extension has its own npm package; install its dependencies too.

cd src/snp_jupyter
npm install
cd ../..

Build and install the extension. This will build the JupyterLab / Notebook v7 labextension and copy it into Jupyter's labextensions/ directory:

npm run build

For classic Jupyter Notebook (v6), npm run build-nbv6 installs + enables the nbextension:

npm run build-nbv6

While developing the frontend, npm run watch-ts rebuilds dist/plugin.js on change. The bundle is re-inlined into the output on each manual cell re-run, so you just re-run the cell to pick up changes.

Test it by opening jupyter and running src/snp.ipynb.

OPENAI_API_KEY=your_key jupyter notebook

If OPENAI_API_KEY is not provided, the AI panel will not show.

It should also work in Jupyter Lab:

OPENAI_API_KEY=your_key jupyter lab

If making screenshots, set window.sessionStorage.setItem('make_stuff_nice_for_screenshots', 'true'). This whitens the background, remove the execution counts and cell selection styling, and recenters the Plottery UI.

%%javascript
window.sessionStorage.removeItem('make_stuff_nice_for_screenshots')
window.sessionStorage.setItem('make_stuff_nice_for_screenshots', 'true')

If making videos, set in_demo_mode. In addition to adding the make_stuff_nice_for_screenshots styling, this will make the plot slightly smaller and will surpress some distracting errors and warnings from Matplotlib.

window.sessionStorage.removeItem('in_demo_mode')
window.sessionStorage.setItem('in_demo_mode', 'true')

---

The remaining sections below are a technical explanation of Plottery's operation.

1. Big picture & data flow

There are roughly three technical parts.

1. Jupyter/JupyterLab extension (src/nbextension_snp/main.js / src/snp_jupyter/snp_jupyter.js) sets up three things: (a) automatic import snp so user does not have to do anything to access Plottery, (b) adds a "New Plot" button to the interface, and (c) sets ups a cell code rewriter to change plt.show() to snp.show() (stashes the figure instead of showing it) and append snp.show_ui() to the end of the cell (which actually shows the figure and UI). The JupyterLab version also lifts some Jupyter and CodeMirror modules to globals so we can reference them later.
2. The Python backend (src/snp.py) gathers all the information required to build the UI (function arguments and types via mypy, which user variables are type compatible with which arguments, the user's names for axes etc, and the regions where MPL shapes are on the plot).
3. The Typescript frontend (src/frontend/main.ts) builds the UI, attaches layer "marks" to text regions of the code editor, and coordinates event handling.

                  Jupyter browser extension                    Python kernel
                  (nbextension OR labextension)
  kernel    ───►  tells Python kernel to import snp       ───► import snp
  startup                                                      snp.py adds an AST rewriter that, on
                                                               each cell execution, wraps function
                                                               calls with logging code to set
                                                               _snp_came_from_call_id on the call
                                                               results so we know which MPL shapes
                                                               came from which calls.

  user runs ───►  rewrites the cell code:                 ───► the AST rewritter adds call logging
  a cell          plt.show(...)  →  snp.show(...)              snp.show(...) stashes the figure
                  also appends snp.show_ui(fig_idx=N)          snp.show_ui() builds an `SNP` object
                  to end of cell                               that has a ._repr_html_() function
                                                               that builds the SNP UI and bundles its
                                                               Javascript and styles. It returns one big
                                                               HTML blob: <img> of the plot,
                                                               an SVG of "hover regions", the inlined
                                                               built JS (dist/plugin.js), inlined CSS,
                                                               and a <style onload="attach_snp(...serialized
                                                               types/calls/AST...)"> that triggers the
                                                               Javascript to initialize the UI
  attach_snp() builds the UI ◄──────────────────────────────────────────────────────────────┘

  later, on GUI edit:
  frontend edits CodeMirror text ─► re-executes the cell with a trailing
  snp.show_ui(snp_class=SNPFigureOnly|SNPFigureAndHoverRegions) ─►
  kernel returns a new PNG (and potentially SVG hover regions) over iopub ─►
  frontend swaps the <img>/<svg> in place (no full re-render unless needed)

---

2. Repo layout (the files that matter)

Path	What it is
src/snp.py	The Python core. Rendering, type inference, hover regions, provenance, serialization, the SNP output object classes, show() / show_ui().
src/frontend/	The actual UI (TypeScript, built with Vite → dist/plugin.js). Entry point is attach_snp() in main.ts.
src/nbextension_snp/main.js	Browser extension for Jupyter Notebook (classic, v6).
src/snp_jupyter/	Browser extension for JupyterLab / Jupyter Notebook v7 (its own npm package + build).
src/serialize.py	Turns arbitrary Python/mypy object graphs into JSON (arbitrary_to_json). Inverse of the frontend's deserialize.ts.
src/visitor_mypy.py	A TraverserVisitor over mypy AST nodes (vendored/derived from mypy). Used to walk typed calls.
src/python-type-stubs-main/	Our fork of Microsoft's .pyi type stubs (matplotlib, numpy, etc.). These determine the arguments in the Properties panel for each MPL call.
vite.config.js, tsconfig.json, package.json	Frontend build + the npm run scripts that install/enable the extensions.
requirements.txt	Pinned Python deps (note notebook==6.2.0, mypy==1.8.0, numpy<2, shapely==2.0.2).
src/see.py	See(...) — a dev-only HTML object inspector for poking at values in the notebook. Not part of the product flow.
src/*.ipynb, study_2_*/, *.rb	Demo notebooks and user-study data/analysis. Not part of the tool.

---

3. The two browser extensions (and why there are two)

Jupyter Notebook v6 (classic) and JupyterLab/Notebook v7 have different frontend APIs, so Plottery ships the same behavior twice. They do the same three jobs:

1. import snp automatically whenever the kernel (re)starts.
2. Rewrite plt.show(...) in any executed cell into a snp.show(...) call that also forwards the notebook context the kernel can't otherwise see.
3. Append snp.show_ui(fig_idx=N) to the cell so the rich UI is produced as the cell's output, and add a "New Plot" toolbar button with starter code.

The rewrite (identical logic in both) looks for plt.show / matplotlib.pyplot.show / aliased imports and turns:

plt.show()

into:

snp.show(globals() | locals(), cell_lineno, plt_show_lineno_in_cell,
         provenance_is_off_by_n_lines, "<entire notebook code through this cell>")
last_snp = snp.show_ui(fig_idx=0)   # appended
last_snp

snp.show() needs notebook_code_through_cell because the IPython kernel only ever sees one cell at a time, but type inference needs the whole notebook so far.

How they differ

	src/nbextension_snp/main.js	src/snp_jupyter/snp_jupyter.js
Target	Jupyter Notebook classic (v6)	JupyterLab / Notebook v7
Module system	RequireJS AMD (define([...])), entry load_ipython_extension	ES module exporting a JupyterFrontEndPlugin (activate)
Notebook access	global Jupyter.notebook / IPython	INotebookTracker injected by Lab
Hooks the rewrite via	listens to execution_request.Kernel event; monkeypatches IPython.CodeCell.prototype.execute to know the running cell	monkeypatches kernel.sendShellMessage and inspects execute_request messages
Per-cell UI state (fig_idx)	read from the cell DOM output_area … dataset.fig_idx	read from sessionStorage key cell-<id>-snp-<key>
Editor it drives	CodeMirror 5	CodeMirror 6 — so it stashes CM6 classes on window (__CM6EditorView, __CM6Decoration, StateField, …) and Lab classes (__JupyterCodeCellModule, __JupyterNotebookActionsModule) for the Plottery frontend to use
Build	none — plain JS, installed via jupyter nbextension install/enable	its own npm package; jupyter labextension build → src/snp_jupyter/labextension, then copied into Lab's labextensions/

Both also pass a doesnt_need_snp_show_ui flag through: when the frontend triggers a re-execute that already includes its own snp.show_ui(...) (e.g. a fast figure-only redraw), the extension must not append a second show_ui.

The frontend itself (src/frontend/) is shared by both — it's injected into the page by Python (see §5), not by the extensions. The extensions only handle code rewriting of plt.show(), kernel bootstrapping, and the toolbar button.

---

4. Python vs. frontend — who does what

Concern	Where
Rewriting plt.show → snp.show / appending show_ui	Browser extension (main.js / snp_jupyter.js)
Rendering the plot to PNG / SVG hover-regions	Python (SNP*._repr_png_ / _repr_svg_)
Type inference (what args a call takes, their types)	Python (mypy, in snp.py)
Provenance: which MPL shapes came from which line of code	Python (IPython AST transformer + tag_with_call_provenance)
Serializing types/calls/AST to JSON for the UI	Python (serialize.py + snp.py)
Building the UI, widgets, drag handlers, layers, AI panel	Frontend (src/frontend/)
Mapping a drag/GUI change back to a code edit	Frontend writes into CodeMirror, then asks the kernel to re-render
Coordinate math (plot pixels ↔ data values)	Frontend, using data-*-bounds attributes Python writes onto the SVG
Talking to the LLM	Frontend (utils/llm.ts); key comes from OPENAI_API_KEY env, passed through to attach_snp

---

5. Python backend (src/snp.py and friends)

snp.py is imported into the kernel as the module snp. Highlights, in rough order:

• show(locals, cell_lineno, …, notebook_code_through_cell) — replacement for plt.show(). Just stashes the current figure + context into cell_figs. Returns nothing (so multiple show()s in a cell are fine; only the one is surfaced).
• show_ui(fig_idx=0, snp_class=SNP) — pops one of the stashed figures and returns an instance of one of the render classes. snp_class controls how much work is done:
  • SNPFigureOnly — just _repr_png_(). Used for fast live redraws during drags (it even drops DPI mid-drag, capping at ~1000px wide).
  • SNPFigureAndHoverRegions — adds _repr_svg_(), the interactive overlay with clickable shapes on the plot. Builds a map of artist→name and computes per-artist bounding regions (regions2).
  • SNP (the default, full) — additionally runs mypy inference, gathers typed calls, available methods on each artist, the typed AST, parseable comments, and user-defined iterable variables. Its _repr_html_() assembles the final output: the plot <img>, the hover-region <svg>, an inlined <script> of dist/plugin.js, all frontend/**/*.css inlined, and a <style onload="attach_snp(...)"> that hands every serialized blob to the frontend.
• Provenance tracking — at import time, snp.py installs an IPython AST transformer (IPython.get_ipython().kernel.shell.ast_transformers = [RootProvenanceTagger()]). It wraps every call result in snp.tag_with_call_provenance(value, "ax.plot #1"), attaching _snp_came_from_call_id to the returned objects (including tuple/str/list/int/float/etc, using special included Tagged* subclasses). That id is later written onto the SVG regions (data-call-id=...) so the frontend can map a clicked region to the source call.
• Variable-sharing provenance — the same transformer also tracks how a value reached a matplotlib argument through variable bindings, so the Properties panel can show an editable chain like w2 ▸ w1 ▸ 0.5. It attaches a nested _snp_provenance node to values at each binding (snp.tag_with_var_provenance on assignment RHSes and same-cell user-function arguments), and at each matplotlib call site reads the argument's provenance into a global var_provenance_at_call log keyed by (call_id, arg_key) (snp.note_arg_provenance). snp.show() snapshots that log per figure; GatherTypedCalls looks it up per argument and serializes it (converted to notebook coordinates) as given_arg["provenance"].
• serialize_type / GatherTypedCalls — convert mypy types and call sites into the JSON the widgets need (arg names, kinds, types, defaults, source positions).
• Helpers worth knowing: serialize.arbitrary_to_json (generic object-graph → JSON with id-based cycle handling), artist_names (names every matplotlib artist for the UI), methods_for / method_associations (which possible new function calls to offer on which artist, for on-plot buttons/menus).

---

6. Type inference with mypy

Plottery needs accurate argument types to choose the right widget (slider vs color picker vs dropdown of literals vs free code). It does this by running mypy not as a type checker, but as a type oracle.

do_mypy_inference(code) in snp.py:

• Writes the whole notebook-so-far to a temp file __plottery_mypy_temp.py. This is so the file is in the same place on the filesystem as the user's notebook. Currently, the file is also left on disk for debugging.
• Builds a fine-grained incremental mypy build and caches it across cell runs; it only does a full rebuild when the set of import lines changes (cheap edits reuse the cache).
• Sets options tuned for inference over checking: export_types, preserve_asts, check_untyped_defs=True (so bodies of user functions get inferred), and ignore_errors=True (type errors in the user's code must not break the tool).
• Points mypy_path at the bundled stubs (see §7).

The result exposes mypy_result.types (expression → type) and .graph (the typed module tree), which GatherTypedCalls and serialize_type walk to produce calls, methods, and notebook_typed_ast.

---

7. Type stubs (src/python-type-stubs-main/)

An updated copy of Microsoft's python-type-stubs repo — .pyi files describing the public API of matplotlib, numpy, scipy, sklearn, networkx, etc. mypy reads these (via options.mypy_path = ".../python-type-stubs-main/stubs") to know, e.g., that Axes.bar(x, height, width=..., color=...) takes a color, an array-like, etc., so the UI can render the right control. Only the stubs/ subtree is used at runtime; the rest (tests/, utils/, its own README) is upstream baggage. Also includes types for potentially irrelevant projects.

---

8. Frontend (src/frontend/)

TypeScript, bundled by Vite (vite.config.js) from src/frontend/main.ts into dist/plugin.js. It is not loaded by the Jupyter extensions. The Python SNP class's _repr_html_() result bundles the Javascript inline (on every manual cell re-run) for faster dev cycles.

• main.ts — exposes window.attach_snp(...). Given the cell element plus all the serialized blobs from Python, it builds the State, wires up the sidebar, AI panel, layers panel, plot widgets, hover-region drag handlers, selection sync, and keyboard shortcuts.
• code_sync/code_sync.ts — the heart of "GUI change → code change → re-render". It writes generated code into CodeMirror "marks" (the regions of code that correspond to layers), then runs the cell again via the right kernel API (CM5/Notebook vs CM6/Lab) with a trailing show_ui(snp_class=…), and swaps the returned PNG/SVG in place. Has redraw_cell (fast PNG), refresh_hover_regions (SVG overlay), and hard_rerun (full re-execute, used after AI/structural edits).
• sidebar/ — the inspector. call/call.ts (one matplotlib call + its plot-drag → arg mapping), arg/arg.ts (one argument row), methods/method.ts (insert a new method call from an on-plot button/menu), hover-regions/hover_regions.ts (the SVG overlay: select/drag/add-method), plot-widget/plot_widget.ts (inline on-plot text editing).
• sidebar/widgets/ — the typed controls, chosen from mypy types by widget.ts: code_and_control (primary scalar control: a code box synced to a slider/toggle/picker), int, float, bool, color, literal (enum-like values), alias, arbitrary_code (freeform Python fallback), and dropdown (picks among alternatives).
• layer_panel/layer_panel.ts — a structural list of the cell's code, one row per typed AST node (and parseable comments), with reorder-by-drag and show/hide (comment/uncomment) toggles.
• properties_panel/properties_panel.ts — pops the selected call's args out next to the plot.
• ai_panel/ai_panel.ts — natural-language editing; builds a prompt from the code so far, calls the LLM, replaces the cell text, hard_reruns, and highlights changed lines. Hidden unless OPENAI_API_KEY is set.
• menus/menus.ts — reusable dropdown-menu + keyboard-shortcut framework.
• utils/ — types.ts (a Cell abstraction that papers over CM5/Notebook vs CM6/Lab), deserialize.ts (rebuilds Python's id-based JSON graphs into JS objects — inverse of serialize.py), llm.ts (OpenAI client), instrumentation.ts (study logging to localhost:7777), misc.ts, stdlib.ts, and a vendored CodeMirror 5 codemirror/ type-defs folder.
• types.ts / ast_types.ts — app domain types and a (largely unused) mirror of Python ast nodes.

---

9. Detailed UI interaction walkthroughs

This section traces what happens for common interactions. All paths below are under src/frontend/.

Shared infrastructure (assumed by every walkthrough below)

The State object (types.ts). A single State is built by attach_snp() for each rendered Plottery cell and handed to almost every function — it holds the only references to the live DOM and the editor, so "do X to the plot" almost always means "call something with state." Its fields fall into three groups:

• Static inputs from Python (set once, at build): cell (the CM5/CM6-agnostic Cell, see §8), cell_lineno / plt_show_lineno_in_cell / provenance_is_off_by_n_lines (used to map notebook line numbers ↔ in-cell line numbers), calls / calls_with_args, methods / methods_with_code, notebook_typed_defs, user_iterables, avoid_names (names already in use, for non_colliding_name), and llm_api_key.
• Mutable runtime state: last_cell_code_executed and outstanding_kernel_request_time (drive the busy/changed checks below), dragging_layers (the in-flight layer drag), and is_in_dom() — flips to false after a hard rerun, which is how the per-piece change watchers know to stop their requestAnimationFrame loop.
• View handles: snp_outer, plot_area (the plot <img> lives here), stdout_stderr, hover_regions_container + hover_regions_svg() / set_hover_regions_html(), plot_widgets, sidebar_el, layers_panel, properties_el, and command_shortcuts (the keymap).

Per-cell persistent storage (get_persistent_item / set_persistent_item, utils/misc.ts). A cell re-run rebuilds the entire UI DOM, so a little UI state has to be persisted as strings, keyed per cell, with two backends: on Notebook v6 the cell's .output element dataset (classic cells can lack stable ids); on JupyterLab sessionStorage under cell-<sharedModel.id>-snp-<key> (the labextension reads the same keys). Only three keys exist:

• fig_idx — which stashed figure this cell renders; read into the snp.show_ui(...) postfix on every redraw.
• selected_calls — the selected layers (keyed by call_id); written by save_selected_layers, restored by load_selected_layers so selection survives a redraw.
• new_calls — ids of just-inserted calls, so main.ts can auto-select/focus them after the next rebuild, then clears the key.

The kernel request/response contract (kernel_execute, code_sync.ts). Every kernel path funnels through kernel_execute(code, postfix, on_iopub_output, on_shell_reply, state), which papers over v6 vs. Lab:

• Notebook v6: cell.kernel.execute(code + postfix, callbacks, opts) with callbacks.iopub.output / callbacks.shell.reply set to the two handlers. opts carries doesnt_need_snp_show_ui: true (so the extension won't append a second show_ui) and cell (which the nbextension strips back off before anything reaches Python, to avoid "message too large" crashes).
• JupyterLab: kernel.requestExecute({ code: code + postfix, … }, false, metadata); the returned future's .onIOPub / .onReply get the same two handlers, with doesnt_need_snp_show_ui riding along in metadata. Both backends use store_history: false, stop_on_error: true.

Replies come back as CellMessages (typed in utils/types.ts), dispatched on msg.header.msg_type:

• the rendered plot is an execute_result whose content.data["image/png"] is base64 — the frontend just sets img.src = "data:image/png;base64," + …; hover-region refreshes additionally read content.data["image/svg+xml"].
• status / execute_input (Lab-only chatter) are swallowed.
• everything else goes to handle_error_or_stdout_stderr: error → content.evalue, stream → content.text, appended to state.stdout_stderr, with line numbers shifted back by provenance_is_off_by_n_lines so they point at the user's code, not the lines Plottery prepended.

Two "run one request at a time, always finish on the latest code" guards live here and are assumed by every redraw:

• kernel_is_busy(state) trusts state.outstanding_kernel_request_time first (a request < 500 ms old counts as in-flight, because the kernel's own busy flag lags), then falls back to the real kernel status (v6 _pending_messages / iopub_done; Lab kernel.status !== 'idle').
• the reply handler clears outstanding_kernel_request_time, then re-runs immediately if the cell text changed while the kernel was busy, and otherwise settles (e.g. into refresh_hover_regions).

The object model (layer → call_view → arg_view → widget, plus the marks). The sidebar mirrors the cell's code as a tree of plain objects (types.ts, layer_panel/layer_panel.ts):

• state.layers_panel.layers: Layer[] — one Layer per top-level typed AST statement (and per parseable comment). A Layer is { el, mark, target_mark, calls_with_args, call_views }. Its mark spans the whole statement (used for line-highlighting, show/hide, and as the drag source range); target_mark spans just the displayed header — identical to mark for a one-line statement, but for a for/def it covers the header so a drop "below" it lands inside the block. Block layers recurse: a for/def layer is immediately followed in the flat layers array by its body's sub-layers (with a deeper indent-N class).
• Layer.call_views: CallView[] — one CallView per matplotlib (or user-defined) call on that statement's line. A CallView is { els: { el, header_el, name_el, properties_el }, mark, arguments }; its mark spans that call's source text and is exactly what the call's change-watcher writes into (add_sync_code_on_change_watcher(() => call_to_code(call_view), [mark], state), set up in create_call_view).
• CallView.arguments: { arg: Arg, view: ArgView }[] — one ArgView ({ el, widget, disabled, positional }) per argument row.
• ArgView.widget: Widget — the actual typed control, and the only object that must implement to_code() / set_code().

So one editable argument is reached as layer → call_views[i] → arguments[j].view → widget. marks live at the layer level (whole node + header) and the call level (the call's text); standalone editable bits — for-headers, def names (and all their use-sites), def arg-lists — get their own marks + watchers in layers_from_typed_node.

Why call_views is plural. layers_from_typed_node keeps every call whose call.pos.line equals the statement's line, filtered to matplotlib / user-defined ones (the rest are logged and dropped), so a line like ax.plot(x); ax.set_title("t") produces one layer with two call views. In practice it's almost always 0 or 1 (hence the // Layers in practice only have zero or one calls note in attach_events_to_hover_regions). All of a layer's call views are rendered into the layer row; selecting the layer only lifts call_views[0] (or a specific one passed via select_call_view, e.g. clicking that call's on-plot widget) into the Properties panel — the other calls' args stay visible in the layer row, they are not dropped.

DOM ownership across a redraw. A call's argument rows are a single DOM node (call_view.els.properties_el) that is moved, never copied: set_properties_panel_on does args_el.remove(); properties_el.append(args_el) to lift it next to the plot, and clear_properties_panel (run on every (de)selection) moves the same node back under its call view's el in the layer row. Because redraw_cell / refresh_hover_regions only swap the <img> / SVG (see below), this entire object tree — layers, call views, marks, watchers, and the transplanted panel — survives those paths untouched. Only hard_rerun tears it down: it clears all marks, re-executes, and attach_snp rebuilds a brand-new State and tree (the old state.is_in_dom() flips to false). So a transplant can never dangle — between redraws it's moved home by clear_properties_panel, and a structural change rebuilds everything from scratch (selection is then restored from the persisted selected_calls / new_calls).

The shared "edit → redraw" mechanism. Most GUI edits never call the kernel directly. Instead, each editable piece of code (a call, an argument, a for-loop header, a function name) owns a CodeMirror mark — a tracked text range — plus a polling watcher created by add_sync_code_on_change_watcher(get_code, marks, state) (code_sync/code_sync.ts). On every animation frame it compares get_code() (e.g. call_to_code(call_view)) to its last value; on a change it calls sync_code_range, which writes the new text into the mark with cm.replaceRange(...) and then calls redraw_cell(state). So "change a widget" really means "change the text the widget generates" — the watcher notices and re-runs the cell. Widgets therefore only need a working to_code() / set_code().

The three kernel paths (all in code_sync.ts, all routed through kernel_execute, which handles both Notebook v6 cell.kernel.execute and JupyterLab kernel.requestExecute and sets doesnt_need_snp_show_ui so the extension doesn't append another show_ui):

• redraw_cell — fast. Appends snp.show_ui(..., snp_class=snp.SNPFigureOnly), then swaps the plot <img> with the returned PNG. Guards against overlapping requests (kernel_is_busy) and re-runs if the code changed again while the kernel was busy.
• refresh_hover_regions — appends snp.show_ui(..., snp_class=snp.SNPFigureAndHoverRegions), replaces the SVG overlay, re-attaches drag handlers, and swaps in a higher-DPI PNG. Runs once a drag settles.
• hard_rerun — clears all CodeMirror marks and fully re-executes the cell, so Python rebuilds the entire UI from scratch via _repr_html_. Used whenever the code structure changes (new line, reorder, AI rewrite).

Clicking a layer in the Layers panel

Entry: the click listener in add_listeners_and_checkbox_to_layer (layer_panel/layer_panel.ts) → select_layer(layer, state). No kernel round-trip; this is pure DOM/selection:

1. deselect_all_layers(state) clears the previous selection.
2. Adds the selected class to layer.el.
3. set_properties_panel_on(layer.call_views[0], state) (properties_panel/properties_panel.ts) moves that call's argument-rows node (.snp-args, i.e. call_view.els.properties_el) out of the layer row and into the Properties panel next to the plot. (If selection arrived via select_call_view — e.g. clicking a specific call's on-plot widget — that call view is lifted instead of call_views[0]; see "The object model" above for why this is a move, not a copy, and how it survives redraws.)
4. compute_selected_hover_regions(state) (hover-regions/hover_regions.ts) clears .selected on the SVG, then re-adds it to every [data-call-id] region whose id matches a selected call — this is the on-plot highlight.
5. highlight_lines_for_selected_layers(state) adds gutter/background line classes in CodeMirror over the layer's line range.
6. save_selected_layers(state) persists the selected call_ids to per-cell storage so the selection survives the next cell re-run (restored by load_selected_layers).

Selection is keyed on call_id, so it's robust across redraws (and restored after a hard_rerun by load_selected_layers).

Selection is single, despite the plural names. select_layer calls deselect_all_layers first, so at most one layer is ever selected; the selection state is nothing more than the selected CSS class on layer.el (queried by is_layer_selected). The plural API — selected_layers(), deselect_all_layers, highlight_lines_for_selected_layers, the array written by save_selected_layers — is structural/forward-looking (selected_layers even carries a // Should only be 0 or 1 right now note); nothing currently adds to a selection rather than replacing it. That same select_layer is the single funnel reached from every selection gesture: the layer-row click here, an on-plot region click (a mouseup with no drag), a plot-widget click (via select_call_view), the cursor-tracking below, and the post-rerun restore (new_calls / load_selected_layers).

In CodeMirror 5 (classic Notebook) only, moving the text cursor into a layer's range also selects that layer: main.ts listens for cursorActivity, finds the innermost layer whose mark contains the cursor, and selects it (it bails while the user has an active text selection, to avoid fighting a highlight). The JupyterLab CM6 facade doesn't expose on / getCursor, so this is feature-detected and simply skipped there.

Reordering layers (drag-and-drop)

Layer rows are draggable; handlers live in layer_panel.ts:

• dragstart → state.dragging_layers = [layer].
• dragover computes the drop side (top vs. bottom half of the row) and a target indent level from the mouse X (Math.floor(mouseX / 29)), showing feedback via dragover-top / dragover-bottom + drag-indent-N classes.
• drop does the actual code move:
  1. Reads the chosen indent from the drag-indent-N class and re-indents the dragged layers' source text to match.
  2. Computes the insertion point from the target layer's target_mark (distinct from mark: target_mark spans the displayed header, so dropping just below e.g. a for header lands inside the loop body).
  3. Temporarily sets every mark's inclusiveLeft = false (so marks don't swallow the insertion), inserts with cm.replaceRange, then deletes each source layer's original lines.
  4. clean_up_passes(cm) adds/removes pass statements so emptied or newly-filled blocks stay valid Python.
  5. hard_rerun(state) — structure changed, so rerun the cell to do a full UI rebuild.

replace_all_preserving_marks (same file) is the regex-replace helper used here and by the visibility checkbox; it edits text without clobbering existing marks. The per-layer show/hide checkbox uses it to comment/uncomment the layer's lines, then calls redraw_cell (or hard_rerun when re-enabling a plain-text layer that needs its full UI rebuilt).

Clicking an on-plot button (add a method call)

On-plot buttons are drawn by place_add_method_buttons_on_plot (hover-regions/hover_regions.ts): it groups state.methods_with_code by each method's show_on artist id, skips any method already called its max_calls times, and overlays a button on each matching hover region — a single button (e.g. ax.set_title) when one method applies, or a dropdown menu (labeled with the artist name) when several do. Placement uses place_over_shape / place_centered_over_shape + reposition_to_avoid_overlap.

How show_on finds the right shape — it's matched by Python id(). When Python renders the hover-region SVG, region2_to_svg_g (snp.py) stamps each region <g> with data-artist-id="{id(artist)}" (plus data-artist-name). When it builds the method list in _repr_html_, it walks the hard-coded method_associations(artist) tables (axes_method_associations, fig_method_associations, …); each entry is (children_paths, method_name, max_calls, docstring_first_line), and Python sets show_on = [id(eval("artist" + path)) for path in children_paths] — e.g. an Axes-wide method like bar uses path '.patch' so it shows on ax.patch, while set_title uses '.title' → ax.title. max_calls is the per-method cap from those same tables (1 for set_title, float('inf') for bar). The frontend then groups state.methods_with_code by method.method_info.show_on.at(-1) (the leaf id of the path), drops any whose state.calls.count(...) >= max_calls, and places the button(s) on every region whose data-artist-id equals that id. So the id flows Python id(artist) → data-artist-id and show_on, matched by plain equality in the browser; that's also why a hard_rerun (which re-id()s objects) is required after adding a call, not a fast redraw.

Clicking → add_method_call(method, state) (sidebar/methods/method.ts):

1. add_line_of_code(method.code, state) inserts the generated call (e.g. ax.set_title("")) into the cell immediately above plt.show(), matching its indentation.
2. Records the new call's id in the new_calls per-cell store so it gets auto-selected/focused after the rebuild.
3. hard_rerun(state).

The method.code string was precomputed by method_info_to_method_with_args: from the mypy callee type it emits the required positional + keyword args (with default code) and prefixes name = when the return type isn't None (naming ax / fig / etc. via name_for_ret_type). The same method list also populates the "＋ Add Layer" menu in the Layers panel.

Clicking an argument

Argument rows are built by create_arg_view (sidebar/arg/arg.ts) and live in the Properties panel for the selected call. The value control is always a dropdown (make_widget_for_code_and_type, widgets/widget.ts) wrapping the type-derived widget(s).

• (a) Toggle on/off. Clicking the arg name (prefixEl mousedown) calls disable_arg_view (adds snp-arg-disabled); clicking an already-disabled row calls enable_arg_view. Effect on code: call_to_code (call/call.ts) skips disabled args (view.disabled ? null : arg_view_to_code(...)), so a disabled optional kwarg simply drops out of the call. The watcher sees the changed call_to_code() and redraws. (Required args can't be disabled.)
• (b) Dropdown of suggestions. Clicking the selected value toggles the drawer open (.expanded) (widgets/dropdown/dropdown.ts). Items come from make_widget_for_code_and_type: widgets derived from the mypy type, the parameter's default, and type_compatible_code_snippets (user variables/expressions whose type matches this arg — computed in Python as user_typed_snippets / user_iterables), plus an arbitrary-code fallback. Hovering an item sets dropdown.previewing_code, which to_code() returns, so the live redraw_cell shows a preview of that choice; clicking it calls select_dropdown_item, which clones the chosen widget into the selected-item slot (keeping the previous value available in the drawer).
• (c) Editing the arg code. The selected widget is usually arbitrary_code (a contentEditable box, widgets/arbitrary_code/arbitrary_code.ts) or code_and_control (widgets/code_and_control/code_and_control.ts), which pairs that box with a GUI control (slider / toggle / color) kept in sync by a MutationObserver — editing the text updates the control (or drops it if the text no longer matches the type), and moving the control rewrites the text. Either way the new to_code() flows through the same call watcher → sync_code_range → redraw_cell.

How the (b) suggestions are computed (all in Python). The user-variable items come from two structures Python builds in SNP.__init__ and serializes to the frontend:

• user_typed_snippets — a { code_string: mypy_type } map of expressions that could fill an argument. It starts from the user's variables in scope (names that actually appear in the user's code via get_user_nameset, minus trivial/callable ones), each typed by mypy (tree.names[name].type, or, inside a function where mypy doesn't expose it, a builtins fallback inferred from type(value).__name__). It then goes one level into concrete containers using the live values: d[key], d.keys() / .values() / .items() for dicts; xs[i] per element and np.arange(len(xs)) for lists/tuples.
• user_iterables — the subset whose type is an iterable: is_subtype(t, Iterable[Any]) and not is_subtype(t, str) (so strings aren't offered to loop over), plus dynamic fallbacks for 1-D numpy arrays and lists. This feeds the for-loop iterable widget and the "＋ Add Layer → For-loop over …" menu.

Then, per argument, serialize_type computes type_compatible_code_snippets_by_arg_i = [name for name, t in user_typed_snippets.items() if is_subtype(t, arg_type)] (and the analogous …_by_i for **kwargs TypedDicts). So "type compatible" means a mypy subtype — mypy.subtypes.is_subtype, wrapped in snp.py's is_subtype (line ~1709) to resolve type aliases first and to treat a bare Any snippet as not matching a non-Any parameter (otherwise untyped values would flood every dropdown). It's subtyping, not equality: a list[int] variable is therefore offered for an Iterable / Sequence / ArrayLike parameter, and matplotlib's ArrayLike alias resolves so numpy arrays match wherever it's accepted.

Dragging a shape on the plot

Wiring is in attach_events_to_hover_regions (hover-regions/hover_regions.ts). For each call's hover regions it pulls candidate drag handlers from call/call.ts (perhaps_get_drag_xy_handler, _x_, _y_, and the four edge handlers), then:

• a bare mousemove (no button down) picks which handler applies from proximity to an edge vs. the center, and sets the cursor (col-resize / row-resize / move / …);
• mousedown records the start point and the figure's bounding rect;
• document mousemove while pressed computes the pixel delta, calls the active handler, and hides the overlay during the drag;
• mouseup: if the mouse didn't move, it's a click → select_layer; if it moved → refresh_hover_regions (regenerate the SVG since shapes moved).

The handlers (drag_handler_for_arg_view in call/call.ts) translate pixels → code: they parse the target arg's current code into stuff + number, convert the pixel delta to axis units using boundses — the data-fig-px-bounds / data-axes-px-bounds / data-axes-unit-bounds / data-region-px-bounds attributes Python wrote onto each SVG region — compute the new number, and call view.widget.set_code(...) (plus enable_arg_view, so dragging turns the arg on if it was off). Special cases: legend loc (an xy fraction of the axes), set_title y (axes-size coords), and bar width / height edges (reversed). Each set_code is picked up by the call watcher → fast redraw_cell (DPI is dropped mid-drag for responsiveness); refresh_hover_regions runs once on release.

Submitting an LLM prompt

The AI panel (ai_panel/ai_panel.ts) is an input box, shown only when an API key is present. Pressing Enter → submit_prompt:

1. Disables the input and shows a spinner.
2. prompt_for_llm builds the prompt from every code cell up to and including the current one, delimited with ### Cell N ### markers, plus fixed instructions (how to add subplots, axes type annotations), ending with "Return only the new code of Cell N".
3. prompt_llm (utils/llm.ts) calls OpenAI. On success: strip ``` fences and the cell marker, cm.setValue(newCode) to replace the whole cell, `hard_rerun(state)`, then diff old vs. new lines and tag changed lines with `snp-ai-line-changed` (gutter + background). The cursor is moved to the end so a single Cmd/Ctrl-Z undoes the whole AI edit. On error/timeout it surfaces "Oops there was an error." and re-enables the input.

attach_ai_line_highlight_clearing_handlers clears those change highlights the next time the cell is run manually.