feat(annotator): interactive suggest tool with ephemeral preview (#424 slice 3b) - #451
Merged
Conversation
…slice 3b) The user-facing half of click-to-suggest: a sparkles tool on the strip (`S`), clicks that prompt the model, a preview that lives outside the document and the undo stack, and acceptance into one ordinary annotation carrying `provenance: model`. Core (`@visionset/annotator`): - `core/interaction/suggestion.ts` — the session as a pure state machine beside the pointer's. It produces no `Effect`, holds no store and mints no id, which is what makes ephemerality structural rather than a rule. `acceptedAnnotation` is the one door into the document and it builds on `draftAnnotation`. - `s` joins `DEFAULT_BINDINGS` as a host row; `enter` and `escape` are adapter substitutions, so `machine.ts`'s cancel rows are untouched. Adapter: - An armed session is the second occupant of `state.ts`'s pan contract: the machine hears no presses and a gesture in flight is cancelled on arming. - `paint.ts` decides the preview's treatment — reduced opacity plus a dashed stroke, never colour alone — and `TransientLayer` draws it with the prompt points. `@visionset/ui-core`: - `inferenceQueries.ts` — the connection list, fetched only once the tool is armed, and the suggest call. - `SuggestPanel.tsx` — D6's in-editor panel; its action is a callback, and its absence renders the explanation with no control. - `ToolPalette` gains the button, hidden on a schema no class of which can hold the answer. Docs: the gesture in `docs/ui.md`, the write it produces in `docs/annotations.md`. cf. #424, #417, #418, #421
This was referenced Aug 8, 2026
JArmandoAnaya
added a commit
that referenced
this pull request
Aug 9, 2026
…tion is two lines (#473) * fix(ui): a select option is two lines when it is an identifier plus its facts The curated model select put the id, the download size and the hint on one line, inside a trigger measured for one. The text wrapped and read squashed. `SelectItem` takes a `meta`: the children stay the identifier at the label role, `meta` goes underneath at the meta role. Radix renders the selected item's own `ItemText` into the trigger, so the closed control and the open list are the same two lines by construction rather than by a second copy of the layout at the call site. The trigger is `min-h-9` with `py-1` instead of a fixed `h-9`: a one-line option still measures exactly 36px, so every select that shipped before this is unmoved, and a two-line one grows to fit. Nothing truncates and nothing ellipsises — half a model id is not a model id. The styleguide gains the specimen, and its e2e measures what jsdom cannot: the one-line trigger on 36px, the two-line one taller. * fix(annotator): the suggest tool survives a class switch, and parks rather than dying Selecting a different class silently disarmed the tool. That was an artifact of #451, which read "switching tools discards" (D2) and treated moving the active class as switching tools. Directed (Armando, 2026-08-09): arming is a decision about how to work and picking a class is the next thing somebody does, so a class switch ends what is pending and not the session. `withClass` in the core is the whole rule, and it is one function because the three readings are one transition over a different argument: - nothing pending — the class is swapped in, and the next click asks under it; - a preview showing — it is discarded with the clicks that produced it, since it was answered under the old class's `allowed_geometries` and accepting it under the new one could write a kind that class does not admit; - a class that can hold nothing — the session parks, `labelClass` is `null`, no press is diverted, and the armed intent is kept so that returning to a class which can hold an answer picks up with no second press. The serial keeps counting across all three, so an answer to the ask that was in flight when the class moved cannot repaint a preview nobody is on. Parked is a state with a sentence, not a control that goes quiet: the strip button is dimmed with its reason (lit and dimmed at once — both are true), and the panel names the class, says what to pick, says the tool is still armed, and carries the way out, since the button it would otherwise be is the one that is dimmed. The canvas is handed `null` while parked, because the class that parked it may still be drawable and a tool that swallowed those presses would have stopped being parked. Switching assets still disarms and discards, through the `key={asset.id}` remount, unchanged.
JArmandoAnaya
added a commit
that referenced
this pull request
Aug 10, 2026
…preview, selected-only labels (#514) * fix(annotator): a suggest click outside the asset is not a prompt The pane spans the whole stage on purpose, so a press in the margin around the picture reaches the adapter with a coordinate outside the frame. A drag wants that — "make the box this big" survives leaving the picture. A prompt point cannot use it: there is nothing under the margin to segment. Measured before the fix, a click at (900, 700) on a 640x480 asset sent positive: [{"x":900,"y":700}], painted a dot there, and drew the answer. `withinBounds` joins `clampPoint` in core/geometry/primitives — the two questions a frame is asked, and the difference is whether a stray coordinate is work to be salvaged or an instruction that was never given. The adapter drops the press before the host hears about it, so no point is recorded, no request leaves and no preview moves: one guarantee rather than three. Asked in asset pixels, so zoom and pan are already accounted for. cf. #451. * fix(annotator): the canvas says the class, and only for the shape you picked Two rules on one element. The class label renders **only while its shape is selected** — a frame of forty boxes drew forty class names over the picture at all times, which hid the asset behind the annotations of it. The panel is the full inventory; the canvas answers what *this* one is. That rule is DESIGN.md's own, written under the v1 metrics kept as the reference; the React adapter has rendered the label unconditionally since the layer was first written, so this is the adapter starting to obey a rule that was already recorded. And the label says the class and nothing else. A confidence tells somebody whether to accept a proposal; once accepted the shape is a label like any other, so the number stays on the live suggestion preview and leaves the rest of the editor — no percentage on a canvas box, on a panel row, or in a tooltip. The Sparkles glyph is now the only provenance signal, and its tooltip carries the model_ref alone. Nothing is discarded: confidence and model_ref are stored unchanged, and the number's home is the batch review loop. `PaintedAnnotation` gives back the two fields it gained for the old label. A projected field with no reader is what a renderer starts writing next; absence is the rule, enforced where a component cannot reach around it. `confidencePercent` stays exported — one consumer today, and it is the spelling whoever shows the number next must import rather than respell. cf. #417, #425, #451, #512. * test(annotator): the committed layer's element budget is 440, not 660 The itemised count is what the scene test exists to hold, and a selected-only label moves it: two elements per annotation on a frame nobody has picked a shape on rather than three. The `<text>` was the most expensive of the three — a stroke, a paint order and a translate each — so a legibility decision pays here as well.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The user-facing half of click-to-suggest (#424, slice 3b), on top of the route and
adapter slice 3a landed in #448. Click the thing you want, get a shape, refine it,
accept it — or find out in the editor why you cannot.
No server changes. The route, the wire models,
openapi.jsonand the generatedclient are all exactly as #448 left them, which is why the
generatedgate reportsno drift.
What ships
The engine (
@visionset/annotator)core/interaction/suggestion.tsis the session as a pure state machine beside thepointer's, not a variant inside it.
InteractionStateis what the pointer is inthe middle of and
machine.tstypes its table as total over that union — everystate owes a row and every row answers a pointer event synchronously. A suggestion
outlives the press that asked for it, is resolved by a server rather than by a
pointer-up, and survives
pointer-cancel, blur and a hundred pointer-moves. A rowthere would have had to answer eight events it has no opinion about, and the one
event it does care about is not a pointer event at all.
D4's ephemerality is structural rather than a rule. The module produces no
Effect, holds no store and mints no id, so there is no mechanism by which apreview could reach
AnnotatorStore.acceptedAnnotationis the single door intothe document, and it builds on
draftAnnotationso a suggestion inherits theclass's attribute defaults, the asset and the provisional
schema_versionexactlyas a hand-drawn shape does — overriding only
provenance,model_refandconfidence.draftAnnotation's ownprovenance: "human"is untouched and itsdocstring stays true: acceptance is the human act the provenance records.
Refine clicks send the accumulated points every time, never a diff, because the
route is stateless. Each ask stamps a serial and a late answer is dropped whole
(returned by identity), so a slow first answer cannot overwrite a fast second one —
the failure mode a click-to-refine gesture has by construction.
sjoinsDEFAULT_BINDINGSas a host row.enterandescapestay exactly whatthey were: accepting and discarding are adapter substitutions over those two
rows, the same mechanism #383 introduced for the flow verb, so every cancel row in
machine.tsreads as written and a secondenterbinding cannot shadow the ringclose. The text-entry guard now reads the resolved action rather than the
substituted one, so Escape still blurs a field whatever it was substituted into.
The adapter
An armed session is the second occupant of a contract that already existed.
state.tsstates it for panning: "while panning or pinching, the adapter does notforward pointer events to the machine; if a gesture was in flight when the pan
began, it sends
pointer-cancelfirst." Arming does exactly that, and afterwards aprimary press becomes a prompt point instead of a
pointer-down. That is what stopsa click meant for the model from drawing a box.
Suggest is deliberately not a
Tool.tool.tsderives the tool from the activeclass and stores nothing, and it is emphatic about why (v1 held both and spent two
mechanisms keeping them from disagreeing). There is no class that means "ask a
model", so a fifth variant would be a stored mode wearing a derived one's name with
nothing for
toolForto derive it from. It is a mode held beside the class itborrows.
paint.tsdecides the treatment andTransientLayerdraws it: reduced opacity plusa dashed outline, never colour alone — the shape draws in its class's own
colour, so hue cannot be what distinguishes it. The class and the model's confidence
render with it, and the prompt points stay on screen through an in-flight ask so a
refine click does not look dropped.
The product (
@visionset/ui-core)inferenceQueries.ts— the connection list, fetched only once the tool isarmed (a job nobody suggests on makes no inference request at all), and the
suggest call itself.
usableConnectionanswers with a connection or with one ofthree blockers,
checkingamong them, so a click landing in the loading window istold something rather than vanishing.
SuggestPanel.tsx— D6's in-editor panel, and the tool's one voice for all fivethings it can be saying. Not a toast, not a redirect, not a modal: principle 10 is
immovable and no exit may lose work. A server refusal is quoted verbatim,
because
LOCAL_INFERENCE_UNAVAILABLEisexpose_message=Trueprecisely so theinstall command reaches a person and a sentence written in the client would throw
it away.
ToolPalettegains the sparkles button, hidden rather than disabled on aschema no class of which can hold the answer (D3's third case) — the same rule
that keeps a
classification_tagoff the strip. A disabled control has to beexplicable in principle 9's terms, and "no class in this project could accept the
answer" is a fact about the schema, not a capability that is coming.
AnnotationPageholds the session inWorkspacerather than inJobScreen,which is the opposite call from
activeClassandclipboard— and thedifference is what those two exist for. They sit one level up so they survive the
per-asset remount; a suggestion must not, and D2's "switching assets discards" is
therefore enforced by
key={asset.id}rather than by an effect somebody has toremember. Switching tools is one effect on
activeClass, which every door — thestrip, the panel list, a digit hotkey — already ends at.
Which D-positions ship here
S, left-click positive, alt-click negative,↵accepts,Escdiscards, tool/asset switch discardsallowed_geometries, and the tool is not offered where neither is holdableannotatedentry ratified 2026-08-07alt-click is the negative point and right-click is not. D2 offers both; asecondary press on this canvas is a pan, and #380's
contextmenunote sets out whattaking it back would cost. Alt is the spelling that costs no existing gesture.
Mutation verification
Four mutations, each applied to the committed tree, each asserted present before and
after, each reverted by
git apply -Ron its recorded patch with a clean treeconfirmed afterwards.
onSuccessexecutes the annotation as soon as the answer landssuggestFlow.test.tsx› leaves undo untouched while a suggestion is only showingacceptedAnnotationreturnsdraftAnnotation's outputsuggestFlow.test.tsx› writes it through the ordinary create path, carrying where it came fromThe two the dispatch asked to be named are the first two rows.
Test plan
Boundary gates first, because the design was checked against them rather than the
other way round:
pnpm --filter @visionset/annotator lintruns all three (ESLintno-restricted-imports,no-restricted-globals, and the DOM-freetsconfig.core.jsonpass) and all three are green. Nothing insrc/core/gained aReact import, a DOM global or a DOM type in a signature.
core/interaction/suggestion.test.ts— 27 tests: which classes the tool isoffered for, the preview lifecycle, staleness by serial, Escape as the preview's
undo, acceptance, and a whole click→answer→refine→answer session driven against a
real
AnnotatorStoreprovingdocumentis unchanged by reference andcanUndonever moved.suggestFlow.test.tsx— 16 tests throughAnnotationPagewith a stubbed routetable, so what is asserted is the request that actually leaves.
suggestPanel.test.tsx— 16 tests: the three blocked readings, thecallback-absent rule, and the five session readings.
toolPalette.test.tsx— 6 new: offered, hidden on a tag-only schema, lit from themode rather than the derived tool, and the focus rule.
bindings.test.ts—sandmod+sare different chords and neither shadows theother.
paint.test.ts— 7 new: two distinguishing signals, the confidence label at bothends, and
nullfor every status butshown.Gate
bash scripts/check.shrun in stages — the harness kills a command at ~10minutes and the full run is well past that — with pytest split by the directories
ls tests/reported at run time. Every exit code verbatim, from the run afterthe rebase described below.
pytest tests/architecturepytest tests/clipytest tests/examplespytest tests/formatspytest tests/inferencepytest tests/jobspytest tests/kernelpytest tests/mcppytest tests/packagingpytest tests/serverpytest tests/test_versioning.pyruff check .ruff format --check .mypy src/visionsetlint-imports(4 contracts kept, 0 broken)check.sh frontend(build, tests, lint)check.sh generated(openapi, client, mcp reference, version sync)tests/scriptsis not a pytest target — it holds thenode --testgates, which thefrontendandgeneratedgroups run — so it is absent above rather than reportedas exit 5.
Frontend totals: annotator 904 passed (34 files), ui-core 780 passed (44 files).
E2e: 230 passed, 1 flaky. The flaky one is
e2e/gallery.spec.ts› tiles neveroverlap, at any density, which passed on retry; this diff touches no gallery code
and no layout, and the same run before the rebase reported 231/231 green.
Rebased onto #449
The gate first ran at base
714d18f, wherepytest tests/kernelexited 1 on thetwo truncated-clip tests tracked as #444.
8785740(#449) landed mid-run and fixesexactly those, so this branch was rebased onto it and the whole gate re-run
rather than the delta — the rule about a rebase bringing in commits you did not
write.
tests/kernelis green above on its own merits and no baseline-proofexception was used or needed.
Information architecture
No route, tab, screen, nav entry or entry point moves, so the sitemap is unchanged.
D6's action is an optional callback and
frontend/appdeliberately leaves itunwired: the Inference surface waits on #421's open rail question, and
ui-corerenders no dead control when a callback is absent.
Found, not fixed
Nothing.
What remains on #424
and its download size. That form is Inference section — user-configured model connections #421's surface and does not exist yet.
passes none until there is an Inference screen to send somebody to.
cf. #424, #417, #418, #421, #425, #448