Skip to content

feat(annotator): interactive suggest tool with ephemeral preview (#424 slice 3b) - #451

Merged
JArmandoAnaya merged 1 commit into
mainfrom
feat/suggest-tool
Aug 8, 2026
Merged

feat(annotator): interactive suggest tool with ephemeral preview (#424 slice 3b)#451
JArmandoAnaya merged 1 commit into
mainfrom
feat/suggest-tool

Conversation

@JArmandoAnaya

Copy link
Copy Markdown
Contributor

The user-facing half of click-to-suggest (#424, slice 3b), on top of the route and
adapter slice 3a landed in #448. Click the thing you want, get a shape, refine it,
accept it — or find out in the editor why you cannot.

No server changes. The route, the wire models, openapi.json and the generated
client are all exactly as #448 left them, which is why the generated gate reports
no drift.

What ships

The engine (@visionset/annotator)

core/interaction/suggestion.ts is the session as a pure state machine beside the
pointer's
, not a variant inside it. InteractionState is what the pointer is in
the middle of and machine.ts types its table as total over that union — every
state owes a row and every row answers a pointer event synchronously. A suggestion
outlives the press that asked for it, is resolved by a server rather than by a
pointer-up, and survives pointer-cancel, blur and a hundred pointer-moves. A row
there would have had to answer eight events it has no opinion about, and the one
event it does care about is not a pointer event at all.

D4's ephemerality is structural rather than a rule. The module produces no
Effect, holds no store and mints no id, so there is no mechanism by which a
preview could reach AnnotatorStore. acceptedAnnotation is the single door into
the document, and it builds on draftAnnotation so a suggestion inherits the
class's attribute defaults, the asset and the provisional schema_version exactly
as a hand-drawn shape does — overriding only provenance, model_ref and
confidence. draftAnnotation's own provenance: "human" is untouched and its
docstring stays true: acceptance is the human act the provenance records.

Refine clicks send the accumulated points every time, never a diff, because the
route is stateless. Each ask stamps a serial and a late answer is dropped whole
(returned by identity), so a slow first answer cannot overwrite a fast second one —
the failure mode a click-to-refine gesture has by construction.

s joins DEFAULT_BINDINGS as a host row. enter and escape stay exactly what
they were: accepting and discarding are adapter substitutions over those two
rows, the same mechanism #383 introduced for the flow verb, so every cancel row in
machine.ts reads as written and a second enter binding cannot shadow the ring
close. The text-entry guard now reads the resolved action rather than the
substituted one, so Escape still blurs a field whatever it was substituted into.

The adapter

An armed session is the second occupant of a contract that already existed.
state.ts states it for panning: "while panning or pinching, the adapter does not
forward pointer events to the machine; if a gesture was in flight when the pan
began, it sends pointer-cancel first."
Arming does exactly that, and afterwards a
primary press becomes a prompt point instead of a pointer-down. That is what stops
a click meant for the model from drawing a box.

Suggest is deliberately not a Tool. tool.ts derives the tool from the active
class and stores nothing, and it is emphatic about why (v1 held both and spent two
mechanisms keeping them from disagreeing). There is no class that means "ask a
model", so a fifth variant would be a stored mode wearing a derived one's name with
nothing for toolFor to derive it from. It is a mode held beside the class it
borrows.

paint.ts decides the treatment and TransientLayer draws it: reduced opacity plus
a dashed outline, never colour alone — the shape draws in its class's own
colour, so hue cannot be what distinguishes it. The class and the model's confidence
render with it, and the prompt points stay on screen through an in-flight ask so a
refine click does not look dropped.

The product (@visionset/ui-core)

  • inferenceQueries.ts — the connection list, fetched only once the tool is
    armed
    (a job nobody suggests on makes no inference request at all), and the
    suggest call itself. usableConnection answers with a connection or with one of
    three blockers, checking among them, so a click landing in the loading window is
    told something rather than vanishing.
  • SuggestPanel.tsx — D6's in-editor panel, and the tool's one voice for all five
    things it can be saying. Not a toast, not a redirect, not a modal: principle 10 is
    immovable and no exit may lose work. A server refusal is quoted verbatim,
    because LOCAL_INFERENCE_UNAVAILABLE is expose_message=True precisely so the
    install command reaches a person and a sentence written in the client would throw
    it away.
  • ToolPalette gains the sparkles button, hidden rather than disabled on a
    schema no class of which can hold the answer (D3's third case) — the same rule
    that keeps a classification_tag off the strip. A disabled control has to be
    explicable in principle 9's terms, and "no class in this project could accept the
    answer" is a fact about the schema, not a capability that is coming.
  • AnnotationPage holds the session in Workspace rather than in JobScreen,
    which is the opposite call from activeClass and clipboard — and the
    difference is what those two exist for. They sit one level up so they survive the
    per-asset remount; a suggestion must not, and D2's "switching assets discards" is
    therefore enforced by key={asset.id} rather than by an effect somebody has to
    remember. Switching tools is one effect on activeClass, which every door — the
    strip, the panel list, a digit hotkey — already ends at.

Which D-positions ship here

Where
D2 the gesture ships in full: sparkles tool, S, left-click positive, alt-click negative, accepts, Esc discards, tool/asset switch discards
D3 geometry follows the schema ships: the class's own kinds go up as allowed_geometries, and the tool is not offered where neither is holdable
D4 ephemeral preview, acceptance enters history ships in full, including the annotated entry ratified 2026-08-07
D6 the no-connection panel ships; its action destination is an optional callback and the app leaves it unwired, per the dispatch
D1 SAM 2 serving shipped in #448 (slice 3a)
D5 server-side inference through the connection shipped in #448 (slice 3a)

alt-click is the negative point and right-click is not. D2 offers both; a
secondary press on this canvas is a pan, and #380's contextmenu note sets out what
taking it back would cost. Alt is the spelling that costs no existing gesture.

Mutation verification

Four mutations, each applied to the committed tree, each asserted present before and
after, each reverted by git apply -R on its recorded patch with a clean tree
confirmed afterwards.

Mutation Tests that turned red
the preview enters the undo stack — onSuccess executes the annotation as soon as the answer lands 5, led by suggestFlow.test.tsxleaves undo untouched while a suggestion is only showing
acceptance drops the three overrides — acceptedAnnotation returns draftAnnotation's output 3, led by suggestFlow.test.tsxwrites it through the ordinary create path, carrying where it came from
the D6 panel renders its action without a destination 2: renders the explanation and no control when the host wires no destination
the press is no longer diverted from the interaction machine 1: does not draw a shape — the press never reaches the interaction machine

The two the dispatch asked to be named are the first two rows.

Test plan

Boundary gates first, because the design was checked against them rather than the
other way round: pnpm --filter @visionset/annotator lint runs all three (ESLint
no-restricted-imports, no-restricted-globals, and the DOM-free
tsconfig.core.json pass) and all three are green. Nothing in src/core/ gained a
React import, a DOM global or a DOM type in a signature.

  • core/interaction/suggestion.test.ts — 27 tests: which classes the tool is
    offered for, the preview lifecycle, staleness by serial, Escape as the preview's
    undo, acceptance, and a whole click→answer→refine→answer session driven against a
    real AnnotatorStore proving document is unchanged by reference and
    canUndo never moved.
  • suggestFlow.test.tsx — 16 tests through AnnotationPage with a stubbed route
    table, so what is asserted is the request that actually leaves.
  • suggestPanel.test.tsx — 16 tests: the three blocked readings, the
    callback-absent rule, and the five session readings.
  • toolPalette.test.tsx — 6 new: offered, hidden on a tag-only schema, lit from the
    mode rather than the derived tool, and the focus rule.
  • bindings.test.tss and mod+s are different chords and neither shadows the
    other.
  • paint.test.ts — 7 new: two distinguishing signals, the confidence label at both
    ends, and null for every status but shown.

Gate

bash scripts/check.sh run in stages — the harness kills a command at ~10
minutes and the full run is well past that — with pytest split by the directories
ls tests/ reported at run time. Every exit code verbatim, from the run after
the rebase described below.

Stage Exit
pytest tests/architecture 0
pytest tests/cli 0
pytest tests/examples 0
pytest tests/formats 0
pytest tests/inference 0
pytest tests/jobs 0
pytest tests/kernel 0
pytest tests/mcp 0
pytest tests/packaging 0
pytest tests/server 0
pytest tests/test_versioning.py 0
ruff check . 0
ruff format --check . 0
mypy src/visionset 0
lint-imports (4 contracts kept, 0 broken) 0
check.sh frontend (build, tests, lint) 0
check.sh generated (openapi, client, mcp reference, version sync) 0
annotator + app e2e (chromium) 0
browser cycle, real server (chromium) 0

tests/scripts is not a pytest target — it holds the node --test gates, which the
frontend and generated groups run — so it is absent above rather than reported
as exit 5.

Frontend totals: annotator 904 passed (34 files), ui-core 780 passed (44 files).
E2e: 230 passed, 1 flaky. The flaky one is e2e/gallery.spec.tstiles never
overlap, at any density
, which passed on retry; this diff touches no gallery code
and no layout, and the same run before the rebase reported 231/231 green.

Rebased onto #449

The gate first ran at base 714d18f, where pytest tests/kernel exited 1 on the
two truncated-clip tests tracked as #444. 8785740 (#449) landed mid-run and fixes
exactly those, so this branch was rebased onto it and the whole gate re-run
rather than the delta — the rule about a rebase bringing in commits you did not
write. tests/kernel is green above on its own merits and no baseline-proof
exception was used or needed.

Information architecture

No route, tab, screen, nav entry or entry point moves, so the sitemap is unchanged.
D6's action is an optional callback and frontend/app deliberately leaves it
unwired: the Inference surface waits on #421's open rail question, and ui-core
renders no dead control when a callback is absent.

Found, not fixed

Nothing.

What remains on #424

  • D1's "suggested default" — pre-filling the local-connection form with SAM 2
    and its download size. That form is Inference section — user-configured model connections #421's surface and does not exist yet.
  • D6's destination — the panel's action is wired to a callback prop; the app
    passes none until there is an Inference screen to send somebody to.

cf. #424, #417, #418, #421, #425, #448

…slice 3b)

The user-facing half of click-to-suggest: a sparkles tool on the strip (`S`),
clicks that prompt the model, a preview that lives outside the document and the
undo stack, and acceptance into one ordinary annotation carrying
`provenance: model`.

Core (`@visionset/annotator`):
- `core/interaction/suggestion.ts` — the session as a pure state machine beside
  the pointer's. It produces no `Effect`, holds no store and mints no id, which
  is what makes ephemerality structural rather than a rule. `acceptedAnnotation`
  is the one door into the document and it builds on `draftAnnotation`.
- `s` joins `DEFAULT_BINDINGS` as a host row; `enter` and `escape` are adapter
  substitutions, so `machine.ts`'s cancel rows are untouched.

Adapter:
- An armed session is the second occupant of `state.ts`'s pan contract: the
  machine hears no presses and a gesture in flight is cancelled on arming.
- `paint.ts` decides the preview's treatment — reduced opacity plus a dashed
  stroke, never colour alone — and `TransientLayer` draws it with the prompt
  points.

`@visionset/ui-core`:
- `inferenceQueries.ts` — the connection list, fetched only once the tool is
  armed, and the suggest call.
- `SuggestPanel.tsx` — D6's in-editor panel; its action is a callback, and its
  absence renders the explanation with no control.
- `ToolPalette` gains the button, hidden on a schema no class of which can hold
  the answer.

Docs: the gesture in `docs/ui.md`, the write it produces in `docs/annotations.md`.

cf. #424, #417, #418, #421
@JArmandoAnaya
JArmandoAnaya merged commit 8b38eed into main Aug 8, 2026
13 checks passed
@JArmandoAnaya
JArmandoAnaya deleted the feat/suggest-tool branch August 8, 2026 16:34
JArmandoAnaya added a commit that referenced this pull request Aug 9, 2026
…tion is two lines (#473)

* fix(ui): a select option is two lines when it is an identifier plus its facts

The curated model select put the id, the download size and the hint on one
line, inside a trigger measured for one. The text wrapped and read squashed.

`SelectItem` takes a `meta`: the children stay the identifier at the label
role, `meta` goes underneath at the meta role. Radix renders the selected
item's own `ItemText` into the trigger, so the closed control and the open
list are the same two lines by construction rather than by a second copy of
the layout at the call site.

The trigger is `min-h-9` with `py-1` instead of a fixed `h-9`: a one-line
option still measures exactly 36px, so every select that shipped before this
is unmoved, and a two-line one grows to fit. Nothing truncates and nothing
ellipsises — half a model id is not a model id.

The styleguide gains the specimen, and its e2e measures what jsdom cannot:
the one-line trigger on 36px, the two-line one taller.

* fix(annotator): the suggest tool survives a class switch, and parks rather than dying

Selecting a different class silently disarmed the tool. That was an artifact
of #451, which read "switching tools discards" (D2) and treated moving the
active class as switching tools. Directed (Armando, 2026-08-09): arming is a
decision about how to work and picking a class is the next thing somebody
does, so a class switch ends what is pending and not the session.

`withClass` in the core is the whole rule, and it is one function because the
three readings are one transition over a different argument:

- nothing pending — the class is swapped in, and the next click asks under it;
- a preview showing — it is discarded with the clicks that produced it, since
  it was answered under the old class's `allowed_geometries` and accepting it
  under the new one could write a kind that class does not admit;
- a class that can hold nothing — the session parks, `labelClass` is `null`,
  no press is diverted, and the armed intent is kept so that returning to a
  class which can hold an answer picks up with no second press.

The serial keeps counting across all three, so an answer to the ask that was
in flight when the class moved cannot repaint a preview nobody is on.

Parked is a state with a sentence, not a control that goes quiet: the strip
button is dimmed with its reason (lit and dimmed at once — both are true),
and the panel names the class, says what to pick, says the tool is still
armed, and carries the way out, since the button it would otherwise be is the
one that is dimmed. The canvas is handed `null` while parked, because the
class that parked it may still be drawable and a tool that swallowed those
presses would have stopped being parked.

Switching assets still disarms and discards, through the `key={asset.id}`
remount, unchanged.
JArmandoAnaya added a commit that referenced this pull request Aug 10, 2026
…preview, selected-only labels (#514)

* fix(annotator): a suggest click outside the asset is not a prompt

The pane spans the whole stage on purpose, so a press in the margin around the
picture reaches the adapter with a coordinate outside the frame. A drag wants
that — "make the box this big" survives leaving the picture. A prompt point
cannot use it: there is nothing under the margin to segment. Measured before
the fix, a click at (900, 700) on a 640x480 asset sent
positive: [{"x":900,"y":700}], painted a dot there, and drew the answer.

`withinBounds` joins `clampPoint` in core/geometry/primitives — the two
questions a frame is asked, and the difference is whether a stray coordinate is
work to be salvaged or an instruction that was never given. The adapter drops
the press before the host hears about it, so no point is recorded, no request
leaves and no preview moves: one guarantee rather than three. Asked in asset
pixels, so zoom and pan are already accounted for.

cf. #451.

* fix(annotator): the canvas says the class, and only for the shape you picked

Two rules on one element. The class label renders **only while its shape is
selected** — a frame of forty boxes drew forty class names over the picture at
all times, which hid the asset behind the annotations of it. The panel is the
full inventory; the canvas answers what *this* one is. That rule is DESIGN.md's
own, written under the v1 metrics kept as the reference; the React adapter has
rendered the label unconditionally since the layer was first written, so this
is the adapter starting to obey a rule that was already recorded.

And the label says the class and nothing else. A confidence tells somebody
whether to accept a proposal; once accepted the shape is a label like any
other, so the number stays on the live suggestion preview and leaves the rest
of the editor — no percentage on a canvas box, on a panel row, or in a tooltip.
The Sparkles glyph is now the only provenance signal, and its tooltip carries
the model_ref alone. Nothing is discarded: confidence and model_ref are stored
unchanged, and the number's home is the batch review loop.

`PaintedAnnotation` gives back the two fields it gained for the old label. A
projected field with no reader is what a renderer starts writing next;
absence is the rule, enforced where a component cannot reach around it.
`confidencePercent` stays exported — one consumer today, and it is the spelling
whoever shows the number next must import rather than respell.

cf. #417, #425, #451, #512.

* test(annotator): the committed layer's element budget is 440, not 660

The itemised count is what the scene test exists to hold, and a selected-only
label moves it: two elements per annotation on a frame nobody has picked a
shape on rather than three. The `<text>` was the most expensive of the three —
a stroke, a paint order and a translate each — so a legibility decision pays
here as well.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant