Skip to content

feat(ui): replace the stacked operations bar with a one-line strip and an Activity page - #11163

Merged
mudler merged 26 commits into
masterfrom
feat/activity-operations-ui
Jul 28, 2026
Merged

feat(ui): replace the stacked operations bar with a one-line strip and an Activity page#11163
mudler merged 26 commits into
masterfrom
feat/activity-operations-ui

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Problem

OperationsBar renders one row per in-flight operation above every page. Queue four model installs and a backend and it takes most of the viewport, on every route, until the last one finishes.

Two things were conflated: a global "something is happening" signal, which needs one line, and the detail of what is happening, which needs a page. There was also no record: /api/operations drops an operation the moment it succeeds, so a user who stepped away had no way to learn whether an install finished, failed, or never started.

What this does

The strip collapses to a single line, permanently. It shows one operation, a failure first, otherwise the least-advanced running one (which is already what /api/operations sorts first, so it is the most stable choice across polls). A +N more pill links to the new page.

Its now hides the strip and never cancels. This is a deliberate behaviour change: previously the same glyph cancelled a 17 GB download in one row and dismissed a message in the next. Cancelling moved to the page, behind a labelled button.

The Activity page at /app/activity, admin only, under Operate:

  • In progress with the detail the strip has to drop: phase, bytes, derived time remaining, and an expandable per-node breakdown for cluster installs.
  • Needs attention for failures not yet acknowledged, with Cancel and Retry.
  • The record of what finished. Dismissing a failure moves it into the record rather than deleting it, which is why a failure can appear in either place.

History is a bounded 50-entry in-memory ring in OpCache, appended from the two places an operation leaves the cache (DeleteUUID locally, applyEnd over NATS) and deduped by job ID, since the originating replica does both. Two new admin-gated endpoints expose and clear it. GET /api/operations is unchanged: it is polled once a second by every open tab, so the record does not ride along with it.

The sidebar Operate entry carries a count of running plus queued operations; an unacknowledged failure turns it red and it counts failures instead. That count is the one signal that outlives a hidden strip.

Bugs found and fixed along the way

Several of these were pre-existing or latent rather than introduced here:

  • Retry on a failed removal re-downloaded the model. OpStatus.Deletion was set at admission by markQueued and then lost, because UpdateStatus replaces the whole status and only carried Nodes forward. So isDeletion reached the client only during the queued window, which both surfaces skip. It now carries forward alongside Nodes. This also un-deadened the "Removing" branches: a running delete used to render "Installing model X", and a successful one showed a green "Installed model X".
  • isQueued was near-dead. It was set only when the gallery status was nil, but markQueued publishes a status at admission, so a genuinely queued operation rendered as "Installing" with a spinner. Now keyed on PhaseQueued.
  • isCancelled was unemittable. Every writer of Cancelled = true also sets Processed = true, and /api/operations filters those out. The "Cancelling" branches were removed rather than left as unreachable code a future reader would assume worked.
  • Cancelling the last operation showed "Installed model X" for four seconds, because the hold guard tested the dead isCancelled.
  • A long error message pushed the page ~270px past the viewport, giving every page a horizontal scrollbar. Fixed with min-width: 0 on .main-content, which also removes pre-existing horizontal overflow on /app/models at 375px.
  • The ETA blanked for every operation whenever one was verifying or committing. Byte-flat operations were indistinguishable from ones awaiting a second sample, so a multi-shard model blanked the row for minutes. The gate now keys on phase.

Testing

  • Go: 4 new spec files across core/services/galleryop and core/http/routes, all verified red against the unfixed code before being relied on.
  • Playwright: 327 passing, including new specs for the strip and the page. e2e/model-artifact-operation.spec.js updated for the new markup.
  • make lint clean, npm run lint at the pre-existing 599 baseline, hugo builds 204 pages clean.

make test-coverage-check was not run: the host disk is at 100%, and ./core died in the linker with "no space left on device". coverage-baseline.txt is untouched at 54.2, and the new Go code is roughly 300 lines against 539 lines of new Go specs, so the ratio should rise, but CI needs to confirm it.

A note on test discipline, since it caused a real bug here: two Playwright specs were green over isDeletion: true, a payload the server could not actually emit. Assertions about /api/operations field values belong in Go specs against the real handler.

Follow-ups, deliberately not in scope

  • The activity UI strings are English in all six non-English locales.
  • Retrying a variant-pinned model reinstalls with automatic variant selection: /api/operations carries no variant, though ui_api.go already stores it on the ManagementOp at enqueue time.
  • In distributed mode, a replica that restarted mid-install can record a successful peer install as cancelled. Closing it means carrying the outcome on OpCacheEvent.
  • Enabling eslint react/jsx-uses-vars would remove ~59 false-positive warnings project-wide and let the lint baseline ratchet down.

mudler added 24 commits July 28, 2026 07:41
The operations panel drops an operation the moment it succeeds, so a user
who steps away cannot tell whether an install finished, failed or was never
started. OpCache now keeps the last 50 terminal operations, recorded from
the point where an op leaves the cache.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review of the history ring found four gaps. The dedupe guard and the
bounded seen set were unreachable through the exported API and so had no
coverage; an in-package spec file now drives opHistory directly. The
outcome switch claimed an ordering was load bearing that nothing pinned,
so an errored op that never reached Processed now has a spec.

Two behaviour fixes come with it. StartedAt was the zero time for ops
recovered from the store or replicated from a peer, since neither path
stamps a start time, which would have rendered as a two-millennia
duration; it now falls back to the finish time. Reusing a cache key with
a fresh job ID orphaned the previous stamp, so Set and SetBackend now
drop it.

The comment on the outcome switch described a state the code cannot be
in: CancelOperation sets Cancelled and Processed synchronously before the
handler removes the entry, so status.Cancelled already covers the cancel
endpoint. The !Processed clause stays for the dismiss endpoint firing on
an in-flight op, and the comments now say so.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The NATS end event is the only signal a replica gets for an install another
replica ran. Record from applyEnd too, deduped by job ID so the originating
replica does not record its own broadcast twice.

Three start-stamp defects in the same path go with it. applyEnd now drops the
stamp unconditionally, since recordTerminal only cleans up on the path where it
found a cache key and an end event can overtake the local Set. applyStart drops
the stamp of the job whose cache key it replaces, which a peer-driven retry
previously stranded. And recordTerminal reads the stamp once instead of testing
Exists and then reading, so a concurrent record for the same job can no longer
delete the stamp between the two and let the zero time overwrite the
finish-time fallback, which the Activity page would render as a two-millennia
run.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…tatus

A replica that restarts mid-operation hydrates its OpCache keys from
PostgreSQL, but gallery statuses are in-memory only and come back empty. The
end broadcast then landed on recordTerminal's nil-status branch, which reads a
missing status as queued-and-removed and filed a successful install as
cancelled. That reading is right locally and wrong on the peer path, where a
missing status means the outcome was never held here.

recordTerminal now takes the source of the terminal event and records nothing
when the peer path finds no status, restoring what the replica did before the
end event started recording. The local path is unchanged.

Also move the ApplyEndForTest seam to the conventional export_test.go, and stop
the dedupe spec from claiming to guard the ring's seen set: the local delete
removes the status keys, so the broadcast that follows returns before reaching
it. An in-package spec that calls recordTerminal twice does the pinning.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Admin gated like the rest of the operations API. The live /api/operations
payload is unchanged so the one second poll stays small.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Fetched on demand and when the live list shrinks, never on the one second
poll interval.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
… ops in the ETA gate

Refetching history on a shrinking live count missed a completion that
coincided with a start, which is the common case during a batch install.
Track the live job IDs instead, so any departure triggers the refetch
regardless of how the count moved.

An operation that has finished downloading stays live at
currentBytes == totalBytes for the whole commit and install phase and can
never produce an estimate, so counting it in the all-or-nothing gate blanked
every other operation's time remaining for as long as it lasted. Only
operations still moving bytes get a vote.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…mate

Verifying pins an operation flat below its total for the whole sha256 pass:
the AfterDownload hook reports completedBytes plus the finished file against
a total summed over every file, then hashes synchronously without emitting
progress. Files download sequentially, so a 15 shard model enters that
window 14 times, and a byte comparison cannot see it because the counter is
genuinely below the total throughout.

Gating on phase closes resolving, verifying, committing and persisting in
one predicate, so a quiet neighbour no longer blanks every other
operation's estimate for minutes at a time. The byte clauses stay: a
producer can report downloading with bytes already at the total.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Four concurrent installs used to take four rows above every page. The strip
now shows one operation, failure first, with a counter linking to Activity.
The close button hides the strip and no longer cancels an install.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…ting a failure

A long install error made the strip report a 1600px minimum width, which sized
main-content to fit and gave every page under it a horizontal scrollbar.
Inline-size containment plus shrinkable detail and bytes cells keep it inside
the viewport.

Hiding is no longer able to swallow the hidden job's own failure, a completed
removal or staging says so instead of claiming an install, a cancelling
operation renders as cancelling, and the live region no longer covers the
per-second percentage.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…ose progress

min-width on .main-content is what actually lets a long install error shrink,
and unlike inline-size containment it has no browser support floor and no
latent collapse if the strip ever lands in a shrink-to-fit context. It matches
what .app-layout-chat .main-content already does, and it clears pre-existing
horizontal overflow on narrow viewports as a side effect.

The progress track is now a labelled progressbar, so assistive tech can read
the value on demand rather than losing it to the aria-hidden that stopped the
live region re-announcing every poll.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Carries the detail the one-line strip has to drop: phase, bytes, the per-node
breakdown for cluster installs, and a labelled Cancel button. Cancelling is
destructive, so it gets a labelled button rather than a glyph.

A cancelling operation drops its progress bar and its time estimate, the same
call the strip makes: a percentage still climbing under "Cancelling" reads as
the cancel not having taken.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
… per-node detail

The card carried no verb, so an install, a removal and a staging op rendered as
spinner plus name plus kind tag and were indistinguishable. It now runs the same
verb and icon chain as the one-line strip, which is what stops the page that is
meant to carry more detail from carrying less.

The auto-expand default was evaluated once at mount. An operation is listed as
soon as it is admitted but its nodes are filled in only when the fan-out starts
reporting, so a card mounted at creation latched on the empty list and stayed
collapsed. The default is a live expression now, and state holds only an
explicit choice.

Also: an optional onRetry gates a Retry button, so the page can own the install
reconstruction without the card ever showing a control with nothing behind it;
the disclosure moved above the region it controls and gained aria-controls; the
toggle is gated at more than one node so the count is never "1 nodes"; an
unmapped node status is passed through instead of being relabelled "Queued";
error text is clamped with the full string in the title; and file_name plus the
per-node progress bar are rendered again, reviving three CSS rules that had gone
dead along with the detail they styled.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Live operations, unacknowledged failures and the record of what finished, at
/app/activity in the Operate console. Cancelling an install now lives here
behind a labelled button rather than on the strip, and a failed install can be
retried: the retry dismisses the failure first so it still reaches the record,
then reissues the model, backend or node-scoped backend install.

The sidebar Operate entry carries the operation count. The console rail is only
rendered on an Operate route and can be collapsed, so a badge there could
vanish while operations were still running.

Two follow-ups from review fold in here: a failed removal or staging job no
longer reports a failed install on either the card or the strip, and the card's
error text can shrink so one unbroken token cannot widen the card.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…dicting itself

Dismissing resolved the job by display id, but /api/operations strips the
"node:<nodeID>:" prefix before emitting, so a local install and a node-scoped
install of one backend arrive as two jobs sharing one id. Dismissing by id
retired whichever came first. That defeated the guarantee retry was built
around: with the wrong job dismissed, the reinstall overwrote the acted-on
failure's opcache entry in place, bypassing recordTerminal, while an unrelated
failure vanished from Needs attention. dismissFailedOp, the card's dismiss
control and the strip now all pass the jobID, which is what the endpoint takes.

A filter matching nothing rendered the "nothing has ever run" empty state while
the header counted the records the filter had hidden. The empty state is now
gated on the All chip and a narrowed view gets its own message plus a way back;
the header counts the instance rather than the chip, so selecting Backends no
longer reports "Nothing running" over running model installs.

Also: the summary drops a zero clause instead of rendering "0 needs attention"
on the happy path and pluralises both counts; a record duration is floored at
"< 1s" and rejected above a day, so a zero-value start stamp cannot render a
span of millennia and a zero span cannot render "installed in" with nothing
after it; a deletion cancelled mid-flight reports the cancellation rather than
claiming it was removed; and the retry variant comment names the fix instead of
calling the gap closed.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Adds an Activity page under Operations covering the one-line operations
strip, the /app/activity sections and filters, per-operation cancel,
retry and dismiss, the in-memory 50-entry record, and the sidebar count.
Documents GET and DELETE /api/operations/history, and fills the gap in
the admin-only endpoint list, which also omitted the pre-existing
POST /api/operations/:jobID/dismiss.

Corrects the distributed-mode install-watching section: the per-node
breakdown now lives on the Activity page rather than on the strip, which
rolls a fan-out up into a single phrase.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The operations strip never renders a file name: its detail line is the
error, the node roll-up, the target node, the phase or the queued note.
Drops the stale clause in the distributed-mode section, where the
per-node bullet is now the only place a file name is described.

Scopes the phase vocabulary to artifact-backed gallery models, since a
plain GGUF install emits no phase. Corrects the per-node list: the
toggle exists for any fan-out of two or more workers and the four-node
threshold only governs whether it starts open, while the N nodes tag
needs more than one node. Notes that a cancelled operation can sit in
the live section reading Cancelling, that cluster staging never reaches
the record, and that Clear history appears only when the record has
something in it.

Names the operations response envelope, with a JSON example, so callers
do not index a bare array, and stops describing the icon-only dismiss
control as a labelled button.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
An operation can only report isCancelled while it is unprocessed, but
every writer of Cancelled sets Processed in the same breath, on the peer
path as much as the local one, and the cache evicts cancelled entries
before the handler sees them. The state cannot reach the page, so the
live section is described again as running or queued operations.

Byte counts come from the artifact bridge alone, the same producer as
the phase, so a plain GGUF install, a removal and a backend install
report none. Scopes both to artifact-backed gallery models and leaves
the verb, the name and the percentage as what every operation shows. A
worker backend install reports its bytes through fields the operations
payload does not carry, so the distributed section now describes the
percentage and the node roll-up, with per-file counts pointed at the
per-node detail.

Also: staging jobs carry no error, so they never reach Needs attention
and Retry never had a staging case to exclude; an install that involves
workers is no longer called node-scoped, which this page uses for
node-targeted installs; and the record timestamps carry nanoseconds.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The percentage is as conditional as the bytes were: it renders only for
a running operation that has reported progress, so a queued operation, a
failed one and a removal never carry it. A removal in particular sits at
progress zero for its whole visible life, since the delete path reports
none and its completion is filtered out. Both the strip and the card
paragraphs now lead with what always shows and list the rest as
conditions.

The Cluster chip matches on a node list that finished operations do not
carry, so a fan-out install leaves the chip once it reaches the record.
Scoped that claim to the live sections.

Two more of the same shape, found by re-reading each clause alone: the
strip also appears for a failure, which is not running, and the
four-second hold only applies when nothing replaces the operation that
just finished.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…eued real

Three defects that all trace to one root cause: `isCancelled: true` is
unreachable from /api/operations. Every writer of Cancelled=true also sets
Processed=true, the handler skips Processed && Cancelled, and OpCache.GetStatus
evicts a cancelled op before the handler iterates it.

Cancelling the last running operation put a green "Installed model X" on the
strip for four seconds: the completion hold was guarded by
`!previous.isCancelled`, which is dead. A cancellation deletes the operation
server side, so the strip sees exactly what it sees on a completion, and
nothing in the payload separates the two. The signal now comes from the side
that issued the cancel: the operations context remembers the job IDs it
cancelled (pruned after a minute) and the strip asks before it holds anything.
A cancelled operation goes as soon as it stops; the record already reports it
as cancelled.

isQueued was set only when the gallery status was missing, but markQueued
publishes a "queued" status at admission, so a queued op has a status for its
whole queued life and the state was unreachable outside a microsecond window.
Every operation waiting behind a running install rendered as "Installing model
X" with a spinner. The queued phase is now the signal, via an exported
PhaseQueued and a nil-safe OpStatus.IsQueued() next to the writer.

With those two fixed, the Cancelling state has no way to be entered: cancelling
is instantaneous from the API's point of view. Its branches, CSS, locale key
and the isCancelled field itself are removed rather than left for a future
reader to assume they work.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
OpStatus.Deletion was set once, at admission, and lost on the next status
write: UpdateStatus replaces the whole status and only carried Nodes
forward. Every later writer (the worker's first write, the progress
ticks, the failure path) leaves the field at its zero value, so the flag
survived only the queued window, and both surfaces test isQueued first.

The reachable consequence is that a failed removal reported itself as a
failed install, which is exactly the shape the Activity page offers Retry
for, and Retry installs: pressing it on a removal that failed
re-downloaded the model. A running delete also rendered as "Installing
model X" with a spinner, and a successful one as "Installed model X".

Carry Deletion forward the way Nodes already is. A job is a delete or an
install for its whole life; an unset flag means "no new information", not
"this is an install". Pinned by Go specs on both the service and
/api/operations: the existing Playwright specs were green only because
they stubbed a payload the server could not emit.

Also restore the operation's own status message on the Activity card.
Phases and byte counters exist only on the managed-artifact path, so a
legacy files: gallery model and every backend install rendered a sub-row
with nothing in it but the verb. The strip stays terse on purpose.

And give the strip's name a min-width floor: overflow: hidden zeroes its
automatic minimum, so a long error squeezed the name down to "mod…" and
the identity of the thing that broke was the first thing lost.

primaryOperation is made module-private: its comment claimed the Activity
page selected the same operation, but that page shows all of them,
partitioned into failed and running, and never imported it.

Assisted-by: Claude Code:Opus 5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The Activity page's record of finished installs and removals was a 50-entry
in-memory ring per frontend replica. In distributed mode that is the wrong
place for it: each replica keeps its own copy, a replica added by a scale-out
or a rolling deploy starts empty and never backfills, and "Clear history"
clears only the replica that served the request, so the record reappears on
the next poll routed elsewhere.

The data is already in gallery_operations. Read it from there.

GalleryStore gains ListTerminal and ClearTerminal, sharing a lifted
terminalStatuses set with CleanOld so there is one definition of "finished".
ListTerminal orders by updated_at, when the operation reached its terminal
status, because the record reports what finished and when.

OpCache.History and ClearHistory dispatch on whether a store is wired, so the
HTTP handlers and the OpRecord JSON shape are unchanged and the page needed no
change. A failed store read falls back to the local ring rather than blanking
the page, and ClearHistory empties the ring as well so a database blip cannot
resurrect a record the admin just cleared.

The name derivation in recordTerminal is lifted into operationDisplayName and
used by both paths, so the ring and the store cannot name the same operation
differently.

Also fixes a pre-existing bug the store path made visible: the backend channel
hardcoded op_type "backend_install" even for a removal, while the model channel
derives model_install/model_delete from op.Delete. Both channels carry the same
ManagementOp, whose Delete field the backend handler already branches on, so
the backend channel now derives backend_delete the same way.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…led clear

Review follow-up on the store-backed Activity record.

A cancelled install was recorded as a failure. The cancel handler persists
"cancelled" synchronously, then the handler goroutine unwinds with the context
error and Start hands that to updateError unconditionally, which overwrote the
row with "failed: context canceled". The page rendered a cancelled install as a
red failure card offering Retry, with a raw context error as the reason.

Fixed in GalleryStore rather than in Start, because an operation finishes once
and the paths that retire one are not mutually exclusive: UpdateStatus now
refuses to rewrite a row that already reached a terminal status. That also pins
updated_at to when the operation really finished, which is the key the record is
ordered by, and Create's upsert now freezes the same columns so a worker
dequeuing an operation the admin cancelled while it was queued cannot reopen it
as pending.

ClearHistory returned nothing, so a failed delete logged a warning while the
handler still answered 200. The admin watched the record clear and come back on
the next fetch with nothing said about why. It now returns the error, the DELETE
handler answers 500, and the store is cleared before the local ring so a failure
leaves the fallback record intact rather than faking an empty one.

Hydrate is the only reader that decides from op_type whether an operation is a
removal, and it tested for "model_delete" exactly, so the backend_delete added
in the previous commit hydrated as an install: a replica restarting during a
backend removal rendered "Installing backend X". Both discriminations now go
through IsDeleteOpType/IsBackendOpType so a fifth op_type cannot silently read
as an install in whichever consumer was missed.

Also: the backend channel now persists Cancellable as !op.Delete, matching the
model channel; IsBackend falls back to the op_type prefix, since is_backend_op
is only written by UpsertCacheKey and the rows needing the name fallback were
reporting backend operations as models; and an unrecognized terminal status is
logged rather than quietly filed as a success, which is what the comment already
claimed.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The terminal-status freeze added in the previous commit was too wide. It froze
"failed" alongside "completed" and "cancelled", and the stale reaper writes
"failed" onto operations that are still going to run.

The gallery worker is a single goroutine consuming both channels serially, so
an operation queued behind a large download sits in "pending" with nothing
bumping updated_at, and ReapStaleOperations gives up on it after 30 minutes.
That used to be self-healing: the worker dequeued it, Create reset the row to
"pending", and the operation reported its real outcome. With the freeze the row
stayed "failed" forever while the install ran and succeeded underneath it: a red
failure card offering Retry for a model that is installed, omitted from
ListActive so no replica hydrates it, and no longer deduped cluster-wide by
FindDuplicate.

Freeze on ("completed", "cancelled") instead. That is all the cancelled-install
fix ever needed, and it leaves a failure correctable by what actually happened.
The set is separate from terminalStatuses, which ListTerminal, ClearTerminal and
CleanOld all still want in full, because the two mean different things: a
failure can be superseded by a real outcome, a completion or a cancellation is
the real outcome.

UpdateStatus now writes the error column unconditionally, so a corrected
outcome drops the previous attempt's reason rather than being recorded as
completed while still carrying "stale operation reaped" as its error.

Also adds the route-level spec for the 500 branch of DELETE
/api/operations/history, and trims a comment that credited the persisted
cancellable column with more than it survives long enough to do.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@localai-bot

Copy link
Copy Markdown
Collaborator Author

Update: the record is now read from PostgreSQL, not a per-replica ring

The original design kept finished operations in a bounded in-memory ring per frontend replica. That is wrong for distributed mode: each replica keeps its own copy, a replica added by scale-out or a rolling deploy starts empty and never backfills, and Clear history cleared only the replica that served the request, so the record reappeared on the next poll routed elsewhere.

The data was already in gallery_operations. GalleryService.UpdateStatus has always written the terminal status there, and the table already carries the job ID, cache key, backend flag, op type, status, error, and both timestamps. So the ring was a weaker second copy of data already being persisted.

Three commits:

  • c22802d96 adds GalleryStore.ListTerminal / ClearTerminal, dispatches inside OpCache.History() / ClearHistory() so the HTTP handlers and the whole frontend are unchanged, and lifts the name derivation into a helper shared by both paths. The ring stays as the standalone path, since standalone has no PostgreSQL. Also fixes a pre-existing bug where the backend channel recorded a removal as backend_install.
  • 63ebcc5ba fixes a Critical bug found in review: a cancelled install was recorded as a failure. CancelOperation writes cancelled, then the handler goroutine unwinds and Start calls updateError unconditionally, writing failed with context canceled over it. The page showed a cancelled install as a red failure card with a Retry button. Fixed by refusing to overwrite a settled row, plus ClearHistory returning its error so a failed clear returns 500 instead of silently un-clearing, and Hydrate recognising backend_delete.
  • cc81176c1 fixes a regression the previous fix introduced: the freeze set included failed, so a queued operation reaped as stale after 30 minutes could never report its real outcome and the row stayed failed forever while the install actually succeeded. The freeze is now completed and cancelled only, so a real outcome can still correct a reaper-written failure.

Docs updated: the record is per-replica and volatile only in standalone; in distributed mode it is shared across replicas, survives restarts, and Clear history is cluster-wide.

Tests use testcontainers PostgreSQL rather than fakes. go test green across galleryop, http/routes and distributed; make lint 0 issues.

Open question for a maintainer

Should a removal offer a Cancel button at all? Both channels write Cancellable: true unconditionally at handler entry, ahead of the op.Delete branch, and UpdateProgress persists it on every tick, so the Cancellable: !op.Delete value set at admission survives only microseconds. The "A delete is not cancellable; an install is" comment on the model channel is aspirational today. Leaving that as a product decision rather than changing behaviour here.

Follow-ups, deliberately not in scope

  • UpdateProgress has no settled-row guard, so a coalesced progress flush landing after a cancel can still bump updated_at.
  • CleanOld has no callers anywhere, so gallery_operations grows unbounded; ListTerminal would also benefit from a (status, updated_at) index.
  • On a store error the endpoint falls back to the ring, which on a fresh replica is empty, so the page says "nothing has finished" rather than "the record is unavailable". A degraded flag on the response wrapper would let the page caption it.

The cancellable flag was set at both ends of an operation's life and was wrong
at both, in opposite directions.

A queued operation is cancellable whatever it is. EnqueueModelOp and
EnqueueBackendOp select on the operation context, so cancelling one that is
still waiting releases the delivery goroutine and abandonQueued retires it: the
worker never sees it, nothing is downloaded, nothing is deleted. markQueued
nevertheless wrote Cancellable: !deletion, so a queued removal reported
cancellable: false and the UI hid the Cancel button in the one window where
pressing it both works and leaves no trace. A removal queued behind a large
install was stuck there until the install finished.

A running removal is not cancellable at all. DeleteModel and DeleteBackend take
no context, and modelHandler only checks the operation context after the call
returns, so a "cancelled" verdict would land after the model was already gone.
Both handlers nevertheless wrote Cancellable: true unconditionally at entry,
ahead of the op.Delete branch, offering a Cancel button the server cannot
honour.

So the queued phase is more cancellable than the running phase, which is the
reverse of the usual shape. markQueued now reports true unconditionally, and
the handler-entry writes report !op.Delete. Both sites carry a comment saying
why, because reading either one alone suggests the other is a bug.

GalleryStore.Create keeps !op.Delete: it runs at dequeue, so its value already
describes the running phase. Its comment now says so.

Specs cover queued removal, queued install, running removal and running install
through the handlers, plus the queued-removal case through /api/operations
where the flag is consumed, plus the behaviour the whole asymmetry rests on: a
removal cancelled while queued never reaches the worker and deletes nothing.
No existing spec asserted the old values.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Write] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@localai-bot

Copy link
Copy Markdown
Collaborator Author

The Cancel flag was inverted at both ends

Answering my own open question from the previous comment: yes, a queued removal is cancellable, and the code had this backwards in both directions.

Queued: cancellable in reality, reported as not. EnqueueModelOp / EnqueueBackendOp select on the operation context, so cancelling a still-queued operation releases the delivery goroutine and abandonQueued retires it. The worker never sees it, nothing is touched, no partial state. That is true for a removal exactly as it is for an install. But markQueued wrote Cancellable: !deletion, so the button was hidden in the one window where it would have worked.

Running: not cancellable in reality, reported as cancellable. DeleteModel(name string) and DeleteBackend(name string) take no context, so a running removal cannot be interrupted; the cancellation check only runs after the delete has already returned. But both handlers wrote Cancellable: true unconditionally at entry, ahead of the op.Delete branch, so the button appeared where the server could not honour it.

Fixed in f417a7536:

  • markQueued: Cancellable: true unconditionally.
  • Handler entry in models.go and backends.go: Cancellable: !op.Delete.

So starting an operation now narrows what can be cancelled, which is the reverse of the usual shape and is commented at both sites for that reason.

GalleryStore.Create's Cancellable: !op.Delete was already correct and is unchanged: it runs in the worker loop at dequeue, after the delivery goroutine has handed the operation over, so it is a running-phase writer rather than a boundary case. Its comment now says so.

Five new specs, verified red first, including one that drives a real worker with a parked model manager so the running-removal flag comes from the handler rather than the fixture, and one asserting through /api/operations where the flag is actually consumed. No existing spec had encoded the old values.

Docs corrected: Cancel is offered while an operation is queued whatever it is, and while an install is running, but not once a removal has started.

… spec

Running the page against a real local-ai showed the legacy installer message
wrapping to three lines and dominating the card: it embeds an absolute file
path, so it is both long and a single unbreakable token. One line, ellipsised,
full text in the title, matching what the error string already does.

The spec that found it runs with no route stubbing at all. Every other spec
here stubs /api/operations, which is how a payload the server cannot emit
(isDeletion true on a live operation) stayed green through a full review while
the UI rendered a removal as an install. It is skipped unless
LOCALAI_REAL_BINARY is set, so CI is unaffected.

Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@localai-bot

Copy link
Copy Markdown
Collaborator Author

Verified against a real local-ai binary, no stubbing

Built the binary from this branch and drove the real UI against it in Chromium. Every other spec here stubs /api/operations, which is exactly how a payload the server cannot emit (isDeletion: true on a live operation) stayed green through a full review while the UI rendered a removal as an install. These runs used the real handler.

What the real binary confirmed:

  • A real deletion records as a removal. POST /api/models/delete/e2e-victim produced taskType: "deletion", outcome: "completed", real timestamps 11 ms apart. The UI row reads "removed", not "installed".
  • A queued removal offers Cancel, and cancelling it works. With a real gallery download occupying the single worker goroutine, the queued removal reported isQueued: true, isDeletion: true, cancellable: true, taskType: deletion. Clicking Cancel removed it from the live list and recorded it as cancelled. That is the case that was inverted before f417a7536.
  • A real failed install carries the real resolver error (no model found with name ...), lands in Needs attention, offers Retry, and dismissing it moves it into the record rather than destroying it.
  • The strip stays one line with two live operations, showing the 1 more counter, and the page does not scroll sideways with a real error on screen.
  • Clear history empties the record against the real store.

One defect this found that stubs could not

The legacy installer message embeds an absolute file path, so with a real download the card's detail line wrapped to three lines and became the largest thing on the card. Fixed in 47a3e4aef: one line, ellipsised, full text in the title, matching what the error string already does.

The spec is committed as e2e/real-binary-activity.spec.js, self-contained (it triggers its own failing install, no fixtures) and skipped unless LOCALAI_REAL_BINARY=1 is set, so CI is unaffected. Recipe is in the file header.

Full suite still 327 passing, lint at the 599 baseline.

@mudler
mudler merged commit 0f7186f into master Jul 28, 2026
69 checks passed
@mudler
mudler deleted the feat/activity-operations-ui branch July 28, 2026 17:33
@localai-bot localai-bot added the enhancement New feature or request label Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants