Skip to content

Show signal-killed jobs as failed instead of running forever#1761

Merged
BryonLewis merged 1 commit into
mainfrom
dev/crashed-job-status
Jul 16, 2026
Merged

Show signal-killed jobs as failed instead of running forever#1761
BryonLewis merged 1 commit into
mainfrom
dev/crashed-job-status

Conversation

@mattdawkins

Copy link
Copy Markdown
Member

Problem

A pipeline (or any desktop job) whose process dies from a signal — e.g. SIGSEGV from a native crash, or SIGKILL from the OOM killer — is reported by Node with a null exit code. The desktop job UI treats exitCode === null as still running:

  • the Jobs History row shows an indeterminate progress bar and a spinning icon (JobsHistory.vue),
  • the running badge stays lit,
  • the failure dialog is suppressed.

So a crashed job appears to run forever and the user is never told it died. This was hit in practice by a SAM2 segmentation pipeline that segfaults mid-run — the process exited (Process exited with code null) but the job spun indefinitely.

Fix

Normalize a finished job (endTime set) whose exitCode is still null to a conventional "killed by signal" code (139) at the single updateHistory funnel. Every downstream exitCode === null check then renders it as a failed job, and the failure dialog explains it was terminated by a signal (crash / OOM).

Also key runningJobs off endTime rather than exitCode, so "running" is defined unambiguously as "has not reported an end time."

Test plan

  • Run a pipeline whose process crashes (segfault / OOM). The job now shows as Failed (red ✗, stopped progress bar) with a dialog stating it was terminated by a signal, instead of an endless spinner.
  • Normal success (exit 0) and user cancellation (SIGTERM → 143) are unaffected.

🤖 Generated with Claude Code

A pipeline (or any job) process that dies from a signal -- e.g. SIGSEGV
from a native crash, or SIGKILL from the OOM killer -- is reported by
Node with a null exit code. The desktop job UI treats exitCode === null
as "still running": the Jobs History row shows an indeterminate progress
bar and a spinning icon, the running badge stays lit, and the failure
dialog is suppressed. The result is that a crashed job appears to run
forever and the user is never told it died.

Normalize a finished job (endTime set) whose exitCode is still null to a
conventional "killed by signal" code (139) at the single updateHistory
funnel. Every downstream exitCode === null check then renders it as a
failed job, and the failure dialog explains it was terminated by a
signal (crash / OOM). Also key runningJobs off endTime rather than
exitCode so the definition of "running" is unambiguous.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@BryonLewis
BryonLewis merged commit 8a87a18 into main Jul 16, 2026
3 checks passed
@BryonLewis
BryonLewis deleted the dev/crashed-job-status branch July 16, 2026 12:38
mattdawkins added a commit that referenced this pull request Jul 16, 2026
Brings in 8 commits, including suppression regions (#1760), the
multicam camera order fix (#1759), signal-killed pipeline handling
(#1752, #1761), the annotation rollback tombstone fix (#1751), CSV note
columns (#1753), and the vitest/ws dependabot bumps.

Viewer.vue merged cleanly: f557b21 restored the SealTK wiring that
merge 8b0ab93 had dropped, so the file no longer conflicts with main.

Two conflicts, both from suppression regions landing on main while
viame/master already carried its own copy:

* LayerManager.vue -- keep viame/master's ../utils import. main's copy
  of this file diverged and no longer needs those symbols, but the
  merged body still calls geojsonToBound, isRotationValue,
  ROTATION_ATTRIBUTE_NAME and featureHasSegmentationPolygon.
* TypeSettingsPanel.vue -- take main's suppression row (adds mt-5).

package.json resolves to main's vitest 4.1.10 bump (viame/master never
changed it off the 3.2.4 base) while keeping onnxruntime-web.

Verified: client tests 611/611 passing on Node 22, eslint clean, desktop
app builds and runs with no console errors, and on a multicam stereo
dataset ImportAnnotations still completes setup with no Annotation Set
combobox and a live openUpload.

Note: vitest 4 requires Node ^20 || ^22 || >=24, so `npm test` now fails
to start on Node 18. client/README.md still says "Requires Node 18+".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants