Summary
On a DGX Spark (GB10 / sm_121a), an illegal memory access raised in the middle of a long prefill poisons the CUDA context. Every subsequent request then fails, but the process stays alive, so Restart=on-failure never fires and the server keeps answering errors indefinitely. In our case it served nothing but errors for 22 hours before a manual restart cleared it.
The failure produces no finish=error line, so nothing in the journal reads as a service failure — the only visible symptom is cuda resumed prefill failed while extending checkpoint reaching the client.
Environment
What happened
A 120,324-token prefill was progressing normally at ~400 t/s and died at 35.5%:
21:01:30 chat ctx=6475..126799:120324 TOOLS prefill chunk 38581/120324 (32.1%) chunk=370.57 t/s avg=403.54 t/s 95.606s
21:01:40 chat ctx=6475..126799:120324 TOOLS prefill chunk 42677/120324 (35.5%) chunk=400.59 t/s avg=403.26 t/s 105.831s
21:01:50 ds4: CUDA end commands failed: an illegal memory access was encountered
21:01:50 ds4: CUDA synchronize failed: an illegal memory access was encountered
21:01:50 ds4: Metal synchronize after layer-major prefill failure also failed
21:01:50 ds4: CUDA synchronize failed: an illegal memory access was encountered
21:01:50 ds4: Metal synchronize after chunked prefill failure also failed
21:01:50 kv cache discarded reason=prefill-failed file=<kv-dir>/<hash>.kv
From that point on, every new session fails immediately at the first kernel launch:
21:01:50 chat ctx=0..64657:64657 TOOLS prompt start
21:01:50 ds4: CUDA tensor fill f32 launch failed: an illegal memory access was encountered
21:01:50 chat ctx=0..298:298 prompt start
21:01:50 ds4: CUDA tensor fill f32 launch failed: an illegal memory access was encountered
17 log lines in that one second, then silence. The next 22 hours of requests all failed with cuda resumed prefill failed while extending checkpoint (emitted from ds4_session_sync(), the "extend existing checkpoint" branch, when metal_graph_prefill_chunked_range() returns false). A restart cleared it and it has not recurred since.
Note there were concurrent sessions in flight when it happened (a 120K prefill plus at least three other chats), which may matter more than the prompt size alone.
Two separate issues, I think
-
The illegal memory access itself. I cannot reproduce on demand yet — it happened once in several days of use. Happy to run instrumented builds or compute-sanitizer if that helps.
-
The recovery behaviour, which is the more damaging half. A sticky CUDA error is unrecoverable within the process: once the context is poisoned, every kernel launch fails forever. Continuing to accept requests and answer errors is strictly worse than exiting, because no supervisor can tell the difference between "running" and "running but permanently broken" — systemctl is-active said active the whole time.
Would you consider treating a sticky CUDA error as fatal (log and exit(1)), so Restart=on-failure or any process supervisor can do its job? Alternatively a health signal that flips to unhealthy would be enough to build on.
I worked around it with an external watchdog that probes /v1/models rather than the unit state, but that only exists because the process lies about being healthy.
Summary
On a DGX Spark (GB10 /
sm_121a), anillegal memory accessraised in the middle of a long prefill poisons the CUDA context. Every subsequent request then fails, but the process stays alive, soRestart=on-failurenever fires and the server keeps answering errors indefinitely. In our case it served nothing but errors for 22 hours before a manual restart cleared it.The failure produces no
finish=errorline, so nothing in the journal reads as a service failure — the only visible symptom iscuda resumed prefill failed while extending checkpointreaching the client.Environment
b030961, built withmake cuda-spark(-gencode arch=compute_121a,code=sm_121a)DeepSeek-V4-FlashIQ2_XXS/Q2_K imatrix (~81 GB resident)ds4-server --cuda --ctx 131072 --kv-disk-dir <dir> --kv-disk-space-mb 8192--ssd-streaming(so this is not CUDA --ssd-streaming: prefill dies with illegal memory access at exactly 128 prompt tokens #734), not using--mtp(not cuda-generic: --mtp startup fails with cascading "illegal memory access" (RTX 4090 / WSL2) #522)What happened
A 120,324-token prefill was progressing normally at ~400 t/s and died at 35.5%:
From that point on, every new session fails immediately at the first kernel launch:
17 log lines in that one second, then silence. The next 22 hours of requests all failed with
cuda resumed prefill failed while extending checkpoint(emitted fromds4_session_sync(), the "extend existing checkpoint" branch, whenmetal_graph_prefill_chunked_range()returns false). A restart cleared it and it has not recurred since.Note there were concurrent sessions in flight when it happened (a 120K prefill plus at least three other chats), which may matter more than the prompt size alone.
Two separate issues, I think
The illegal memory access itself. I cannot reproduce on demand yet — it happened once in several days of use. Happy to run instrumented builds or
compute-sanitizerif that helps.The recovery behaviour, which is the more damaging half. A sticky CUDA error is unrecoverable within the process: once the context is poisoned, every kernel launch fails forever. Continuing to accept requests and answer errors is strictly worse than exiting, because no supervisor can tell the difference between "running" and "running but permanently broken" —
systemctl is-activesaidactivethe whole time.Would you consider treating a sticky CUDA error as fatal (log and
exit(1)), soRestart=on-failureor any process supervisor can do its job? Alternatively a health signal that flips to unhealthy would be enough to build on.I worked around it with an external watchdog that probes
/v1/modelsrather than the unit state, but that only exists because the process lies about being healthy.