fix: resolve result_future on inference error instead of tearing down the model - #157
Conversation
|
@AmirF194 Thanks for the PR! RE:
Does this mean the proposed fix has not been tested on hardware? |
|
Correct, not on real Intel GPU/OpenCL hardware, this session doesn't have one. What's verified: the hung future (the caller never gets a response) and that all 8 worker functions share the same break-before-resolve pattern, both confirmed by running the workers directly. The SIGABRT itself I can't reproduce without a genuine device error on real hardware, so the fix removes the crash-triggering call ( |
…odel _commit_completed_packet (added in the upstream stream-error-surfacing rework) still fires registry.register_unload() on every inference error and exits the worker loop. That unload runs the pipeline destructor, which can touch a corrupted device context after a real GPU error and SIGABRT the whole process, taking every other loaded model down with it (the second half of SearchSavior#153, the first half was fixed by the stream-error rework already on main). Resolve the caller's future and keep the worker/model alive instead. Fixes SearchSavior#153
a4b5416 to
54c861c
Compare
|
Rebased onto main. #158 landed a stream-error-surfacing rework (204f543) that already fixes the hang half of this issue and unifies all 8 workers through |
Root cause
_commit_completed_packetfiresregistry.register_unload(model_name)on everyinference error before returning True, which breaks every
queue_worker_*loopout of its
while True. Unload runs the pipeline destructor, which can touch acorrupted device context after a real GPU error and SIGABRT the whole process,
taking every other loaded model down with it.
This is the second half of #153. The first half (the HTTP caller hanging
forever because
result_futurewas never resolved) is already fixed on mainby 204f543 (PR #158's
_mark_inference_error/_commit_completed_packetrework), which also unified all 8 workers through one function. That refactor
kept the unload-on-error call, so the SIGABRT risk this issue reports is still
live on current main.
Fix
_commit_completed_packetnow resolves the caller's future with the error andreturns
Falseunconditionally, so the worker stays in its loop and the modelstays loaded, instead of unloading and exiting.
Verification
test_commit_resolves_error_without_unloading_the_model(renamed fromtest_commit_treats_error_field_not_error_prefix, same file) now asserts_commit_completed_packetreturnsFalseandregistry.unloaded == []onan error packet. Confirmed fails on unmodified main (
AssertionError: assert True is False, current behavior unloads) and passes on thisbranch, in a clean
python:3.12-slimcontainer.uv run pytest -W ignore tests/unit: 125 passed, no regressions.Intel GPU/OpenCL context available in this session. This removes the
register_unloadcall on the error path entirely, but I can't reproducethe abort to confirm it's gone, only that the destructor is no longer
called on that path.
Fixes #153