Bound FITS-header fetch so a dead peer can't stall the frame - #765
Conversation
|
Found and fixed two bugs while reviewing this (commit caa5621):
Added regression tests for both, plus a |
thusser
left a comment
There was a problem hiding this comment.
Verified on head caa5621: pytest (fitsheader + basecamera/basespectrograph/basevideo) 87 passed, ruff/black/pyrefly clean.
Fix is correct and a real improvement (concurrent await under one deadline vs the old sequential awaits). Two non-blocking notes:
-
future.result()only guardsRemoteError. A peer that errors with something else (or a task cancelled externally) still propagates. Same as the oldawait, so no regression, but worth a follow-up if peers ever fail with non-RemoteError. -
1e2b81d5(test-baseline plan 'implemented') rides along here and touches specs/. Drop it if you want a clean single-purpose PR; otherwise harmless doc churn.
LGTM.
add_requested_fits_headers() awaited every header future with no timeout, so a peer that never answers its IQ (e.g. a laptop put to sleep without closing its client) stalled frame finalization for the full ~120s XMPP IQ timeout. Bound the collection to a single asyncio.wait() deadline and skip anything still pending. New fits_header_timeout kwarg (default 15s) on FitsHeaderMixin. Fixes #764.
…g BaseCamera/BaseVideo asyncio.wait() raised ValueError on an empty futures dict (any module with no comm, or no peer implementing IFitsHeaderBefore/After), crashing every exposure. BaseCamera and BaseVideo also passed an explicit keyword list to ImageFitsHeaderMixin.__init__ instead of forwarding **kwargs, so a configured fits_header_timeout was silently dropped for exactly the module (fli230, BaseCamera) that motivated this PR. Adds regression tests for both, plus a fits_header_timeout-reaches-mixin guard for BaseSpectrograph (already correct via **kwargs, previously untested).
caa5621 to
d6aa40e
Compare
* Bound FITS-header fetch so a dead peer can't stall the frame add_requested_fits_headers() awaited every header future with no timeout, so a peer that never answers its IQ (e.g. a laptop put to sleep without closing its client) stalled frame finalization for the full ~120s XMPP IQ timeout. Bound the collection to a single asyncio.wait() deadline and skip anything still pending. New fits_header_timeout kwarg (default 15s) on FitsHeaderMixin. Fixes #764. * Fix crash on empty header futures and fits_header_timeout not reaching BaseCamera/BaseVideo asyncio.wait() raised ValueError on an empty futures dict (any module with no comm, or no peer implementing IFitsHeaderBefore/After), crashing every exposure. BaseCamera and BaseVideo also passed an explicit keyword list to ImageFitsHeaderMixin.__init__ instead of forwarding **kwargs, so a configured fits_header_timeout was silently dropped for exactly the module (fli230, BaseCamera) that motivated this PR. Adds regression tests for both, plus a fits_header_timeout-reaches-mixin guard for BaseSpectrograph (already correct via **kwargs, previously untested).
Summary
FitsHeaderMixin.add_requested_fits_headers()awaited every requested header future with no timeout. The futures are created before the exposure and awaited after readout, so a peer that never answers its IQ stalls frame finalization for the full XMPP IQ timeout (~120s — slixmpp'sIq.senddefault).Trigger (issue #764):
adminwas a pyobs-gui on a laptop sent to sleep without closing the client. Its session lingered on ejabberd, andfli230spent ~120s per exposure fetching headers from it before giving up.Change
asyncio.wait(..., timeout=...)deadline; cancel + gather anything still pending, then skip it with atimed outwarning (mirrors the existingRemoteErrorpath).fits_header_timeout: float = 15.0kwarg onFitsHeaderMixin, stored as_fitsheadermixin_header_timeout.A
@timeoutdecorator was considered and rejected: it works by having the server reply with amethod_timeoutIQ before running, and a dead peer never replies at all.Test plan
ruff check/black --checkclean;pyrefly checkclean onfitsheader.py.pytest tests/mixins/test_fitsheader.py: 43 passed, including two new tests (dead peer times out and is cancelled; live+dead mix adds live headers and skips dead).See
specs/plans/2026-08-16-fits-header-fetch-timeout.mdfor the plan.Note: this branch also contains commit
1e2b81d5("Mark core-tier test baseline plan implemented"), which the graphify automation committed to the branch on checkout — unrelated to this change.