fix(celery): survive a down balena supervisor in reboot/shutdown

[vpetersson]

Issues Fixed

• Fixes reboot_anthias never survives a down balena supervisor: RetryError after 5×1s (Sentry ANTHIAS-3F) #3123 (Sentry ANTHIAS-3F)

Description

reboot_anthias/shutdown_anthias retried the balena supervisor API 5 times, 1 second apart, with no HTTP timeout and no status check. The supervisor is routinely down exactly when these fire — it restarts itself during an OTA apply — so the tasks died in an unhandled RetryError (60 Sentry events in a day from one pi4-64) and the power command was silently dropped.

Now:

• Retries with exponential backoff (capped at 15s) for up to 2 minutes, which covers a supervisor restart.
• Each HTTP attempt is bounded at 10s so a hung supervisor socket can't pin the Celery worker; the task gets soft/hard time limits as the SIGKILL backstop (same pattern as the periodic pokes from fix(celery): soft-limit the display-power and telemetry pokes #3063).
• raise_for_status() so a 5xx from a half-started supervisor retries instead of silently counting as "command accepted".
• Terminal failure logs a warning with an actionable message instead of an unhandled traceback — a down supervisor is an expected transient, and per the module's Sentry-noise policy it shouldn't page as an error.

Validation on real hardware

This path only runs under balenaOS (it talks to the balena supervisor API), and our testbeds are plain docker-compose devices — so there is no supervisor to exercise it against on the testbed. The behaviour is covered by unit tests (retry-until-recovery, non-transport-error propagation, terminal-failure-warns-not-raises for both reboot and shutdown). The non-balena branch (r.publish('hostcmd', ...)) is unchanged. Real-device confirmation would need a balena fleet device.

Checklist

• I have performed a self-review of my own code.
• New and existing unit tests pass locally and on CI with my changes.
• I have done an end-to-end test for Raspberry Pi devices.
• I have tested my changes for x86 devices.
• I added a documentation for the changes I have made (when necessary).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR hardens the Balena supervisor-backed reboot_anthias / shutdown_anthias Celery tasks so they tolerate the supervisor being temporarily down (e.g., during OTA apply), bounding each HTTP attempt with a timeout and avoiding unhandled RetryError noise.

Changes:

• Replace fixed-count retries with a 2-minute retry window using exponential backoff, per-attempt HTTP timeouts, and task soft/hard time limits.
• Add a shared _run_supervisor_command() helper that retries raise_for_status() failures and logs a terminal warning instead of erroring.
• Update Balena supervisor power-command helpers to accept and forward a timeout=..., and add unit tests for retry and terminal-failure behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
src/anthias_server/celery_tasks.py	Adds retry/time-limit constants, a shared supervisor-command retry helper, and applies it to reboot/shutdown tasks.
src/anthias_common/utils.py	Extends supervisor reboot/shutdown helpers to accept a forwarded request timeout.
tests/test_celery_tasks.py	Adds tests ensuring reboot/shutdown retry behavior and terminal failure warning behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.