Keep HttpEventTrigger asset watchers polling after a failed request

[rjgoyln]

Summary

HttpEventTrigger wrapped its whole poll loop in one except Exception that logged str(e) and returned, so any error — a 503, a connection reset, a broken response_check_path — ended the generator. The watcher stopped firing even when the next poll would have succeeded, poll_interval was skipped because the sleep only sat on the success path, and the cause survived as a bare message with no type or traceback, leaving the triggerer nothing to record either.

Retrying forever is the opposite failure mode, so the retry gives up after max_consecutive_failures consecutive failures and raises. The triggerer records the traceback and recreates the watcher, while anything deferred on the trigger fails instead of hanging. The default of 10 is chosen against the default poll_interval of 60 seconds — roughly ten minutes of continuous failure before escalating, long enough to ride out an outage and short enough that a misconfigured watcher surfaces the same day. Effective tolerance is that product rather than the count, which the parameter documentation now states.

The except stays broad deliberately: response_check is arbitrary user code, so a KeyError on an unexpected payload is as likely to be transient as a 503, and the cap bounds the genuinely permanent case. Cancellation is unaffected — CancelledError and GeneratorExit are BaseException, and the backoff sleep sits outside the try.

Before / after

Both versions of run() driven through the same scenarios with asyncio.sleep stubbed:

Scenario	Before	After
Transient 503, next poll would succeed	never fires (1 poll)	fires (2 polls, 1 backoff)
Endpoint down indefinitely	exits after 1 poll with no backoff, and the triggerer respawns the watcher ~1s later	raises after 10 polls, one poll_interval apart
response_check raises	exits after 1 poll	retries and raises after 10 consecutive failures
Cancelled during backoff	never reaches a backoff	CancelledError propagates

Tests

test_trigger_on_post_with_data relied on the swallowed exception to end the generator, so it now drives one successful poll instead.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines