fix(upload): stop spamming when flight-review is offline - #30
Closed
dakejahl wants to merge 1 commit into
Closed
Conversation
When the local flight-review server is down, the upload loop previously probed and logged a failure for every pending log. Probe once, log a single unreachable message with a 60s cooldown, and bail the current upload batch (HTTP 503) instead of walking the queue. (cherry picked from commit 7a23c8e)
This was referenced Jul 28, 2026
Collaborator
Author
|
Superseded by #32, which contains this branch's commits unchanged and with authorship intact — merging #32 merges this. Reviewing this stack turned up structural problems in the code underneath it, and fixing those meant rewriting the parts these PRs sit on. Keeping them open would have meant reviewing #29's Nothing here is dropped: the FTP transport, the reachability cooldown and the API-key headers are all in #32. The one deliberate change is that 401/403 no longer blacklist a log — see the note in #32. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second of three stacked PRs replacing #28. Based on #29 — review that one first, and this diff will shrink to a single commit once #29 merges.
Carries over
7a23c8efrom #28 unchanged, rebased onto the FTP work. Authorship preserved.Problem
When the local flight-review server is down, the upload loop probes it and logs a failure for every pending log. With a queue of logs and
:5006not running, one pass produces a wall of identicalConnection to ... failedlines and aTEMPORARILY FAILEDper log.Worse, the loop is
while (num_logs_to_upload())and a failed upload does not reduce that count, so a dead server means an unbounded hot loop hammering it as fast as the connection attempts return.Solution
Probe at most once per 60s cooldown, log only on the down/up transitions, and give the reachability failure an explicit 503 so the caller can bail the batch instead of walking the queue. Connect/read timeouts on the probe keep a black-holed server from stalling the thread.
Measured on hardware in #28: 1 unreachable line instead of a per-log wall, 0
TEMPORARILY FAILED.