feat(camera): make an SDES first-media stall report why it stalled - #224
Merged
Conversation
A session that delivers no media is, in a log, indistinguishable from one that delivered late: the first-media wait expires, the serve launches with unknown payload types, and nothing states the reason. The reason is knowable at that moment and was thrown away. Media only ever follows the AVIO LIVING trigger, and that trigger is armed by exactly one thing: an inbound STUN Binding Success Response from the camera. That response only comes back if something we nominated was reachable. When the camera's answer carries a single candidate on a subnet this host cannot route to, the only usable addresses are the ones its own probes arrive from - and a relay-carried probe is dropped by one of two vetoes that are both completely silent: _is_self_peer_ip refusing the XOR-PEER-ADDRESS so _br_cam_peer is None, or the _bsrc fallback refusing because the packet's source is the TURN server itself. Which one fires decides what a fix would have to change, and neither logs anything. One WARNING at the wait's expiry now carries the nominated candidates, whether USE-CANDIDATE went out, how many inbound Binding Success responses arrived, whether the trigger was sent, and the verdict on every inbound probe source with the two vetoes named apart. Addresses and counts only - this line reaches home-assistant.log. Instrumentation only. Nothing about what is nominated, permissioned, learned or sent changes, and an open that delivers media reaches none of it. The per-probe self-loop drop stays at DEBUG: it fires per packet and would drown the one line that matters. The inbound Binding Success counter is deliberately incremented before the trigger's own guard. Counting inside it would make it an alias for _tutk_trigger_sent and collapse two different diagnoses - "none ever arrived" and "one arrived and the trigger still did not go" - into a single indistinguishable state. A source-level test pins that ordering. Note that _br_stun_resp_count counts Binding Success responses this side SENDS, on the direct path only, and was until now dead code. It is published for completeness but it cannot answer "did a Binding Success arrive", which is why the new counter exists. The message building and the per-probe classification are pure module-level helpers so both are testable without a camera; the bridge loop itself cannot be driven from a unit test, so the call site is pinned by a source-level guard in the style of the existing teardown-ordering one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The named next experiment for road-to-1.0 item 3. Instrumentation only - no
change to what is nominated, permissioned, learned or sent.
Item 3 established the per-session gate: media only ever follows the AVIO
LIVING trigger, and that trigger is armed by exactly one thing, an inbound
STUN Binding Success Response from the camera. It also left one thing
inferred rather than measured - on a host-only answer the camera's probes do
arrive, as TURN Data Indications, and are not learned, because a
relay-carried peer is dropped by one of two vetoes. Both are silent, so which
one fires is a guess. This closes that gap.
What it emits
One WARNING, at the first-media wait's expiry, gated on the wait's own unmet
exit condition (
_first_video_pt[0] is None). Verbatim, for the five casesthe experiment has to tell apart:
1. Trigger fired and media still never came - the stated kill for the
per-session model, and it has to be legible as one:
2. No Binding Success ever arrived:
3. Probe seen, vetoed by self-IP (
_is_self_peer_iprefused theXOR-PEER-ADDRESS, so
_br_cam_peeris None):4. Probe seen, vetoed by the
_bsrcfallback (no usable peer address andthe packet's source is the TURN server itself):
5. No probes at all - here shown for the fifth row of the item-3 table,
the open whose answer carried no ICE credentials:
The two vetoes are now different strings, which is the whole point of the
change. A test asserts they differ rather than only that each is present.
use-candidateis carried beyond the four facts asked for because it is whatseparates "nominated one candidate on an unroutable subnet" from "nothing was
ever nominated", and the item-3 table has a row for each.
A healthy open emits nothing new above DEBUG
The report is reached only when the first-media wait expires with no video
payload type observed - the loop's own unmet exit condition, not the broader
_vpt not in _SDP_VIDEO_PTScheck below it. Every open that delivers mediareturns before it. The per-probe self-loop drop stays at DEBUG: it fires per
packet and would drown the single line. A source-level test pins both.
Correction to the item-3 write-up
_br_stun_resp_countcannot answer "did a Binding Success arrive". It countsBinding Success responses THIS side sends in reply to inbound Binding
Requests, it only increments on the direct
sendtopath (a relay sessionanswers via Send Indication and leaves it at 0 even when it did respond), and
it was dead code - initialised, incremented, never read. It is published
anyway, but a separate inbound counter was added for the actual fact.
That counter's placement is load-bearing: it increments before the
_use_plain_rtp and not _tutk_trigger_sentgate, not inside it. Inside, itwould be an alias for
_tutk_trigger_sentand "none ever arrived" and "onearrived and the trigger still did not go" would stop being distinguishable -
which is half of what the run is for. A source-level test asserts the
ordering.
The write-up's
sdes_open.py:4689for the emit site is stale onmain(thatline is now the sprop capture). The two veto sites are exactly as described.
Fleet-shared
This runs on the SDES bridge path, so it is shared by every SDES camera in
the fleet, not scoped to the failing unit. It is logging-only: the four
bridge-thread facts are published onto
_bridge_fnat their write sites, theway
_sprop_donealready is, and nothing in the bridge reads them back.Method
The bridge loop cannot be driven from a unit test, so the message building
and the per-probe classification are pure module-level helpers tested
directly, and the call site is pinned by a source-level guard in the style of
the existing teardown-ordering one. The classifier takes the values the
bridge already computed (
_br_cam_peer,_br_obs) rather than re-derivingthe decision, so it cannot drift from the sites it describes.
Addresses and counts only - no key material, no ICE credentials. The line
reaches
home-assistant.logand users paste that into public issue reports.23 new tests. Full unit tier green (1070 passed, 5 skipped);
ruff check aidot_cameras/ tests/clean.