video: native video proxying with recording and browser playback - #42
Merged
Conversation
The implementation was written for MAVLink, where a message is at most 280 bytes and the peer always speaks first. Video moves megabytes a second through the same code and broke every one of those assumptions. Fixed: the handshake matched the literal "GET / HTTP/1.1", so any other path was refused; the pending buffer was 1024 bytes, which dropped any frame larger than ~1017; the payload was assembled in a variable-length array sized from the wire, so the peer chose a stack allocation; a short write returned 0, which the caller read as failure and tore the connection down; and recv() silently discarded the remainder when the caller's buffer was smaller than the frame. Framing now happens once into a queue with a sent-offset, so a partial write resumes instead of re-sending a header, and fragmentation, ping and close are handled rather than assumed absent.
Grows KeyEntry 168 -> 344 bytes for the video settings: up to three ports, per-slot options, viewer and publish keys, a disk budget, the MAVLink grace window and a per-slot RTMP path. The append-only contract holds in both directions -- readers zero-extend a short record, writers preserve a tail they do not understand -- so old and new binaries interoperate. The layout was an unchecked ABI shared with keydb_lib.py's PACK_FORMAT. It is now asserted: sizeof, every offset, and that int and float are four bytes. Writing that revealed the same hazard in ConnEntry, where a naively appended field lands at offset 60 inside padding that older Python writers zero, with sizeof unchanged and no size check to catch it -- so the padding is named and the video fields start at 64. Cleanup gains a second budget rather than one shared pool. Video and telemetry are deleted under the same retention but from separate quotas, so a busy camera can never evict a user's tlogs.
Brings video into the machinery that already solves NAT traversal, per-entry credentials and log retention for MAVLink, so recordings sit beside the tlogs under the same rules and mediamtx -- which ran with no authentication, no supervision and no retention -- can be switched off. Video runs in a long-lived child of the parent, not of the MAVLink session child. The session child idles out after 10 s, which would take video with it, and a password holder must be able to publish with no MAVLink at all. Admission has two independent paths: a publish password standalone, or a MAVLink session from the same address within a grace window, which is what lets video ride through a telemetry dropout. Ingest is MPEG-TS over UDP, RTSP, or RTMP. RTSP is spliced to a loopback ffmpeg untouched from its first byte: the protocol is request/response, so classifying before ANNOUNCE deadlocks, and answering OPTIONS makes ffmpeg's listener reject the following request for being CSeq 2. RTMP could not use that splice. ffmpeg's listener answers FCPublish with a bare "onFCPublish" -- the command name alone, no transaction id, no null, no status object. Measured against the Phoenix camera: the exchange completes as far as createStream, ffmpeg grants stream id 1, and the camera then waits 5 s and hangs up without ever sending publish or a frame. Replaying its bytes at a local ffmpeg produced byte-identical responses, and the same camera published 74 MB to a server that answers properly. So RTMP is spoken here -- handshake, chunk demux, and the commands a publisher uses -- and the media converted to FLV, which ffmpeg is happy to demux from a pipe. That also drops the loopback port, and gives RTMP somewhere to carry a publish password. Fan-out is a memcpy from a per-slot ring: the publisher never inspects viewer state, never blocks and never allocates, so one slow viewer cannot stall the stream or the others. Viewers join at a validated PAT/PMT/random-access point so the stream is decodable from the first byte they see. Two things a camera needed that a synthetic publisher never showed. ffmpeg's AVCC to Annex-B conversion emits a zero-length NAL unit ahead of every access unit for this camera -- 8362 of 25254 on a clean capture -- which is invalid H.264 that Chrome refuses outright and Firefox plays regardless, so -bsf:v h264_metadata rewrites them out. And the muxer buffered a 32 KiB AVIO block before writing, most of a second at this bitrate, so the live flags the design called for are now actually passed. An unauthenticated peer drives the RTMP parser, since admission needs the credential out of publish, so the exposure is bounded throughout. A handshake owns nothing: several negotiate side by side in a pending pool and the slot is awarded on publish, after admission -- if a pending handshake held the slot, one byte from anywhere would deny publishing, and letting a newcomer evict the incumbent only makes that last-arrival-wins. The pre-publish phase is capped in bytes and seconds, every length from the wire is checked against what is buffered, chunk state is committed only once a whole chunk is present (otherwise ordinary TCP segmentation makes a header parse twice and apply its timestamp delta again), only fmt 3 may continue a message, and assembly is bounded across the session rather than per chunk stream. Acknowledgements and ping replies are sent: a publisher that sets a window and never sees a type-3 back is entitled to stop, which presents as a camera that streams for a while and then stalls. Credentials are redacted before anything is logged. A viewer may authenticate with ?pw=, which unlike the 60-second view token is long-lived, and the log is kept on disk and rendered into the admin UI.
|
BTW, we've been improving RPanion's video support and I was coincidentally thinking of asking you about re-publishing of video like you did at the last conference: stephendade/Rpanion-server#417 |
Video settings on the owner and admin forms, with ports allocated by an admin -- they share one listening-port namespace, so letting every owner pick invites collisions -- and everything else owner-controlled. Browser playback is WebSocket to mpegts.js, vendored rather than loaded from a CDN so an external script cannot change what runs in an operator's browser. It authenticates with a short-lived token signed with the entry's existing MAVLink key, minted on demand: a token rendered once at page load is refused by every reconnect after the first minute, which presents as "it only works if I reload". The player's failure handling took several passes, each of which looked right. A publisher going away is a clean close, reported as LOADING_COMPLETE, so listening only for ERROR missed the one case reconnect exists for. The video element is kept across reconnects, because replacing it drops Picture-in-Picture. Only an unsupported codec is permanent -- treating every MEDIA_ERROR that way gave up on the first transient append failure, which Chrome raises far more readily than Firefox, so an H.264 stream Firefox was playing reported "cannot decode, probably HEVC" in Chrome and latched. And progress comes from the element's own events, not MEDIA_INFO: that only fires once MediaInfo.isComplete(), which requires hasAudio to be exactly true or false, and it starts null and is set only when audio metadata arrives -- so a video-only stream never fires it, leaving the status stuck and the error counter never reset. Recordings can be deleted, one at a time or a whole day. An owner deletes from their own entry and an admin from any, using the same auth decorators and path grammar as the listing: the owner routes take no port2, so there is nothing there to tamper with, and only names SESSION_RE accepts are ever unlinked. Each directory component is opened O_NOFOLLOW and the unlink is relative to that descriptor, so a symlink swapped in along the path cannot redirect it. A file the daemon still has open is refused -- asked of the kernel through /proc, not guessed from mtime, because a quiet session's log can go untouched for longer than any grace window while still being written, and unlinking it would not stop the write. Adds a server page: the daemon's log tailing live, and a restart. The restart signals rather than shelling out, since the unit sets Restart=always and runs as the same user as gunicorn, so systemctl and the sudo it would need are avoided. Which process to signal is an identity question, not a search: systemd's MainPID when it answers, otherwise a supportproxy process with this installation's working directory whose parent is not also supportproxy -- min(pid) alone would pick up a second instance on the host or a reparented session child. The signal goes through a pidfd where available, so the gap between finding the pid and using it cannot land on a recycled number. The tail is polled rather than streamed -- sync workers would be tied up for as long as the page is open -- bounded at both ends, and it carries the file identity so a rotation that truncates and regrows past the old offset is reported rather than silently skipped. Credentials are redacted again here, for logs written before the daemon learned to. Row actions are icons -- download, delete, play -- as inline SVG rather than an icon font or emoji, which the pages cannot fetch and which render inconsistently; a bin glyph in particular is missing on plenty of systems. Play is last because it is the only one not on every row, so the others stay put. Each carries a title and an aria-label.
test_video.py publishes a test pattern with a clock over any of the supported transports and can check a stream back, so the matrix can be exercised without a camera. It kills its children through PDEATHSIG: an orphaned ffmpeg from an earlier run holds the publisher slot and the next attempt is refused as slot-busy, which reads as the proxy being broken. CI installs ffmpeg. It is a real test dependency rather than a convenience: RTSP and RTMP ingest hand the stream to an ffmpeg child, and every test that needs one skips itself without it -- so the runner was quietly skipping 36 tests and the whole ingest, recording and viewer suite never ran there at all. The tool's own tests no longer assume an encoder is present. Those that shell out to it are skipped without one, and the command-construction tests stub the drawtext probe: ffmpeg_publish_cmd() asks the local binary whether it can use drawtext, so on a machine with no ffmpeg they were asserting on a command built without -vf and failing on its absence rather than on anything they meant to test. Log rotation by logrotate with copytruncate, which is required rather than preferred: the unit writes its log with systemd's StandardOutput=append:, so the descriptor is held for the life of the process and renaming would leave the daemon writing to a file nothing can see. The trade is that whatever is emitted between the copy and the truncate is lost -- not merely a line -- and the alternative is restarting the daemon to make it reopen, which drops every live session. Rotated on size rather than a schedule, since the rate depends entirely on what the proxy is doing; no delaycompress, which buys nothing when the rotated file is already a finished copy; and no create, which logrotate ignores under copytruncate. README documents video for the first time: the transports and what credential each can carry, that plain MPEG-TS over UDP can carry none and why an entry used for video wants bidi_sign or a publish password, the ffmpeg dependency for RTSP and RTMP, that the video ports are public and need opening, the separate disk budget, the browser's H.264-only limitation, and how to install the rotation config.
Three flakes, all of which only showed up once CI grew a video suite to run alongside the rest. None was bad luck; each was a real race that an idle machine happened to win. Two tests connected as soon as the proxy logged "Added port". That line is printed before the socket exists -- supportproxy.cpp prints it and then calls open_sockets() -- so the connect was always racing the bind. Wait for the port to reach LISTEN instead. Read from /proc rather than probed with a connection: connecting to a user port latches conn1, so a probe would consume the very thing the test is about to set up. The orphaned-ffmpeg check counted every ffmpeg on the machine, so a sibling xdist worker starting one between the two samples reported a leak against a test that leaked nothing. Count our own children.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds optional video proxying alongside the MAVLink link. A user points a
camera at one of their entry's video ports and any number of ground
stations can watch, using the NAT traversal, per-entry credentials and
log retention the MAVLink side already provides. Recordings land beside
the tlogs under
logs/<port2>/<date>/.This replaces a separate mediamtx instance that ran with no
authentication. Off unless an entry enables it.
What's here
.tssegments, own disk budget, same retention as tlogsVideo runs in a long-lived child of the parent, not of the MAVLink
session child. That child idles out after 10 s, which would take video
with it, and a password holder must be able to publish with no MAVLink
at all. Admission has two independent paths: a publish password
standalone, or a MAVLink session from the same address within a grace
window — the latter is what lets video ride through a telemetry dropout
rather than being cut by a link flap.
Why RTMP is parsed rather than relayed
RTSP is spliced to a loopback ffmpeg untouched from its first byte.
Classifying before
ANNOUNCEdeadlocks (RTSP is request/response), andanswering
OPTIONSourselves makes ffmpeg's listener reject thefollowing request for being CSeq 2.
RTMP could not use that splice. ffmpeg's listener answers
FCPublishwith a bare
onFCPublish— the command name alone, no transaction id,no null, no status object. Measured against a real camera: the exchange
completes as far as
createStream, ffmpeg grants stream id 1, and thecamera waits 5 s and hangs up without ever sending
publishor a frame.Replaying the camera's bytes at a local ffmpeg produced byte-identical
responses, and the same camera published 74 MB to a server that answers
properly. Linking libavformat would not have helped — that is the same
rtmpproto.con the wire.So RTMP is spoken here and the media converted to FLV, which ffmpeg
demuxes happily from a pipe. That also removes the loopback port and
gives RTMP somewhere to carry a publish password.
Two further things a real camera needed that a synthetic publisher never
showed:
of every access unit for this camera (8362 of 25254 on a clean
capture). That is invalid H.264: Chrome refuses the sample outright,
Firefox plays it, and
ffprobereports nothing.-bsf:v h264_metadatarewrites them out.second at this bitrate.
Hostile-input handling
An unauthenticated peer drives the RTMP parser, because admission needs
the credential that arrives with
publish. So:pool and the slot is awarded on
publish, after admission. If apending handshake held the slot, one byte from anywhere would deny
publishing; letting a newcomer evict the incumbent only makes that
last-arrival-wins.
is checked against what is buffered; chunk state is committed only
once a whole chunk is present (otherwise ordinary TCP segmentation
makes a header parse twice and apply its timestamp delta again); only
fmt 3 may continue a message; assembly is bounded across the session
rather than per chunk stream.
Deployment
apt install ffmpeg— needed for RTSP and RTMP ingest. Plain MPEG-TSover UDP needs nothing extra.
the firewall.
sudo install -m 644 scripts/supportproxy.logrotate /etc/logrotate.d/supportproxyEntries used for video should set
bidi_signor a publish password.Plain MPEG-TS over UDP cannot carry a credential, so it is admitted on
the MAVLink-session address. On a non-bidi entry any datagram latches
the user side, so a scanner between flights can become the authorised
address and the aircraft's video is then refused until the grace window
expires. Measured on a live server: 786 distinct addresses latched conn1
across all entries over ~2 days.
Testing
531 tests pass. Beyond the unit and integration coverage, this has been
run against a real camera end to end — both an H.264 RTMP stream and an
HEVC-then-H.264 RTSP stream — recording, browser playback in Chrome and
Firefox, publisher restart, and reconnection after a proxy restart.
Reviewed by Codex over two rounds; the findings from both are fixed.
Known limitations
Linux, and nothing here transcodes. Non-playable streams get a
copy-pasteable
ffplay/vlccommand instead of a broken player.viewer must start at a keyframe, so a 4 s GOP means joining 0–4 s
behind live. Shortening the GOP on the camera is the fix.
pcm_s16beandpcm_mulawcannot be carried in MPEG-TS. Audio is offby default; enabling it re-encodes to AAC.
path. Verified against ffmpeg, VLC and GStreamer.