PSA: UDP stream mpegts formatting

[wobbals]

In investigating some decoding issues encountered when using live preview on a HERO5, I came to realize an important point I haven't seen documented elsewhere, please forgive me if this is duplicate information. I suspect few issues posted to this repo are suffering from this behavior:

UDP payloads received from the device live preview feature are not well-formed mpegts

...at least without some manipulation of the payload. Depending on the demuxer / container parser implementation downstream, this may disrupt bitstream decoding.

For this analysis I used:

• GoPro HERO5
• 2.7K Res, 30FPS
• Triggering live preview with combination of http://10.5.5.9/gp/gpControl/execute?p1=gpStream&a1=proto_v2&c1=restart and UDP keepalive messages, as defined in this repo and https://github.com/KonradIT/goprowifihack (thanks for all this work, btw -- hats off to you, @KonradIT!)

Observe:

1. mpegts defines a multiplexed bitstream segmented into (at most) 188-byte packets, each starting with the sync byte 0x47
2. In my testing, I observe received UDP payload size of 1328 bytes per datagram/packet. I suspect this could change with different video configurations. This is not enough to indicate a problem, but in inspecting the bitstream, we see that
3. The sync byte for the first TS packet(s) in every UDP payload is at a fixed offset from the first byte in the (UDP) packet.

It appears the device is prepending each UDP payload with a 12-byte header. I don't recognize the header format (doesn't seem to be RTP or SRT), but it appears there is at least a sequence number (for the UDP packet sequence, not the TS packets) and a seqno overflow counter packed in to the header. If you strip away the first 12 bytes of each received UDP packet and pass it to a TS parser, many the warnings previously emitted by ffmpeg/ffplay (eg ffplay udp://:8554) and gstreamer about the input bitstream go away. Additionally, when decoding the AVC bitstream, there's a significant reduction in visible artifacts. I suspect those artifacts and errors are introduced when the fixed header seqno rolls over the TS sync byte magic number 0x47; this would be enough to confuse most demuxers I have worked with.

I'll follow up if I figure out anything else that's interesting/relevant, but wanted to get this out there. I haven't figured out a good way to integrate a solution to this repository, but some form of packet preprocessing before passing to the TS demuxer might go a long way to improve the live stream experience. I would suggest to spawn a separate child to bind local UDP port 8554, strip the first 12 bytes of each received payload from the camera, and replay the result to a loopback port which your player/parser can bind to.

[KonradIT]

Interesting! I'll investigate further with my cameras. Seems a simple UDP server/client would suffice to redirect the stripped packets to a localhost feed.

[khaitu]

Nice investigation!

Did you get a chance to test this? Running it through a small UDP client/server to strip out those 12 bytes yields a whole lot more valid packets as you say, but there are still a lot of "corrupt packets" reported by FFmpeg. Setting the log level to debug yields Continuity issues with the packets in the form:

Continuity check failed for pid 4113 expected 5 got 12
Packet corrupt (stream = 3, dts = 374400).

I suspect GoPro either didn't follow the spec properly or purposefully broke it to fool parsers. I'll dig a bit deeper meanwhile. Using a GoPro Hero Session 5.

[wobbals]

Hmm, I did not have the same result when using a custom AVIOContext for my input format. Maybe check to ensure the packet ordering is being preserved? Within that 12-byte custom header is some kind of sequence number between UDP packets. It rolls over periodically but might be helpful still.

[khaitu]

Ah. Thanks for the tip! I was trying to proxy the stream as you suggested initially. Tried both that and a custom AVIO context in the end and strangely the proxy seems more CPU efficient. Though being an FFmpeg novice I'm sure I'm not using the AVIO context the most efficient way. Interestingly, FFmpeg handles the stream errors better via the proxy than via the custom context. Thanks again!

[jcam]

Sorry for dragging this up from the depths... do you have your work documented somewhere? I am trying to get this going with my hero5 black and it seems to draw the first 1/8th-1/4 of the screen then gets corrupted...

[pasdVn]

Hi guys,

and thank you for this extremely useful information @wobbals ! I am using gstreamer for processing and viewing the stream and always had those ugly artifacts within the picture. Gstreamer's tsdemux throwed a lot of warnings (CONTINUITY: Mismatch packet...) and the h264 decoder as well...
When cutting the first 12 bytes of the udp pdu the picture is super clean!

Gstreamer also still complaine about some invalid packets (tsdemux). However I found out, that you can fix this by cutting off the "unused" part of the udp pdu: Gopro seems to fill unsued bytes with zeros, so that each udp pdu has a fixed size of 1328 bytes. I just look for the first "zero" within the 188-Byte TS packet grid and dropped the rest. This eliminates all warnings for me (with gstreamer).

Thank you one more time for your investigations!

PS: Regarding the counter in the 12Byte header I think that the counter overflows after a fixed time (around 0.5s). At least this time keeps quite constant and so the point of overflow varies with the bandwith (i.e. udp pakets per second).

PPS: If you like and provide a little python script that modifies and forwards the udp packages.