support live555 servers older than 2017.06.04 (eg some Reolink models), which have buggy RTP/TCP #17
Comments
scottlamb
added
the
interop
|
A possible tool: a program that uses the Rust rtsp client that simply opens a connection and tallies the packets received and logs anomalies. This, along with other items you may have learned about with other cameras, could give a quality report on the camera's ability to keep a stream going (and the libraries ability to handle error/resets). It could be run outside of moonfire-nvr and left running for a period of time, say up to 24-72 hours. Almost like a nurse taking vitals before a doctor sees a patient: getting a profile of the camera's ability to send packets conforming to the specifications. |
|
I was starting to do something like what you looking into in this issue with ffmpeg and the real server when I got interrupted or distracted. (Having these hardware issues really throws me off track.) Generating logs has taken some time for me to perfect and I'll go the extra distance (i.e. handling & preserving ANSI codes) where a lot of prospective users may not. If there were a simple program that you feed the IP, logon credentials to that would then generate a log on the camera's ability to manage an rtsp feed and its resilience (and that of the network) could help crystallize issues that need consideration or handling and provide a stream-lined path of zeroing in on issues. |
|
This specific issue is in theory fixed in Retina. I'm going to fix another Reolink problem I found on my own test camera then prepare a Moonfire NVR change to pick up both. Re: tool: did you see this comment? It's a little bit like what you're talking about. But I'm not super motivated right now to polish it into something that gets detailed spec compliance info automatically. I have more improvements I want to make than time already, so I'm not going spend a lot of time trying to to get detailed bug reports from people who don't care enough to gather them. I'm prioritizing getting Retina and Moonfire NVR working really well on cameras I have access to (including polished on-NVR and on-camera analytics, live view, and scrub bar UI) over getting it working on every camera out there. |
|
I found a live555 changelog entry about this: The problematic Reolink cameras say My workaround isn't working well: at the timeout of the original session, the camera seems to be sending bytes which aren't not properly framed as RTSP interleaved data, we're dropping the connection, and the cycles repeats. Another option is to just ensure we send a |
|
I'm updating the ip 48 GarageWest camera which is: and if successfully accomplished, I will test with the Retina client example. |
|
You have to be careful in selecting which upgrade package. Firmware number is not sufficient, you also must align to the model number of your camera. I mistakenly downloaded the first firmware match only read the PDF within the zip archive that the model that upgrade applied to did not match my model. I then returned to Reolink's site and found several matches to my firmware, and thus discerned which one applied to my camera model. I applied the patch and rebooted. Will test this evening with Retina. |
|
Here's the cause of the framing error in the old live555 code: Boolean RTPInterface::sendDataOverTCP(int socketNum, u_int8_t const* data, unsigned dataSize, Boolean forceSendToSucceed) {
if (send(socketNum, (char const*)data, dataSize, 0/*flags*/) != (int)dataSize) {
// The TCP send() failed.
if (forceSendToSucceed && envir().getErrno() == EAGAIN) {
// The OS's TCP send buffer has filled up (because the stream's bitrate has exceeded the capacity of the TCP connection!).
// Force this data write to succeed, by blocking if necessary until it does:
#ifdef DEBUG_SEND
fprintf(stderr, "sendDataOverTCP: resending %d-byte send (blocking)\n", dataSize); fflush(stderr);
#endif
makeSocketBlocking(socketNum);
Boolean sendSuccess = send(socketNum, (char const*)data, dataSize, 0/*flags*/) == (int)dataSize;
makeSocketNonBlocking(socketNum);
return sendSuccess;
}
return False;
}
return True;
}
(btw, live555's latest code is still bad. If one client's TCP window fills, the entire server will stall waiting for it to come back.) |
|
I flashed a RLC-420-5MP, hardware version IPC_51516M5M with the 12/11/2020 firmware, aka Here are some options I see:
|
scottlamb
changed the title
|
I just wrote to Reolink. |
|
or I'd be interested to learn what other manufacturers use as servers and their conformance to standards. It seems having Scott spend time on one brand which has been documented to be using a buggy server has diminishing returns. I've been using Reolink because they're cheap and if they get stolen or vandalized, it's not going to be expensive to replace. If other brands provide a lot less problems conforming to standards, then it would be appropriate to weigh their cost. If Reolinks cost $40, and Brand X costs $70 and works well within the RTSP protocol, then it may make sense to switch brands. Now, if the next least expensive brand that plays nicely costs $150, then it might merit further consideration. I think further time Scott spends on Reolink's issues is time away from other enhancements that affect all users of moonfire-nvr. |
|
I'm currently not sure what manufacturers to recommend, though. :-(
|
|
I flashed my older one as described above with the correct firmware version for the model I had. I, too, have the session (using the LIVE555 openRTSPclient) showing: I tried several sessions using the Retina client and continued to have problems. The best session would be about 16 seconds and then I'd get an error message. Examples: I had to wait several minutes or I would get receiving data packets type errors which is line with Scott's analysis. Now, I did successfully save a 10 minute video (with audio) using the Live555 openRTSP client. There may be blips, but it continued on. Here's what was printed to my console during the 10 session: |
|
It looks like live555/openRTSP is just skipping all the bytes it doesn't understand. The frame before will be corrupt (because it has some duplicated data in place of some fresh data), and it might do weird things as it interprets video/audio data as headers. I'd prefer to avoid that if possible. Not only does it feel gross, but it could mask other problems (including bugs in Retina itself), and it'd be a security problem in some contexts (the equivalent of an HTTP desync attack). I think if we add a workaround for the bad framing, I'd prefer to do one more narrowly focused on this bug. Or just avoid the issue by adding UDP support and recommending UDP with these cameras. |
|
what about having three modes in Retina? 1) as-is - croaks on certain anomolies as it does now, 2) tries to continue, but logs problem for retrospective analysis, 3) dangerous mode --- imitates ffmpeg & live and does what it can to continue saving MP4 files without error reporting? I like the 2nd mode if it could have a log file that might be reviewed every 24-48 hours that could be processed with a grep command to give statistics such as 453 bad packets, 800 incomplete data, 1300 time aberrations. These, in turn, could provide some profile of a particular camera helping to triage the panoply is issues. Maybe even have a facility for pushing the logs files to a server which digests the entries? |
|
I've moved a little in the direction of log-but-continue with eg dadbd44 (which logs about something that used to be a fatal error, respecting enabled log levels), and I'll likely continue to as Retina gets more mature. But it's a fair bit of work to generate excellent logs, especially after things start going downhill. Eg once we're no longer sure we're at a correct packet boundary, every assumption we make from then on is suspect. A central server is even more work, and I imagine many privacy-conscious users wouldn't be interested in using it, and/or we'd have to be very careful about what information it collects. The most useful thing to me is pcaps, followed by full TCP-level stream contents according to Retina. Either of those would be out of the question. Realistically, this would stay on my list for a very long time before I run out of tasks with lower cost and more certainty of pay-off. |
This was an attempt at #17 that didn't work out. Besides additionally hitting the framing error, my Reolink test camera never timed out the old session while the new one was in progress. This behavior is different than the vanilla 2013.04.06 live555 server, so apparently Reolink sprinkled in their own brokenness on top. In any case, the other solutions described in that bug are more likely to work. There's one narrow case I want to keep working: ignoring RTCP messages immediately prior to the PLAY response. More recent Reolink cameras do this. I just have this behavior on all the time now.
scottlamb
changed the title
|
For the Reolink RLC-420-5MP / hardware version IPC_51516M5M: at my request, Reolink built firmware v3.0.0.589_21091583 which advertises The behavior when the TCP window fills is still bad. (I verified the current version of live555 is buggy; Reolink neither added nor fixed this bug.) But this is likely to come up less often now that there's no extra data from stale TCP sessions. Reolink told me that the RLC-410 / hardware version IPC_3816M is too old and has been discontinued, so they won't be updating its firmware. |
|
For posterity, I purchased directly from Reolink 2 RLC-410 5mp cameras on 7/14/2021; $76.98 inc. shipping for 2 cameras. They arrived 7/24/21. Here is the firmware they currently have |
|
I ran ffmpeg to see what the rtsp server version is. Note, you need to include "-loglevel debug " to get line 45 "a=tool:LIVE555 Streaming Media v2013.04.08" Conclusion: The change from 2013 to 2020 of the LIVE555 server in the Reolink camera occurred after July 2021. |
|
I think I'm going to call this done. The major improvements here we've made to date are:
I haven't added any kind of "resync" to recover from these cameras' buggy behavior after the TCP window fills. Maybe I'll add that yet, but I'm not seeing complaints right now anyway. |
|
I would like to kindly point out my findings on Reolink cameras if this is not already known: Reolink RLC-410-5MP IP camera firmware unpacker and reverse engineered technical details. Maybe someone is able to patch the camera Linux system so that the Live555 server is updated (... or even replaced with a completely different streaming-server). |
|
Wow! I gave this a quick 2 minute scan and felt like I had entered
Heaven. This is fantastic. I had contacted George Hillard and he did not
reply so I gave up. But what you have accomplished here is what I have
dreamt about if i had the time. I've been pursuing a Raspberyy Pi based
approach using vetted hadware components and then hoping to use gstreamer
and then whatever Scott comes up with that gives a no problem stream, or if
there are problems, they can be identified as to what the cause is, i.e.
long CATV lines causing voltage drops that is afflicting the hardware, or
packet congestion at a node such as the PoE router.
I cannot wait to sink my teeth into your project, though I'm going to help
the GenPI64 project if I can to create Gentoo images for Raspberry Pi so I
have a completely controlled playing field.
As for LIVE555 -- I really do not think the maintainter's approach of
limited transparency is good and possibly a red flag. I suspect he may
have conflicted interests in having his code used by camera manufacturers
such as Reolink and projects like VLC and therefore is unwilling to stage
the codebase in a repository that makes the history of changes fully
transparent. That plus his requirement that I must use an email address
other than gmail to post to his list despite his knowing I've been a
developer at a major software company for over 22 years.
I can't wait to carefully read your project. (I've been embarking on the
side to learn electronics... again so I can look at something like the
Reolink circuit board and have a better understanding of their design. I
also confess I'm tired of my Zylex switches dying because of a cheap
capacitor in the power board -- twice I hunted down the blown capacitors
and replace them so I could resuce the switch. I have one now where my
capacitor tests indicate all of the capacitors are okay, so I need to learn
more to determine the power unit has failed and how to fix, or better yet,
replace it. All this inspired by a posting on the Amazon comments for the
Zylex showing a way on how to replace the blown capacitors.) I've
digressed, but I do that when i get excited after finding projects such as
yours and Scott's. I'll be in touch.
John
1566 Court ST NE
Salem, OR 97301-4241
(707) 812-1323
…On Thu, Mar 10, 2022, 1:58 AM Hajo Noerenberg ***@***.***> wrote:
I would like to kindly point out my findings on Reolink cameras if this is
not already known: Reolink RLC-410-5MP IP camera firmware unpacker and
reverse engineered technical details
<https://github.com/hn/reolink-camera>. Maybe someone is able to patch
the camera Linux system so that the Live555 server is updated (... or even
replaced with a completely different streaming-server).
—
Reply to this email directly, view it on GitHub
<#17 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABXP4JK66B7LJYNILKTN6TU7HBSVANCNFSM5CFRZCNA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Latest understanding: these cameras are using
LIVE555 Streaming Media v2013.04.08(according to the session-leveltoolattribute in theirDESCRIBEresponse) which has at least two major bugs described in live555's changelog:also, there was a follow-up to the 2015.05.03 entry two years later:
Currently your best bet to successfully talk with these cameras is to ask the manufacturer for a firmware upgrade (see note below for the Reolink 420-5MP / hardware version IPC_51516M5M) or connect with UDP via
retina::client::Transport::Udp(#30). We have some other ideas below.Original comment:
As mentioned in this Apr 29th comment on a moonfire-nvr issue, Reolink cameras have an odd behavior. They sometimes set up a stream with one ssrc, eg:
then send RTP packets for both this one and another ssrc. I think the latter is for an earlier failed session or something. Sometimes they'll even send the RTP packets for the other ssrc before the connection is even authenticated, which seems like a security hole (if you expose the cameras to an untrusted network, which is probably a bad idea anyway).
Currently when this happens, Retina will fail with an error such as the following (from one of @jlpoolen 's logs today in this moonfire-nvr issue):
I think we should add an option to so that when we know what
ssrcto expect, we just (log and) ignore packets with any otherssrc. This will likely make these cameras behave acceptably, especially if (as I suspect) the otherssrcgoes away after a minute or so when the stale session times out.The text was updated successfully, but these errors were encountered: