RTSP Stream Artifacting and Distorting

[preeefix]

Project Version

0.1.2

Description

During viewing and recording the the RTSP Stream, I noticed significant artifacting and distorting of the following clip.
tmp.mp4.zip

Viewing the file in VLC leads to the image showing as distortion, while viewing in QuickTime Player the clips shows as artifacting at the top and a black/green screen at the bottom.

FFMPEG -version

ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
built with Apple clang version 11.0.0 (clang-1100.0.33.17)
configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
libavutil      56. 31.100 / 56. 31.100
libavcodec     58. 54.100 / 58. 54.100
libavformat    58. 29.100 / 58. 29.100
libavdevice    58.  8.100 / 58.  8.100
libavfilter     7. 57.100 /  7. 57.100
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  5.100 /  5.  5.100
libswresample   3.  5.100 /  3.  5.100
libpostproc    55.  5.100 / 55.  5.100

VLC Version

3.0.8

QuickTime Player Version

10.5 (1015.2.1)

[roleoroleo]

The RTSP stream is one of the 2 big problems of this alpha version.
The artifacts at the top of the image are a known issue and I don't know how to solve it.
It depends on how the frames are extracted from the system.

I try to explain better...
The frames are extracted from a mmapped file: /dev/shm/fshare_frame_buf
It is a circular buffer of about 1.7 MB which contains the video streams (without audio): both high and low res.
The file has a 300 byte header that is difficult to understand.
However at address 0x12 and address 0x16 it seems that there are circular buffer read/write indexes.

These indexes point to a buffer address where a frame is present.
Each frame consists of a 22 byte header and a payload.
The first 4 bytes define the size of the frame (without header), an absolute frame counter follows.
Bytes 19 and 20 appear to be a frame counter differentiated for high res and low res.
The problem is this: SPS-PPS and IDR are easy to identify because SPS contains the frame size (width and height).
But the other frames have no identifier and I'm not able to attribute them to the HD or SD stream.
This last counter seems to be a good way to identify them but when the system boots hd counter and sd coutner start from the same 0 value.
When the 2 numbers assume different values then it is possible to use the counter as identifier.
The artifact on the top is the low res p-frame (wrongly recognized) applied to the high res i-frame.

Some other info.
The SPS frame is an exception to the others because it does not start immediately with the NALU but contains another 6 byte header.
The other frames all begin with the NALU 0x00000001.
SPS-PPS-IDR are always in sequence all together and after IDR there is always a frame that seems to have nothing to do with the video (it does not start with a NALU) but has a header with a frame counter.

I wrote a demon that reads the address in the header at 0x16 and takes it as the upper limit.
From there he starts looking for NALUs and waits for the first SPS.
Once this is done, it extracts all subsequent frames by basing the recognition on the progressive counter.
The algorithm can certainly be improved, if someone wants to go further.

[bytesex]

@roleoroleo just wondering - how to get a knowledge to hack in such a way you do it?
Have absolutely no idea how to deal with unknown firmware having no source codes.

What tools do you use? Maybe there are good books to get started?:)

[bytesex]

@crdnl could you please tell, how to reach rtsp from VLC?
what is the url? I tried rtsp://ip_address:554/ch0_0.h264 with no success

[roleoroleo]

I got this knowledge only with a lot of work (and I don't mean my job) in embedded system.
And a lot of google searches.
Several years ago with routers and now with the cams (guudgo, ezviz and yi).
And that's why I can make big mistakes compared to people who have learned true "scientific" knowledge.
I have a solid foundation in C programming and a lot of little in-depth knowledge in other languages.
There are probably good books, but I can't advise you.

[preeefix]

@crdnl could you please tell, how to reach rtsp from VLC?
what is the url? I tried rtsp://ip_address:554/ch0_0.h264 with no success

File a new issue for it. I can help try to debug there.

[starbuck93]

@crdnl could you please tell, how to reach rtsp from VLC?
what is the url? I tried rtsp://ip_address:554/ch0_0.h264 with no success

I got it to work without using :554. Just rtsp://ip_address/ch0_0.h264

[keatond]

installed firmware but when I try to stream to my Synology the video flashes to black every few seconds and then the camera resets.

[JakobStadlhuber]

First of all nice work, thank you for what you do.
My Question is if you have any news about the rtsp stream or an alternative streaming ideas because the rtsp stream is a little bit unreliable.

[roleoroleo]

The known issue discussed here, is related only to some artifact in the images when the system boots.
After a few minutes there should be no more problems.
What problems do you have?

[JakobStadlhuber]

The known issue discussed here, is related only to some artifact in the images when the system boots.
After a few minutes there should be no more problems.
What problems do you have?

After a longer period of time my VLC-Client disconnect from the RTSP stream with an error. Then I have to connect again (this is only possible after a couple of seconds),

[roleoroleo]

Unfortunately sometimes there are crashes of RTSP server.
This is the reason why there is a watchdog.
I think that this problem is not solvable.

[koichirose]

I'm having the same issues with this cam.
I purchased one of them a while ago (the one with the Mstar chipset, I'm not sure if Allwinner is newer), found your Mstar hack, flashed it and I was quite happy. I bought 3 more, but this time they came with the Allwinner chipset.

Both cams have the same amount of RAM, the rmm process uses about 40M on Allwinner, 50M on Mstar, but the Mstar has ~8M free, while the Allwinner has ~2M. Not sure why, there must be something else running. They have pretty much the same config:

SWITCH_ON=yes
SAVE_VIDEO_ON_MOTION=yes
SENSITIVITY=low
LED=yes
ROTATE=no
IR=yes
--
HTTPD=yes
TELNETD=no
SSHD=yes
FTPD=no
BUSYBOX_FTPD=no
DISABLE_CLOUD=yes
REC_WITHOUT_CLOUD=yes
MQTT=no
RTSP=yes
RTSP_STREAM=high
ONVIF=no
ONVIF_PROFILE=high
NTPD=yes
NTP_SERVER=pool.ntp.org
PROXYCHAINSNG=no
RTSP_PORT=554
RTSP1_PORT=8554
ONVIF_PORT=80
HTTPD_PORT=8080
USERNAME=
PASSWORD=
FREE_SPACE=0
ONVIF_WM_SNAPSHOT=no
TIMEZONE=CET-1CEST,M3.5.0,M10.5.0/3

The RTSP stream is sometimes choppy.

Do you think this can be fixed? Is it better to return them and hope for a Mstar?
Please let me know if I can be of help for testing and thank you for this amazing work, it is unbelievable.

[roleoroleo]

I don't know if it's possible to fix it.
It would take the work of an expert
But I think it is just as difficult to be able to receive MStar cams. You could try.

[preeefix]

Maybe a bit too early to close this.

I'm seeing a few things, though hard as they are to demonstrate. Now running 0.1.4

There's frame hitches, but they are less frequent and only seem to appear when there isn't a huge amount of movement in the frame.

Additionally, and rarely, there's a complete dropout of the frame and it goes gray.

Significant improvement over 0.1.3 though.

[rambo90]

Hello. I have just installed your hack. I left working only rtsp (high) and http services.
But rtsp stream stops after 10-30 seconds after running. What can I do?

[roleoroleo]

What client are you using?
Which settings?
Did you try 0.1.5?

[rambo90]

After upgrade it works better. Thanks.

[roleoroleo]

See this post for more details: #81 (comment)

[Daylights]

I am running 0.2.0 on 8.2.0.0A_202007211802. All works fine, but I am still seeing occasional artefacts and distortions on RTSP High. It's not seen on the Cloud app, only on RTSP. It's worse during the day, even when I am not using the WiFi for other work?

What I've tried and didn't make a difference:

• Tested 3 different camera's, results are the same
• Disabled all other services including Cloud
• 81% WiFi Strength, WiFi is barely used by other devices
• Changed WiFi channels
• Tested multiple RTSP viewers, like Zoneminder and VLC
• RTSP Low seems more stable, but 640x360 is not an ideal resolution

I couldn't find how to verbose log the RTSP service?

[roleoroleo]

Try to run rtsp server manually and check if there are any errors in console.

[Daylights]

No errors, just "Play this stream using the URL..."

[roleoroleo]

Sorry, try debug option and check the output when the client reads the stream.

[Daylights]

Well I cannot make up a lot out of it, since there is a lot of information and no time entries. This is the log of about 30 minutes and it happened twice during that time.

log_rtsp.txt.zip

[roleoroleo]

Approximately at lines 8340 and 14200 there is a wrong sequence: two IDR frames close together.
You can distinguish them by the length: "frame_len: 20".
I don't know why, it seems that the search for frames in the buffer has stopped for a while.
But there are no other errors.

[Daylights]

Is there any other way I could troubleshoot this? I can see it happening way more during the day, barely at night (less light, movement?). Interestingly, the location inside the house also plays a role. I have 3 camera's, which all work almost flawless inside the living, even in the loft with poorer WiFi reception. But in the hall, they all give me the same results with occasional distortion.

I'm not that bothered by the 2 seconds of distortion every now and then, but it's enough for Zoneminder to record a motion event for it.

[roleoroleo]

I could try to add other logs.

[roleoroleo]

Try this:
rRTSPServer.gz
I increased verbosity and buffer.

[Daylights]

Sorry for not having replied in a while. With your supplied version it looks to be more stable than it was, so getting a log also looks to be more difficult. When I run /home/yi-hack/bin/rRTSPServer -d 2> /tmp/log_rtsp.txt it takes a while before it shows distortion, but when it does the SSH session was already expired and the log is removed? I do see a couple of debug logs being created now.

Any suggestion how I can help more?

[roleoroleo]

I fixed other errors.
I will send a new version.

Here it is:
rRTSPServer.gz

[Daylights]

I have been testing with the 0.2.1 version, unfortunately still distortions. In the attached log_rtsp.txt somewhere near the end an example of visible distortion. I also see many frame_len: 20 records, although I have no idea is that is expected right now.
logs.zip

Edit:

Also added logs from the rRTSPServer version you added above this post.
logs_2.zip

[roleoroleo]

Try this version, I fixed a problem.
rRTSPServer.gz

[Daylights]

Alas, still seeing distortion. Added new logs with the version above.

logs.zip

[roleoroleo]

The log is perfect.
No lost frames, no freezes, nothing.

[WMP]

That black strips is normal?

[roleoroleo]

No, it isn't.
Frame or snapshot?

[WMP]

Screenshot from VLC, i have this from 0.1.8, and i thinks that can me power supply issue, becaouse i have connected USB mini to ethernet cable. I have this same on 0.2.2, but additional 0.2.2 is unstable.

[roleoroleo]

Occasionally a few frames are lost, both internally between processes and across the network.
If it happens infrequently it's ok, but if it happens often it's not normal.

[WMP]

I have this all time, but only on ethernet cat 6. On normal USB cable 1m i havent this problems.

[WMP]

I adjust voltage on my power supply and now i havent strips ;)

[krzada]

Hi @roleoroleo, I am experiencing similar problem with BFUS 8.2 (yi-hack v0.2.4). High res stream is chopping so badly and artifacts are visible occasionally. Everything is disabled besides hi-res stream. Please find my log file enclosed. Any idea what is wrong ? I would appreciate any kind of support !

Moreover, do you know why there are both 360 and 1080 frames captured ?

1587155154271: frame_len: 755 - cb_current->size: 49152
1587155154271: frame_len: 755 - frame_counter: 11793 - resolution: 360
1587155154271: frame_write_index: 2 - cb_current->output_frame_size 42
1587155154271: frame detected - frame_len: 2052 - frame_counter: 11791 - frame_counter_last_valid: 11791 - resolution: 1080
1587155154271: frame_len: 2052 - cb_current->size: 262144
1587155154271: frame_len: 2052 - frame_counter: 11791 - resolution: 1080
1587155154271: frame_write_index: 0 - cb_current->output_frame_size 42
1587155154271: frame detected - frame_len: 509 - frame_counter: 11794 - frame_counter_last_valid: 11794 - resolution: 360

yi_log.txt

[roleoroleo]

The log is ok, no frame lost, no sync lost.
The capturing part is ok.
Check your network connection.

About frames captured, the buffer contains both, high res and low res.
The algorithm detects both types of frames and send them to a buffer.
Only the resolution selected is streamed.

[krzada]

@roleoroleo, thanks for quick response and checking the logs. Wifi strength is quite ok to my mind - around 80% - or you had something else in mind by indicating wifi connection ?

Additionally, swap is created, system metrics are ok as well. Maybe I missed something and that is why I cannot make it working properly ?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.