New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize computer vision performance on single core boards #83

Open
cedricve opened this Issue Apr 26, 2017 · 8 comments

Comments

3 participants
@cedricve
Copy link
Member

cedricve commented Apr 26, 2017

Kerberos.io works "fine" on a Raspberry Pi 2 and 3. When recording video we can write at a framerate of 7-10FPS at a resolution of 640x480. On a Raspberry Pi Zero (W) or Raspberry Pi 1, the framerate is at average 3FPS. The CPU is "acceptable", but we believe it can be reduced a lot. A possible solution would be to move some stuff to the GPU, and use hardware acceleration for video recording.

Functions to optimize

With this issue we want to tackle different parts of the machinery, that may be improved. Most of them are related to standard OpenCV functions.

  1. (TODO) Differential images, calculating the changed pixels between frames.
    https://github.com/kerberos-io/machinery/blob/master/src/kerberos/machinery/algorithm/DifferentialCollins.cpp#L28-L40

    • Possible solution is to resize the images to a lower res, before applying the different math operations.
  2. (TODO) Expositor, iterating over pixels to find the changed pixels within a ROI.
    https://github.com/kerberos-io/machinery/blob/master/src/kerberos/machinery/expositor/RectangleExpositor.cpp#L31-L44

  3. (TODO) Encoding MJPEG streaming, for viewing a live stream from the web interface.
    https://github.com/kerberos-io/machinery/blob/master/src/kerberos/capture/Stream.cpp#L167~~

    • Possible solution is to use libturbo-jpeg instead of traditional jpeg. However this is powered by NEON, which isn't available on Raspberry Pi 1 or Zero (W).
    • Edit: We started focusing on the Raspberry Pi Camera, and integrated the OpenMax CPP library which comes with native hardware acceleration to encode MJPEG. A todo is to integrate this hw encoding also for IP cameras and USB cameras.
    • Achievement: When using a Raspberry Pi and Raspberry Pi camera: you can stream MJPEG at 10-20FPS on a Raspberry Pi 1.
    • Investigate how to do it for other cameras as well.
  4. (Done) Video writing with FFMPEG (x264)
    https://github.com/kerberos-io/machinery/blob/master/src/kerberos/machinery/io/IoVideo.cpp#L300-L323

    • Recording with the h264_omx encoder on a Raspberry Pi 1 at 30FPS (640x480), with only consuming 30% of CPU power. Need to find a way to include this in OpenCV or if not possible directly in the machinery. Not an easy task atm.
    • For the Raspberry Pi Camera, we also use OpenMax to encode h264 videos at 30FPS. Similar to the MJPEG streaming, we should also replace (or create a new) IoVideo device (e.g. IoVideoOMX) which will call the ffmpeg (avcodec) directly and uses the h264_omx codec. At the moment of writing this is not available in the OpenCV repository by default. opencv/opencv#8682
    • Achievement: When using a Raspberry Pi and Raspberry Pi camera: you can record video (1280x960) at 30FPS on a Raspberry Pi 1.
    • Todo: Recording at high speed on a USB camera will not be possible because we only have 1 videoport to connect to the camera (or we need to split the videoport). For IP cameras it is possible to open multiple streams (if the camera can handle it), and therefore we could open an additional record thread and encode it using OMX h264. This would improve the framerate enormously.
    • Started experimenting with modifying the OpenCV video writer class. Hardcoded the h264_omx codec. https://github.com/cedricve/opencv/blob/h264-preset/modules/videoio/src/cap_ffmpeg_impl.hpp#L1458-L1462. Encoding is successful on 800 x 600 and 1920x1080, however doesn't work on other resolutions (600x480 and 1280x720). Need to investigate why. @jasaw reported a similar issue in his PR for the motioneyeos project. ccrisan/motioneyeos#365
    • Achievement: Successfully encoding with h264_omx codec, for USB and IP cameras, after upgrading to FFMPEG 3.4.3 and removing zero_copy with patch (ccrisan/motioneyeos#998). Thanks to @jasaw.

@cedricve cedricve self-assigned this May 3, 2017

@cedricve cedricve referenced this issue May 22, 2017

Open

Progress #13

@cedricve cedricve added this to In Progress in Work In Progress May 22, 2017

@muten84

This comment has been minimized.

Copy link
Member

muten84 commented Jul 22, 2017

Hi any chance to get it work with IPCamera too? I'm interested in video recording optmize from IPCamera.

@cedricve

This comment has been minimized.

Copy link
Member

cedricve commented Jul 22, 2017

it's possible for IP camera's also. The idea is instead of using OpenCV to write video files (which is slow and heavy operation), we'll fire up an FFMPEG command with omx hardware encoder. This will give same performance as for the Raspberry Pi camera (30FPS).

Problem that needs to be solved is compiling the omx libraries properly on Raspberry Pi (KiOS).
ccrisan/motioneyeos#998

@muten84

This comment has been minimized.

Copy link
Member

muten84 commented Jul 22, 2017

I'm going to check this reddit. However do you think Raspbian users of Kerberos.io can just use hardware accelerated video encoding??

@cedricve

This comment has been minimized.

Copy link
Member

cedricve commented Jul 22, 2017

Yes, but you'll need to adjust the IoVideo class to use FFMPEG instead of OpenCV.

@muten84

This comment has been minimized.

Copy link
Member

muten84 commented Jul 22, 2017

i will give it a try using a command line pipe from live stream and ffmpeg encoding .... if it works the next step could be to use the pipe for writing images and then start ffmpeg encoding that reads from the just created pipe. I didn't remember how to use pipes or shared memory in c++ but i think this is possible

@douggb

This comment has been minimized.

Copy link

douggb commented Sep 13, 2017

If you need people to test, I have a Pi2, Pi3 and an OrangePi+2e. IP camera Hikvision 4mp. Trying to get 1080p at 15fps. Still learning Linux, so would need some guidance.

@cedricve

This comment has been minimized.

Copy link
Member

cedricve commented Nov 4, 2017

Update on hardware encoding

Started experimenting with modifying the OpenCV video writer class. Hardcoded the h264_omx codec. https://github.com/cedricve/opencv/blob/h264-preset/modules/videoio/src/cap_ffmpeg_impl.hpp#L1458-L1462. Encoding is successful on 800 x 600 and 1920x1080, however doesn't work on other resolutions (600x480 and 1280x720). Need to investigate why. @jasaw reported a similar issue in his PR for the motioneyeos project. ccrisan/motioneyeos#365

@cedricve

This comment has been minimized.

Copy link
Member

cedricve commented Nov 15, 2017

Successfully encoding with h264_omx codec, for USB and IP cameras, after upgrading to FFMPEG 3.4.3 and removing zero_copy with patch (ccrisan/motioneyeos#998). Thanks to @jasaw.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment