Camera Image Overlay Demos

Table of contents

1. Target devices
2. Supported environments
3. Supported USB cameras
4. Overview
5. Software
6. How to compile demos
7. How to run demos
8. FAQ
9. Revision history

Target devices

• RZ/G2L Evaluation Board Kit.
• RZ/V2L Evaluation Board Kit.

Supported USB cameras

• Logitech C270 HD Webcam.
• Logitech C930e Business Webcam.
• Logitech BRIO Ultra HD Pro Business Webcam.
• Logitech C920 HD Pro Webcam (*).

(*) The first buffer from the camera is likely to be corrupted.
Note: Other cameras may also work. Please use at your own risks.

Supported environments

• VLP 3.0.2.
• VLP 3.0.2-update1.
• VLP 3.0.3.

Note: Other environments may also work. Please use at your own risks.

Software

• Camera capture: V4L2, uvcvideo driver (include changes from Renesas to support dmabuf).
• Text/graphics overlay: EGL, OpenGL ES (proprietary).
• H.264 encoding: OMX IL (proprietary).
• Display: Wayland/Weston.

Overview

The demos explain how to overlay raw video images with text and 2D shape (such as: rectangle). Finally, the images are either encoded to H.264, shown on Wayland/Weston Desktop, or streamed to other device.

Application	Description
h264-to-file	Capture video images from camera -> Overlay the images with text and rectangle -> Encode the images to H.264 data -> Write H.264 data to file.
raw-video-to-lcd	Capture video images from camera -> Overlay the images with text and rectangle -> Show the images on Wayland/Weston Desktop.
video-to-lcd-and-file	Same as h264-to-file and raw-video-to-lcd. The video images are both shown on Wayland/Weston Desktop and encoded to H.264 data which is written to file.
h264-to-rtsp	(To be implemented) Capture video images from camera -> Overlay the images with text and rectangle -> Encode the images to H.264 data -> Stream with RTSP protocol.

Note: Camera capture, text/graphics overlay, H.264 encoding, and Wayland/Weston display are hardware accelerated.

Demo h264-to-file

Demo raw-video-to-lcd

Demo video-to-lcd-and-file

For output videos, please refer to h264-to-file and raw-video-to-lcd demos.

Source code

Directory	File name	Summary
common/inc/cglm	*.h	The directory includes header files for cglm library which provides utils to help math operations to be fast and quick to write. In our case, we are going to use it to create orthographic projection matrix and translation/scale/rotation matrix.
common/inc, common/src	egl.h, egl.c	Contain functions that connect/disconnect EGL display, create EGL context, check EGL extensions, and create/delete EGLImage objects.
common/inc, common/src	gl.h, gl.c	Contain struct gl_res_t, RGB colors (BLACK, WHITE...), and functions that create shaders, check OpenGL ES extensions, create/delete YUYV and RGB textures, create/delete framebuffers, create/delete resources, convert YUYV textures, draw rectangle, and render text.
common/inc, common/src	mmngr.h, mmngr.c	Contain structs: mmngr_dmabuf_exp_t, mmngr_buf_t, and functions that allocate/free NV12 buffers.
common/inc, common/src	omx.h, omx.c	Contain macros that calculate stride, slice height from video resolution and functions that wait for OMX state, get/set input/output port, allocate/free buffers for input/output ports...
common/inc, common/src	queue.h, queue.c	Contain struct queue_t and functions that create/delete queue, check if queue is empty or full, enqueue/dequeue element to/from queue.
common/inc, common/src	ttf.h, ttf.c	Contain struct glyph_t and functions that generate/delete an array of glyph_t objects from TrueType font file.
common/inc, common/src	v4l2.h, v4l2.c	Contain struct v4l2_dmabuf_exp_t and functions that open/verify device, get/set format and framerate, allocate/free buffers, enqueue/dequeue buffers, enable/disable capturing...
common/inc, common/src	wl.h, wl.c	Contain structs: wl_display_t, wl_window_t, and functions that connect/disconnect Wayland display, create/delete window.
common/inc, common/src	util.h, util.c	Contain utility functions.
common/ttf	LiberationSans-Regular.ttf	TrueType font.
h264-to-file	h264-to-file.sh, main.c	Demo h264-to-file. The script file will run the demo after reprobing uvcvideo with parameter allocators=1. Without it, video frames will contain noises.
h264-to-file	yuyv-to-rgb.vs.glsl, yuyv-to-rgb.fs.glsl	Convert YUYV textures to RGB.
h264-to-file	rectangle.vs.glsl, rectangle.fs.glsl	Draw rectangle on RGB texture.
h264-to-file	text.vs.glsl, text.fs.glsl	Draw text on RGB texture.
h264-to-file	rgb-to-nv12.vs.glsl, rgb-to-nv12.fs.glsl	Convert RGB textures to NV12.
raw-video-to-lcd	raw-video-to-lcd.sh, main.c	Demo raw-video-to-lcd.
raw-video-to-lcd	yuyv-to-rgb.vs.glsl, yuyv-to-rgb.fs.glsl	Convert YUYV textures to RGB.
raw-video-to-lcd	rectangle.vs.glsl, rectangle.fs.glsl	Draw rectangle on RGB texture.
raw-video-to-lcd	text.vs.glsl, text.fs.glsl	Draw text on RGB texture.
video-to-lcd-and-file	video-to-lcd-and-file.sh, main.c	Demo video-to-lcd-and-file. The script file will run the demo after reprobing uvcvideo with parameter allocators=1. Without it, video frames will contain noises.
video-to-lcd-and-file	yuyv-to-rgb.vs.glsl, yuyv-to-rgb.fs.glsl	Convert YUYV textures to RGB.
video-to-lcd-and-file	rectangle.vs.glsl, rectangle.fs.glsl	Draw rectangle on RGB texture.
video-to-lcd-and-file	text.vs.glsl, text.fs.glsl	Draw text on RGB texture.
video-to-lcd-and-file	render-rgb.vs.glsl, render-rgb.fs.glsl	Draw RGB texture on default framebuffer.
video-to-lcd-and-file	rgb-to-nv12.vs.glsl, rgb-to-nv12.fs.glsl	Convert RGB textures to NV12.

How to compile demos

Note 1: The SDK must be generated from either core-image-weston or core-image-qt.
Note 2: If using SDK v3.0.4 or older, please add the following to build/conf/local.conf. Then, generate the SDK.

DISTRO_FEATURES_NATIVESDK_append = " wayland"
DISTRO_FEATURES_NATIVESDK_remove = " x11"

• Source the environment setup script of SDK:

  user@ubuntu:~$ source /path/to/sdk/environment-setup-aarch64-poky-linux

• Go to directory demo-camera-image-overlay and run make command:

  user@ubuntu:~$ cd demo-camera-image-overlay
  user@ubuntu:~/demo-camera-image-overlay$ make

• After compilation, the demos should be in directories h264-to-file, raw-video-to-lcd, and video-to-lcd-and-file.

  h264-to-file/
  ├── h264-to-file.sh
  ├── main
  ├── rectangle.fs.glsl
  ├── rectangle.vs.glsl
  ├── rgb-to-nv12.fs.glsl
  ├── rgb-to-nv12.vs.glsl
  ├── text.fs.glsl
  ├── text.vs.glsl
  ├── yuyv-to-rgb.fs.glsl
  ├── yuyv-to-rgb.vs.glsl
  └── LiberationSans-Regular.ttf
  
  raw-video-to-lcd/
  ├── raw-video-to-lcd.sh
  ├── main
  ├── rectangle.fs.glsl
  ├── rectangle.vs.glsl
  ├── text.fs.glsl
  ├── text.vs.glsl
  ├── yuyv-to-rgb.fs.glsl
  ├── yuyv-to-rgb.vs.glsl
  └── LiberationSans-Regular.ttf
  
  video-to-lcd-and-file/
  ├── LiberationSans-Regular.ttf
  ├── main
  ├── rectangle.fs.glsl
  ├── rectangle.vs.glsl
  ├── render-rgb.fs.glsl
  ├── render-rgb.vs.glsl
  ├── rgb-to-nv12.fs.glsl
  ├── rgb-to-nv12.vs.glsl
  ├── text.fs.glsl
  ├── text.vs.glsl
  ├── video-to-lcd-and-file.sh
  ├── yuyv-to-rgb.fs.glsl
  └── yuyv-to-rgb.vs.glsl

How to run demos

h264-to-file

• After compilation, copy directory h264-to-file to directory /home/root/ of RZ/G2L or RZ/V2L board.
  Then, run the following commands:

  root@smarc-rzv2l:~# cd h264-to-file
  root@smarc-rzv2l:~/h264-to-file# chmod 755 h264-to-file.sh
  root@smarc-rzv2l:~/h264-to-file# chmod 755 main
  root@smarc-rzv2l:~/h264-to-file# ./h264-to-file.sh

• The script h264-to-file.sh should generate the below messages:

  root@smarc-rzv2l:~/h264-to-file# ./h264-to-file.sh
  Removing 'uvcvideo' from kernel
  Adding 'uvcvideo' to kernel
  Running sample application
  V4L2 device:
    Name: 'UVC Camera (046d:0825)'
    Bus: 'usb-11c70100.usb-1.2'
    Driver: 'uvcvideo (v5.10.158)'
  V4L2 format:
    Frame width (pixels): '640'
    Frame height (pixels): '480'
    Bytes per line: '1280'
    Frame size (bytes): '614400'
    Pixel format: 'YUYV'
    Scan type: 'Progressive'
  V4L2 framerate: '30.0'
  OMX media component's role: 'video_encoder.avc'
  OMX state: 'OMX_StateIdle'
  OMX state: 'OMX_StateExecuting'
  EmptyBufferDone exited
  FillBufferDone exited
  FillBufferDone exited
  EmptyBufferDone exited
  FillBufferDone exited
  ...
  ^CEmptyBufferDone exited
  FillBufferDone exited
  OMX event: 'End-of-Stream'
  FillBufferDone exited
  Thread 'thread_input' exited
  Thread 'thread_output' exited
  FillBufferDone exited
  OMX state: 'OMX_StateIdle'
  OMX state: 'OMX_StateLoaded'

• Press Ctrl-C to exit the demo. The output video will also be generated:

  root@smarc-rzv2l:~/h264-to-file# ls -l out-*
  -rw-r--r-- 1 root root 287961 Sep 20 11:08 out-h264-640x480.264

• You can open it with Media Classic Player on Windows (recommended), Videos application on Ubuntu, or GStreamer pipeline on VLP environment as below:

  gst-launch-1.0 filesrc location=out-h264-640x480.264 ! h264parse ! omxh264dec ! videorate ! video/x-raw, framerate=30/1 ! waylandsink

raw-video-to-lcd

• After compilation, copy directories raw-video-to-lcd to directory /home/root/ of RZ/G2L or RZ/V2L board.
  Then, run the following commands:

  root@smarc-rzv2l:~# cd raw-video-to-lcd
  root@smarc-rzv2l:~/raw-video-to-lcd# chmod 755 raw-video-to-lcd.sh
  root@smarc-rzv2l:~/raw-video-to-lcd# chmod 755 main
  root@smarc-rzv2l:~/raw-video-to-lcd# ./raw-video-to-lcd.sh

• The script raw-video-to-lcd.sh should generate the below messages:

  root@smarc-rzv2l:~/raw-video-to-lcd# ./raw-video-to-lcd.sh
  Removing 'uvcvideo' from kernel
  Adding 'uvcvideo' to kernel
  Running sample application
  V4L2 device:
    Name: 'UVC Camera (046d:0825)'
    Bus: 'usb-11c70100.usb-1.2'
    Driver: 'uvcvideo (v5.10.158)'
  V4L2 format:
    Frame width (pixels): '640'
    Frame height (pixels): '480'
    Bytes per line: '1280'
    Frame size (bytes): '614400'
    Pixel format: 'YUYV'
    Scan type: 'Progressive'
  V4L2 framerate: '30.0'
  149 frames in 5 seconds: 29.8 fps
  149 frames in 5 seconds: 29.8 fps
  ...

• The demo should be shown on Wayland/Weston desktop. You can press Ctrl-C to exit the demo.

video-to-lcd-and-file

• Please refer to h264-to-file and raw-video-to-lcd.

FAQ

• Q1: How to fix the below compilation error?

  user@ubuntu:~/demo-camera-image-overlay$ make
  ...
  wayland-scanner code /usr/share/wayland-protocols/stable/xdg-shell/xdg-shell.xml common/src/xdg-shell-protocol.c
  Could not open input file: No such file or directory
  Makefile:68: recipe for target 'common/src/xdg-shell-protocol.c' failed
  make: *** [common/src/xdg-shell-protocol.c] Error 1

• A1: The make command needs the wayland-scanner tool to generate the xdg-shell-protocol.c file from the xdg-shell.xml file. But the XML file doesn't exist, so the tool crashes and the compilation stops. To fix this, make sure you are using the SDK generated from core-image-weston or core-image-qt.

• Q2: How to fix "wayland-scanner: Command not found"?

  user@ubuntu:~/demo-camera-image-overlay$ make
  ...
  make: wayland-scanner: Command not found
  make: *** [Makefile:68: common/src/xdg-shell-protocol.c] Error 127

• A2: The make command needs the wayland-scanner tool. This is installed in the SDK if you modified the local.conf as mentioned in the instructions (added " wayland", removed " x11"). However, you can also just install the tool manually by using the command below.

  $ sudo apt-get install libwayland-dev

Revision history

Version	Date	Summary
1.0	Jan 05, 2023	Add demo h264-to-file. However, it only encodes 1 frame and outputs 1 H.264 frame. It also outputs 3 raw frame for debugging purposes.
1.1	Feb 23, 2023	Demo h264-to-file is able to encode multiple frames and outputs 1 H.264 video.
1.2	Mar 14, 2023	Convert demo h264-to-file to multi-threaded application for better performance.
2.0	Apr 06, 2023	Add demo raw-video-to-lcd.
2.1	Apr 26, 2023	Render text and transform rectangle.
3.0	Jul 31, 2023	Add demo video-to-lcd-and-file.