Layered Vision

Goals of this project is:

• A tool to allow the composition of images or videos via a configuration file (e.g. as a virtual webcam).

This project is still very much experimental and may change significantly.

Install

Install with all dependencies:

pip install layered-vision[all]

Install with minimal dependencies:

pip install layered-vision

Extras are provided to make it easier to provide or exclude dependencies when using this project as a library:

extra name	description
bodypix	For bodypix filter
webcam	Virtual Webcam support via pyfakewebcam
youtube	YouTube support via pafy and youtube_dl
mediapipe	Selfie Segmentation using MediaPipe.
all	All of the libraries

Virtual Webcam For Linux

You do not need to use a webcam to use the project, as you could feed a video file. But if you do want to use a webcam (currently only supported on Linux), this section provides a bit more information.

On Linux, /dev/video0 often refers to the true webcam device.

You can use v4l2loopback to create a virtual webcam device. e.g. you could /dev/video2.

Most applications looking for a webcam should then be able to use that virtual device. (Applications might include Chromium, Skype etc.)

Once installed, you can create /dev/video2 via the following command:

modprobe v4l2loopback devices=1 video_nr=2 exclusive_caps=1 card_label="VirtualCam 1"

To create the device after every reboot, you might want to create /etc/modprobe.d/v4l2loopback.conf:

options v4l2loopback devices=1 video_nr=2 exclusive_caps=1 card_label="VirtualCam 1"

Configuration

The configuration format is file is YAML.

There are a number of example configuration files.

Layers

Every configuration file will contain layers. Layers are generally described from top to down. With the last layer usually being the output layer.

The source to the output layer will be the layer above.

A very simple configuration file that downloads the numpy logo and saves it to a file might look like (example-config/save-image.yml):

layers:
- id: in
  input_path: "https://raw.githubusercontent.com/numpy/numpy/v1.20.1/branding/logo/logomark/numpylogoicon.png"
- id: out
  output_path: "numpy-logo.png"

You could also have two outputs (example-config/two-outputs.yml):

layers:
  - id: in
    input_path: "https://raw.githubusercontent.com/numpy/numpy/v1.20.1/branding/logo/logomark/numpylogoicon.png"
  - id: out1
    output_path: "data/numpy-logo1.png"
  - id: out2
    output_path: "data/numpy-logo2.png"

In that case the source layer for both out1 and out2 is in.

By using window as the output_path, the image is displayed in a window (example-config/display-image.yml):

layers:
  - id: in
    input_path: "https://raw.githubusercontent.com/numpy/numpy/v1.20.1/branding/logo/logomark/numpylogoicon.png"
    width: 480
    height: 300
    repeat: true
  - id: out
    output_path: window

Input Layer (Source)

A layer that has an input_path property.

The following inputs are currently supported:

type name	description
image	Static image (e.g. .png)
video	Video (e.g. .mp4)
webcam	Linux Webcam (/dev/videoN)
fill	Fills a new image with a color
youtube	YouTube stream (e.g. https://youtu.be/f0cGgOv3l4c, see example config)
mss	Screen capture using mss (see example config)

The input_path may point to a remote location (as is the case with all examples). In that case it will be downloaded and cached locally.

In most cases the type name is inferred from the input_path. You can also specify the type explicitly via the type property or by prefixing the path, e.g.: webcam:/dev/video0.

Output Layer (Sink)

A layer that has an output_path property.

The following outputs are currently supported:

type name	description
image_writer	Write to a static image (e.g. .png)
v4l2	Linux Virtual Webcam (/dev/videoN)
window	Display a window
web	Provide output as JPEG stream

As is the case with the input_path, in most cases the type name is inferred from the output_path. You can also specify the type explicitly via the type property or by prefixing the path, e.g.: v4l2:/dev/video2.

Web Stream (Experimental)

The output may also be provide as a JPEG stream. That way it can be viewed in a browser.

The following configuration options are supported:

name	default value	description
host	0.0.0.0	Host to listen to, 0.0.0.0 to listen on any host. This could also be set to say 127.0.0.1 to prevent the stream from being accessed from another machine.
port	8280	The port to listen to.

With the default configuration, opening http://localhost:8280 will provide a link to the stream. The stream will contineously provide JPEG frames to the browser (as a single request).

Currently only one stream consumer is supported.

Filter Layer

A layer that has a filter property.

The following filters are currently supported:

name	description
box_blur	Blurs the image or channel.
bodypix	Uses the bodypix model to mask a person.
chroma_key	Uses a chroma key (colour) to add a mask
copy	Copies the input. Mainly useful as a placeholder layer with branches.
dilate	Dilates the image or channel. For example to increase the alpha mask after using erode
erode	Erodes the image or channel. That could be useful to remove outliers from an alpha mask.
bilateral	Applies a bilateral filter, using d, sigma_color and sigma_space parameters.
motion_blur	Adds a motion blur to the image or channel. That could be used to make an alpha mask move more slowly
mp_selfie_segmentation	Uses the MediaPipe's Selfie Segmentation to mask a person (similar to bodypix).
pixelate	Pixelates the input.
fill	Fills the input or a selected channel with a color / value. e.g. with color set to blue
invert	Inverts the input. e.g. black to white
multiply	Multiplies the input with a constant value. e.g. to adjust the alpha channel
warp_perspective	Warps the perspective of the input image given a list of target_points. e.g. to display it in a corner of the output image

Every filter may have additional properties. Please refer to the examples (or come back in the future) for more detailed information. In particular display-video-segmentation-replace-background-template.yml provides examples of most filters (often disabled by default).

Filter: mp_selfie_segmentation (Experimental)

MediaPipe's Selfie Segmentation allows background segmentation (similar to bodypix). As it is more optimized, it will usually be faster than using bodypix.

The following parameters are supported:

name	default value	description
model_selection	1	The model to use, 0 or 1 (please refer MediaPipe's Selfie Segmentation documentation for further details).
threshold	0.1	The threshold for the segmentation mask.
cache_model_result_secs	0.0	The number of seconds to cache the mask for.

python -m layered_vision start \
  --config-file \
  "example-config/display-video-segmentation-replace-background-template.yml" \
  --set "bodypix.enabled=false" \
  --set "mp_selfie_segmentation.enabled=true"

Branches Layer

A layer that has an branches property. Each branch is required to have a layers property. The input to each set of branch layers is the input to the branches layer. The branches are then combined (added on top of each other). To make branches useful, at least the last branch image should have an alpha mask.

Error Handling

By default, any error such as an invalid path or filter parameter, will result in an exception being thrown, causing the application to exit.

To instead display an image, you could define an input layer with the id on_error:

layers:
  - id: on_error
    # Source: https://pixabay.com/vectors/test-pattern-tv-tv-test-pattern-152459/
    input_path: "https://www.dropbox.com/s/29ycjg9ubht776y/test-pattern-152459_640.png?dl=1"
    repeat: true
  # ...

CLI

CLI Help

python -m layered_vision --help

or

python -m layered_vision <sub command> --help

Example Command

python -m layered_vision start --config-file=example-config/display-image.yml

You could also load the config from a remote location:

python -m layered_vision start --config-file \
  "https://raw.githubusercontent.com/de-code/layered-vision/develop/example-config/display-video-chroma-key-replace-background.yml"

It is also possible to override config values via command line arguments, e.g.:

python -m layered_vision start --config-file=example-config/display-image.yml \
    --set out.output_path=/path/to/output.png

You could also try replacing the background with a YouTube stream:

python -m layered_vision start \
  --config-file \
  "https://raw.githubusercontent.com/de-code/layered-vision/develop/example-config/webcam-bodypix-replace-background-to-v4l2loopback.yml" \
  --set bg.input_path="https://youtu.be/f0cGgOv3l4c" \
  --set bg.fps=30 \
  --set in.input_path="/dev/video0" \
  --set out.output_path="/dev/video2"

Note: you may need to specify the fps

If a local configuration file was specified, the application will attempt to reload it on change.

Docker Usage

You could also use the Docker image if you prefer. The entrypoint will by default delegate to the CLI, except for python or bash commands.

docker pull de4code/layered-vision

docker run --rm \
  --device /dev/video0 \
  --device /dev/video2 \
  de4code/layered-vision start \
  --config-file \
  "https://raw.githubusercontent.com/de-code/layered-vision/develop/example-config/webcam-bodypix-replace-background-to-v4l2loopback.yml" \
  --set bg.input_path="https://www.dropbox.com/s/4debg4lrgn5g36l/toy-train-3288425.mp4?dl=1" \
  --set in.input_path="/dev/video0" \
  --set out.output_path="/dev/video2"

(Background: Toy Train)

Acknowledgements

• virtual_webcam_background, a somewhat similar project (more focused on bodypix)
• OBS Studio, conceptually a source of inspiration. (with UI etc)