We discuss here how to use GPAC's compositor for complex scenes or simple AV as a filter rather than the player.
Simple AV scene: No compositor :)
Let's start with something simple before pulling out the hammer.
Assume you have a source (MP4, RTP, DASH, ...) and you want to output the raw decoded data of this source to something such as pipes , sockets , or raw data.
The simplest way to do this in GPAC is to use the reframer filter, enforcing it to use RAW data.
gpac -i $SOURCE reframer:raw @ -o dump.yuv gpac -i $SOURCE reframer:raw @ -o dump.pcm gpac -i $SOURCE reframer:raw @ -o dump.pcm @1 -o dump.yuv
@1 link argument indicates here that the following filter (file dump.yuv) only accepts input from the previous minus one declared filter, here reframer, as documented here.
If you need to perform say resampling and resizing:
gpac -i $SOURCE reframer:raw:FID=1 resample:sr=48k:ch=1:SID=1 @ -o dump.pcm ffsws:osize=256x256:SID=1 @ -o dump.yuv
You can obviously replace the output filters with any filters you like.
gpac -i $SOURCE reframer:raw @ pipe://videopipe @1 udp://127.0.0.1:1234/
Complex AV scene
Assume you want to insert some text and logo over a video. To do this with GPAC, you can:
- use a BIFS/BT/XMT scene
In this HowTo, we will only talk about using the compositor filter.
Design your scene, play it in MP4Client to see the result. You can get a lot of inspiration from the gpac BIFS tests.
In order to perform overlays, you need to use a Background2D node with a url
gpid:// cf this example. This will instruct the video to take the first visual PID connecting to the compositor and use it as source for this background node.
You can then compose the scene as follows:
gpac -i $SOURCE -i $BTSCENE compositor @ -o dump.pcm @1 -o dump.rgb
In this mode, the compositor will generate a new visual frame for each incoming video frame (keeping its timing info), update the scene time based on that frame time and then render animations and mix audio according to this clock.
This is a work in progress in GPAC, we might want to use the overlay facility with SVG or VRML/X3D as well (but these scene graphs don't have the convenient notion of 2D video background).
Discussion The compositor is quite flexible and provides many options. It can be instructed to enforce a given output pixel format or audio format. When using 3D scenes, the compositor will output OpenGL textures as data packets. When using 2D scenes, it will output (unless enforced otherwise) regular data packets, by default in the source video pixel format. If the filter chains allows it, the video processing is done in place on the video input frame; this requires no other filter use these frames obviously. If the output format is enforced, video data will be copied. Audio is send as PCM data using the compositor audio options.
This is very similar to overlaying, except you don't need to use any specific
gpid:// url. Just take your scene and dump it:
gpac -i $SCENE compositor @ -o dump.pcm @1 -o dump.rgb