Unable to save video with custom Gstreamer pipe

[dharmatarha]

Thank you for all the work you put into Psychtoolbox!

I reported the problem in the forum (link) but there was no response there and I think this might warrant opening an issue. I would be glad for any advice.

System:
Ubuntu 18.04.5 LTS
Intel Core i5-6500 CPU
Intel HD Graphics 530 (SKL GT2)
Gnome 3.28.2
GStreamer 1.14.5
Matlab 2017a
Psychtoolbox 3.0.16 - Flavor: Debian package - psychtoolbox-3 (3.0.16.20200326.dfsg1-1~nd18.04+1)

Gist:
When using any custom gst-launch-1.0 -style Gstreamer pipeline for capturebin with Screen('OpenVideoCapture'), I seemingly cannot record video from my cameras, only display it. A video file is opened with the given name, but its size remains 0 byte, no data is written to it. With default Psychtoolbox settings everything works as expected, the recording is saved out. Is this the intended behavior? Is there perhaps any command I would need to add for saving video in this case?

Originally, my aim was to display + record video from a simple webcam (Logitech c920). To use its best resolution and fps capabilities (with mjpg format), I tried to open it from Psychtoolbox with custom Gstreamer specifications (e.g. 'v4l2src device=/dev/video0 ! jpegdec ! videoconvert'). Soon I found that I cannot record video to file with any version of the pipeline. Tested on another camera (built-in laptop cam) as well, with the same outcome. Note that recording + display works as expected with gst-launch-1.0 (code later).

Minimal code to reproduce:
Insert the following lines to VideoRecordingDemo.m instead of line 227:

% Custom Gstreamer pipeline definition:
capturebinspec = 'v4l2src device=/dev/video0';
Screen('SetVideoCaptureParameter', -1, sprintf('SetNextCaptureBinSpec=%s', capturebinspec));
grabber = Screen('OpenVideoCapture', win, -9, [0 0 640 480], [], [], [], codec, withsound, [], 8);

When running VideoRecordingDemo with the custom Gstreamer pipe the video file remains 0 bytes.

With Screen('Preference', 'Verbosity', 6), the output of the demo with the custom pipeline is here: customVideoRecordingDemo_gstreamerPipe_output

For comparison, the output of running the "original" (only edited for same resolution) VideoRecordingDemo with the same arguments: customVideoRecordingDemo_default_output

The main difference is that with the custom Gstreamer pipe Psychtoolbox seemingly cannot query the device capabilities / frame info (?):

PTB-DEBUG: No frame info available after preroll.
PTB-DEBUG: No frame info for video source available after preroll.

Also, with the custom Gstreamer pipe but not with the default VideoRecordingDemo I see the following terminal output:

(MATLAB:10629): GStreamer-CRITICAL **: 13:53:33.716: gst_pad_get_peer: assertion 'GST_IS_PAD (pad)' failed

(MATLAB:10629): GStreamer-CRITICAL **: 13:53:33.716: gst_object_unref: assertion 'object != NULL' failed

(MATLAB:10629): GStreamer-CRITICAL **: 13:53:35.737: gst_pad_add_probe: assertion 'GST_IS_PAD (pad)' failed

(MATLAB:10629): GStreamer-CRITICAL **: 13:53:35.737: gst_object_unref: assertion 'object != NULL' failed

A minimal script to reproduce the problem and try different params under Linux is here: minExample.m

Things that I tried and did not work:

• Adding "queue", "videoconvert", etc. elements to the Gstreamer pipe, converting to different formats (jpeg to yuy2 or the other way around). E.g. these fail to save video:
  'v4l2src device=/dev/video0 ! jpegdec;
  'v4l2src device=/dev/video0 ! jpegdec ! videoconvert';
  'v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegdec ! videoconvert ! video/x-raw,width=1280,height=720,framerate=30/1,format=YUY2 ! videoconvert';
• Different recording flags supplied to Screen('OpenVideCapture'); avoiding live display by supplying waitForImage=4 for Screen('GetCapturedImage'); different resolutions and frame rates; different codec settings

gst-launch-1.0 code that works with cameras:

gst-launch-1.0 v4l2src device=/dev/video0 \
    ! jpegdec \
    ! video/x-raw,width=1280,height=720,framerate=30/1 \
    ! tee name=t t. \
    ! queue ! xvimagesink sync=false t. \
    ! queue ! videoconvert ! x264enc tune=zerolatency ! h264parse \
    ! matroskamux ! filesink location='test.mkv' sync=false

best,
adam

[dharmatarha]

Thank you Mario, the problem is indeed fixed as far as I can tell.