v4l2loopback not providing correct timestamp on frames

[MarioMey]

I made a 3D scene (in Blender Game Engine) with two VideoTextures: one shows the webcam image, the other one, a v4l2loopback device capturing second-display screen, using gst-launch-1.0. No matter what command I use (*), when BlenderPlayer starts, it only shows the first frame of the v4l2loopback device. Webcam works well.

What I tested:

• Using VLC, webcam and any v4l2loopback works OK. I get smooth playback.
• Using Blender, webcam and another capture device (EasyCap), also I get smooth playback of both devices.
• Using two gst-launch-1.0 instances, both devices are displayed well in different windows, at the same time (I use "ximagesink" instead of "v4l2sink...")

(*) My command:

gst-launch-1.0 ximagesrc startx=1920 starty=0 endx=2943 endy=768 show-pointer=0 use-damage=0 \
! video/x-raw,framerate=30/1 \
! videoscale method=0 add-borders=false \
! video/x-raw,width=640,height=360 \
! videoflip method=horizontal-flip \
! v4l2sink device=/dev/video1

Also, I tried with the simplest one:
gst-launch videotestsrc ! v4l2sink device=/dev/video1

This is the blend file (I can't upload here, not even I compress to a ZIP)
https://dl.dropboxusercontent.com/u/13242430/95-video-texture-v4l2loopback.blend

To reproduce the "issue"

• Install v4l2loopback module
• Connect a webcam.
• Run the simple gst-launch command mentioned above.
• Open VLC and test both devices.
• Close VLC.
• Open the blend file in Blender (download from www.blender.org, last version).
• Start Game Engine.

What it should do
One object should show the webcam device and, the other, a video-test-pattern with motion.

I also reported this issue in Blender Development page: https://developer.blender.org/T48692

[MarioMey]

As I reported this in Blender Development page, Benoit Bolsee found that the problem, in Blender, is about caching with multi-thread. Also, he found some anomalies in v4l2loopback code. Anyone can see the bug thread in https://developer.blender.org/T48692, and I will paste the important information, here:

Benoit:

It appears that multi-thread caching doesn't work well with that particular v4l source.
I was able to get a smooth output by disabling caching.
I will analyze further and determine if I need to fix caching or disable it or make it selectable by API (caching is known to work with webcam source).

Benoit:

I did more tests and the problem is caused by the v4l2loopback not providing correct timestamp on frames: vt thinks that the frames are in the past and drops them.
There is a flag to fix this: m_isStreaming; this flags tells vt to ignore timestamp. Then caching works but not smoothly: there is a strange jitter in the stream as if the frames were not properly ordered. It seems that the loopback device doesn't implement a proper frame queue.
I'll investigate further but the best option apparently is to make caching optional by API.

Me:

As I reported this same bug in v4l2loopback issues github page, can I add this information that you are telling to me? Is there any other further information to post there?

Benoit:

Yes you can pass that information to them. They will probably find useful to look at the source code that's doing the caching. It's all in the VideoFFmpeg::cacheThread() function.
I noticed the following anomalies:

• av_read_frame() never returns < 0, which would mean 'no frame available at this time' no matter how frequently I call this function and despite the fact that the AVFMT_FLAG_NONBLOCK flag is set in the format context. This is abnormal, v4l2loopback should not return more frames than the target frame rate that is set in the configuration of the device.
• After decoding with avcodec_decode_video2(), the resulting AVPacket has a dts field set to a fixed value. This is abnormal, I expect dts to be a proper timestamp of the frame. It should advance as a real time clock.

[umlaeute]

could you check whether this is fixed in 0.10.0?

[MarioMey]

I tested the new version and tested the file: the issue is still there. Buuut....

I reinstalled the module, but I don't know if I did it well. I don't know if the one it is installed, is the new version. I do:

mario@circo3d:~/src/v4l2loopback-master$ modinfo /lib/modules/$(uname -r)/extra/v4l2loopback.ko
filename:       /lib/modules/3.16.0-77-generic/extra/v4l2loopback.ko
license:        GPL
author:         Vasily Levin, IOhannes m zmoelnig <zmoelnig@iem.at>,Stefan Diewald,Anton Novikovet al.
description:    V4L2 loopback video device
srcversion:     6E7A76B2B8585B54C6F812E
depends:        videodev
vermagic:       3.16.0-77-generic SMP mod_unload modversions 
parm:           debug:debugging level (higher values == more verbose) (int)
parm:           max_buffers:how many buffers should be allocated (int)
parm:           max_openers:how many users can open loopback device (int)
parm:           devices:how many devices should be created (int)
parm:           video_nr:video device numbers (-1=auto, 0=/dev/video0, etc.) (array of int)
parm:           card_label:card labels for every device (array of charp)
parm:           exclusive_caps:whether to announce OUTPUT/CAPTURE capabilities exclusively or not (array of bool)
parm:           max_width:maximum frame width (int)
parm:           max_height:maximum frame height (int)

But I don't find "0.10.0" anywhere. I'd like to see that number somewhere... where?

Now, as I reported above, in Blender there is an issue with "multi-thread caching", particulary with v4l2loopback device. So, If this issue continues and it is the main bug... then, no. The problem is still there.

I'm reporting this new v4l2loopback version in the other bug report. Maybe Benoit can tell us more information.

Thanks.

[umlaeute]

to find out the current version of the module, run dmesg | grep v4l2loopback (or journalctl | grep v4l2loopback, if you are using systemd), which should print something like:

v4l2loopback driver version 0.10.0 loaded

[MarioMey]

Well, yes. I installed well. And no, the problem with Blender is still there.

But remember that Blender had an issue with "multi-thread caching"... Benoit had sent me a patch that avoid this caching and it worked well. Maybe, now the problem is only in Blender.

Let's wait for Benoit feedback.

[panshengjie]

I got a similar problem
I was using ffmpeg -f video4linux2 -i /dev/video3 .... in which video3 was create by v4l2lookback 0.10 and gstreamer v4l2sink
it seems timestamp has something wrong, and ffmpeg shows:

DTS -1493973937735357, next:47099529 st:0 invalid dropping
PTS -1493973937735357, next:47099529 invalid dropping st:0

I am not sure if it is v4l2lookback's problem or the gstreamer stamped wrong

[MarioMey]

Hello, there. What happened with this issue? 0.10.0 didn't fix the problem...

[umlaeute]

i think one of the issues is that it is unclear who should actually provide the timestamp: the module or the producer? (I tend towards the producer, but I don't know whether there is a single producer out there that actually does it)

[umlaeute]

oh, i just noticed that there actually is code in place that generates timestamps if the producer doesn't provide one (since v4l2loopback@v0.4.0) so it seems that the problem is indeed with (de)queueing...

@panshengjie since your setup seems to be easier to reproduce: could you provide a full (minimal) set of commands (both producer and consumer) to trigger the behaviour.

[panshengjie]

@umlaeute
my setup : fetch pictures of PointGrey Camera from its SDK, using gstreamer and v4l2lookback to make a virtual v4l2 device /dev/video0 , then

ffmpeg -i /dev/video0 test.mp4

and the ffmpeg will report the error

DTS -1493973937735357, next:47099529 st:0 invalid dropping
PTS -1493973937735357, next:47099529 invalid dropping st:0

if you don't have a true camera, maybe you can try videotestsrc in gstreamer as a video source
hope it helps