OpenCV inputDevice.read() is blocking

[alex-ong]

It's blocking which is a double edged sword.

Two solutions:
We could set our main-loop sleep time to 0 if we know that capture is blocking.
I.e. CaptureMethod should expose IsBlocking()
That way we are guaranteed to process the frame as soon as its ready.

or, we wrap it asynchronously; but we can lose frames sometimes.
http://blog.blitzblit.com/2017/12/24/asynchronous-video-capture-in-python-with-opencv/

This goes along with the "should we make a frame buffer" idea.

[blakegong]

I've had 3 runs of https://github.com/gilbertfrancois/video-capture-async/blob/master/test/videocapturethreading.py (from your blog link), each resulted in 10x+ improvements. Are you freaking kidding me??

(video-capture-async-vP3tuEVX) ~/Code/blakegong/video-capture-async  master ✗         1 ⚠️
» python test/videocapturethreading.py
[i] Frames per second: 23.42, with_threading=False
.[i] Frames per second: 311.01, with_threading=True
.
----------------------------------------------------------------------
Ran 2 tests in 24.747s

OK

(video-capture-async-vP3tuEVX) ~/Code/blakegong/video-capture-async  master ✗
» python test/videocapturethreading.py
[i] Frames per second: 24.47, with_threading=False
.[i] Frames per second: 292.13, with_threading=True
.
----------------------------------------------------------------------
Ran 2 tests in 23.847s

OK

(video-capture-async-vP3tuEVX) ~/Code/blakegong/video-capture-async  master ✗
» python test/videocapturethreading.py
[i] Frames per second: 24.47, with_threading=False
.[i] Frames per second: 314.69, with_threading=True
.
----------------------------------------------------------------------
Ran 2 tests in 23.653s

OK

I'm gonna add this to OpenCV capture with one frame buffer for now. Soon.

[blakegong]

I think this sub 30 FPS explained why @timotheeg and my stream always appeared to be a bit laggy. The losing frame kind of laggy. Oh boy changing this is so overdue!

[alex-ong]

Yeah I noticed Tim's stream was definitely sub 30 fps.

I think a short buffer (3-5)frames should be more than sufficient?

As noted the opencv code was 22ms(majority is blocking) and the ocr code in fastest_strategy is like 1ms

[timotheeg]

Tim's stream was definitely sub 30 fps.

I think the recent stream you popped in was when I was trying to OCR and restream on live streams. The timing issue for that is quite bad indeed. I've been playing around on this a little (not very much yet), but getting good timing is hard. I tried reading much faster, but in such case, I reach the tip of the input stream and then it stutters. Sigh

When Blake and I both OCR locally from device and both data streams are sent to me or him for restreaming, the fps is not that bad at all. @blakegong we could have a game later to demo to Alex :)

[blakegong]

Actually when I looked back to those streams on Twitch, I can notice dropped frames 😅

I think that 25 FPS test result explains it. Remember with additional OCR, that 25 FPS would further dip a bit. So in the end, we might have been doing 20 FPS all this time @timotheeg. To me that sub 30 FPS is quite obvious...

Game is good! I will also just try to incorporate this threaded OpenCV capturing, just to see if this blocking capturing is really the issue.

[timotheeg]

Hmm, I don't know to which extent dropped frame can be an issue with upload to twitch or online replay 🤔

I can see some variability in frame rate, which maybe a bit a bit of buffering could help with, but I do not see dropped frames on my local setup. from my local view, the game is pretty much in sync with the gameplay.

I think I'll do a local recording capture and upload that somewhere to compare to a live stream..

I'm thinking I could also add a widget to the UI to render the fps there, so we can also see and review the value as it runs.

Looking forward to see the results of your experiments too :D

[alex-ong]

BUFFER_SIZE= 3
MAX_BUFFER= 5
The way i see it working is buffer BUFFER_SIZE frames on startup. Run capturing in a background thread. If it ever exceeds MAX_BUFFER, cull oldest frame. The worker will switch the "currently active frame" in its own thread.

In mainthread, you can call the worker at any frame rate. The worker will be in charge of which frame to present. If you call it too fast, it will just present the same frame. If you call it too slow, you lose frames. Ideally what you do is you can add an API for captureworker.frameNumber(), and just sleep until its changed. We can use this on OpenCV, but not Quartz/Win32.

[alex-ong]

I think that 25 FPS test result explains it. Remember with additional OCR, that 25 FPS would further dip a bit. So in the end, we might have been doing 20 FPS all this time @timotheeg. To me that sub 30 FPS is quite obvious...

When watching Yobi's stream in his normal stats setup, I could see it was not 30fps. It was definitely something ... less; i'd estimate say 20-23.

[timotheeg]

Yeah, but I don't think it's the capture and rerendering 🤔 . I think a combination of OBS, my wifi, and Twitch itself are to blame (at least partly), resulting in the viewers of the stream getting increasing delays 😞

I didn't really bother checking before, but now I think I should. I'll do some fps measurements on past streams and see what number I get, and I'll do a local fps measurement entirely locally, and I'll also do a local recording to see if OBS contributes to issues even locally.

[alex-ong]

Ahh, the jerkyness that i noticed might also be a result of it being a 30fps stream and you playing 18, which is 3f (at 60fps) gravity. So it takes 1.5f (at 30fps) per drop, so even if you had a perfect setup, it would drop 1 cell than 2, alternating. But i have seen other people's 30fps streams, and 18 does look choppy, but in a different less obvious way.

[blakegong]

I just checked. If I run main.py right now, it's quite obvious from those timestamps that it's sub 30 (sometimes there are 3 frames per 0.1 sec, but sometimes there are just 2).

Once I switch to the multi-threaded one, it's constantly hitting 30 FPS (as limited by code). The reasoning is obvious, as the multi-threaded one is unblocking. So cap.read() takes 0 ms.

@alex-ong sounds like 60 FPS might work out better? And suggest 50 FPS for PAL capturing, if anyone is using it for PAL?