Built-in virtual camera does not work with OpenCV

[upgradeQ]

Platform

Operating system and version: Windows 7 64 bit
OBS Studio version: 26.0.2 64 bit
Python version: Python 3.6 64 bit

Expected Behavior

Built-in virtual camera should be usable with OpenCV python package

Current Behavior

OpenCV window shows black screen if OBS Virtual Camera is selected.

Steps to Reproduce

1. Start virtual camera
2. Run check_cv.py

Additional information

Black screen

However OpenCV works with obs-virtual-cam plugin, if OBS-Camera is selected.

Code used to check it:

# check_cv.py

import numpy as np
import cv2 as cv

def check_cam_by_index(i):
    cap = cv.VideoCapture(i)
    cap.set(cv.CAP_PROP_FRAME_WIDTH, 1280)
    cap.set(cv.CAP_PROP_FRAME_HEIGHT, 720)

    if not cap.isOpened():
        print("Cannot open camera")
        exit()
    while True:
        # Capture frame-by-frame
        ret, frame = cap.read()
        # if frame is read correctly ret is True
        if not ret:
            print("Can't receive frame (stream end?). Exiting ...")
            break
        cv.imshow('frame',frame)
        if cv.waitKey(1) == ord('q'):
            break
    # When everything done, release the capture
    cap.release()
    cv.destroyAllWindows()


def check_multiple():
    for i in range(-1,10):
        try:
            print(f'checking camera #{i}')
            check_cam_by_index(i)
        except:
            continue

#check_multiple() # 2 is index of OBS Virtual Camera , 0 is index of OBS-Camera
check_cam_by_index(2) 

I've tried reinstalling OBS, virtual cam drivers, running it as admin - same result.

Also, built-in virtual webcam is accessible from imageio by index, but with high CPU usage.

[Fenrirthviti]

Sounds like OpenCV doesn't support NV12 format. Might need to check with their developers first.

[WizardCM]

Hi there, could you confirm if this is fixed in OBS 26.1.0 RC2? Thanks!

[upgradeQ]

No, it is still black.

[Unmoon]

Also seeing this issue, both laptop webcam and obs-virtualcam plugin work, but built-in virtual camera returns a black image with some glitches at the top.

Windows 10 Pro Build 19041 64bit
OBS 26.0.2 64bit
OBS 26.1 RC2
Python 3.7.6 64bit
- opencv-python 3.4.11.45
- numpy 1.19.3

import cv2

camera = cv2.VideoCapture(0)

while cv2.waitKey(1) != 27:
    _, image = camera.read()
    cv2.imshow("", image)

Built-in:

Plugin:

[TheGuy920]

@Unmoon , I too have experienced the same thing, lucky, after searching for a solution, the plugin option was not too bad

[devedse]

I'm experiencing the same issue. It would be nice if there's a bit more configurability for the built in virtual camera of OBS. E.g. specify the output format.

[sergorl]

Is somebody who find solution?

[letmaik]

Reported here: opencv/opencv#19746

[qinst64]

Sounds like OpenCV doesn't support NV12 format. Might need to check with their developers first.

Version 26.1.1 does not work either. Under setting-advanced, If I switch the video setting from NV12 to others like RGB, I420... no luck though

[mkoeda]

This thread may help you.

How to get virtual Camera Data using opencv？ | OBS Forums https://obsproject.com/forum/threads/how-to-get-virtual-camera-data-using-opencv%EF%BC%9F.137113/

[bmox]

When I'm using obs virtual camera using opencv python I got an greay screen. Any solution ?

[AntumArk]

This thread may help you.

How to get virtual Camera Data using opencv？ | OBS Forums https://obsproject.com/forum/threads/how-to-get-virtual-camera-data-using-opencv%EF%BC%9F.137113/

It did help, though it would be great if the camera could be set up from console.

[DavraYoung]

Same problem, cannot read buildin obs virtual camera from python opencv 4.5.4, windows 11.
.
Tried to change VideoCapture backend api to DirectShow, still no luck(

[RazvanCristian]

I encountered the same problem using:

• Windows 10 19042.1415
• OBS 27.1.3 (64 bit)
• opencv 4.5.4.60

[aliencaocao]

Still same issue OBS 27.1.3 (64 bit) and opencv 4.5.5 on windows 10

[kitrak-rev]

Found any solutions?

[DavraYoung]

still waiting....

[Fenrirthviti]

This is almost definitely going to be something that needs to be reported to the OpenCV developers or whatever library is being used for them to provide support. I'd recommend that anyone having the issue reach out to them, and if there is any new information that points to a bug on our end (which it doesn't seem to be at present, but more that the libraries don't support the formats we use for our virtual cam), we can reopen this.

[ManuGraiph]

Any news on this?

[upgradeQ]

I've stumbled upon this project its uses FFMPEG on udp socket to transfer mjpeg stream over local network,thanks to this helpful SO answer the code goes as follows:

import cv2
cap = cv2.VideoCapture('udp://127.0.0.1:10800',cv2.CAP_FFMPEG)
if not cap.isOpened():
    print('VideoCapture not opened')
    exit(-1)

while True:
    ret, frame = cap.read()

    if not ret:
        print('frame empty')
        break

    cv2.imshow('image', frame)

    if cv2.waitKey(1)&0XFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Click yes to allow local access to this port 10800.

FFMPEG string, qscale 0 - 31 to control quality.

.\ffmpeg.exe -y -f dshow -thread_queue_size 4096 -hwaccel cuda -hwaccel_output_format cuda -i video="OBS Virtual Camera" -f rawvideo -c:v mjpeg -qscale:v 0 -r 120 udp://127.0.0.1:10800

I've tested it on 1280x720 with 120 FPS, there is like .1 sec delay

P.S.

There will big update (28.1+) in OBS Studio later this year
As of July 31 this related pull request(game bar integration with texture sharing IPC) has been set to 28.1 or 29 version by @gxalpha, though works on in-game overlay has been stalled for unknown time.

[jasperan]

Has this issue been fixed?

[Fenrirthviti]

As far as I know, nobody has reached out to OpenCV to report the issue. If you're looking for movement, I would suggest reaching out to them for them to take a look at why they don't support our virtual device.

[Avasam]

As far as I know, nobody has reached out to OpenCV to report the issue. If you're looking for movement, I would suggest reaching out to them for them to take a look at why they don't support our virtual device.

#3635 (comment)

opencv/opencv#19746 It's been tagged low priority, you can go upvote it.

[jasperan]

I can't stress the lengths which I've crossed trying to work around this bug / low priority feature.

If anyone is interested, I tried both Windows 10 & 11, and Windows 11 works for some reason, Windows 10 instead gave me very painful encoding issues with the virtualized camera inputs

12 days have passed since New Years and I feel like I aged a year

[ByPikod]

Still same problem. Also my computer slowing down while viewing obs virtualcam output.

[Avasam]

opencv/opencv#19746 (comment)

Faulting module name: obs-virtualcam-module64.dll, version: 29.0.0.0, time stamp: 0x63b8a352

This is not a part of OpenCV actually:

obs-virtualcam-module64.dll

[...]

Just to clarify, OBS built-in virtual camera uses DirectShow API for virtual camera (CAP_DSHOW), which is deprecated in OpenCV in favor of Microsoft Media Foundation (CAP_MSMF) backend (about 4+ years ago, before OpenCV 4.0). So, due to deprecation there are no plans to significantly rework DSHOW backend. Also, due to deprecation "acceptable fix" is a 2-5 lines disabler of this unsupported case (but this can't satisfy OBS users) to avoid breaking of existed code.

Perhaps, problem is that SetFormat() doesn't fail on this line (hr=0 / S_OK) during request of RGB24 format: 4.7.0/modules/videoio/src/cap_dshow.cpp#L2618 (blame)

After that in debug mode these errors are reported: ERROR: SampleCB() - buffer sizes do not match (this is a case if format is not changed properly)

Forcing "native" resolution of OBS camera crashes the process (by default OpenCV DSHOW backend requests 640x480):

    capture.set(CAP_PROP_FRAME_WIDTH, 1920);
    capture.set(CAP_PROP_FRAME_HEIGHT, 1080);

WinDbgX shows this stacktrace of the crash (from non-main thread):

[0x0]   obs_virtualcam_module64!memcpy_repmovs + 0xe   
[0x1]   obs_virtualcam_module64!VCamFilter::Frame + 0x1b5   
[0x2]   obs_virtualcam_module64!VCamFilter::Thread + 0x24b   
[0x3]   obs_virtualcam_module64!std::thread::_Invoke<std::tuple<<lambda_4e6f779957cbddf8fba6c1db701dac17> >,0> + 0x11   
[0x4]   obs_virtualcam_module64!thread_start<unsigned int (__cdecl*)(void *),1> + 0x5a   
[0x5]   KERNEL32!BaseThreadInitThunk + 0x14   
[0x6]   ntdll!RtlUserThreadStart + 0x21   

with r8d "size" register value: 0x00000000007e9000 (or 8294400).

RGB24 frame size is 1920*1080*3 = 6220800 (this is requested) RGB32 frame buffer size is 1920*1080*4 = 8294400 (this value we see during the crash)

So, looks like format is not switched to RGB24 properly.

I would blame on this OBS code with IAMStreamConfig::SetFormat method implementation which ignores passed input and returns S_OK unconditionally:

obsproject/libdshowcapture@2fa2e48/source/output-filter.cpp#L346-L363

IAMStreamConfig::SetFormat documentation and expected behavior: learn.microsoft.com/en-us/windows/win32/api/strmif/nf-strmif-iamstreamconfig-setformat

[NicoDeLarge]

I had trouble with the built in virtual cam in OBS v28 as well and the only solution I found was downgrading to 25.0.8 and using OBS Virtualcam 2.0.5.

The root cause up to my analysis, is that the built-in virtual cam is hard coded to use NV12 as output format for the direct show filter, but my system (Windows 10 22H2) does not have an appropriate built in downstream DirectShow filter, that converts NV12 to some RGB format, which is required in my case for the renderer.

This can also be observed when using the graphedit tool from the windows SDK, when dragging the "Video Capture Sources" -> "OBS Virtual Camera" to the filter graph, right-clicking on its ouput pin and selecting "Render Pin". This prompts an error, that no appropriate graph could be build to render.

In contrast to the build in virtual camera, the OBS Virtualcam 2.0.5 uses YUY2 as video format. And for that configuration my DirectShow automatically finds a filter graph to convert to render the data.

[Avasam]

I had trouble with the built in virtual cam in OBS v28 as well and the only solution I found was downgrading to 25.0.8 [...]

If you don't wanna have to downgrade OBS, checkout https://github.com/Avasam/obs-virtual-cam/releases .
It is only a bandaid fix though, OBS needs to fix w/e is wrong with their DirectShow cam and/or support MediaFoundation.
Somehow the old plugin works (whatever it did differently), even after porting for Qt6.

@Fenrirthviti There's now 2 reports showing this issue might have to do, at least partially, with something improper on OBS' DirectShow side. It may also resolve itself if someone figures out implementing MediaFoundation support for OBS' virtual camera. Can it be re-opened? It would also help with the visibility of the issue.

[Fenrirthviti]

We would certainly welcome any PRs from someone who understands the changes necessary, or at least can explain clearly what the problem is and what we'd need to change. It will probably be some time before we investigate MF support, however.

A lot of the issue here is lack of resources and interest in the problem in general, just to be honest. As this doesn't affect a lot of users, and the issue isn't really obvious (and is probably far from simple) it's hard to prioritize it over more important bug fixes and feature updates when it comes to allocating the handful of developers who are full time on the project.

That said, I'm happy to reopen if others are interested in investigating this.

[Fenrirthviti]

As an additional note, we do (should) support YUY2 output: 18a73c9

[NicoDeLarge]

Thats good indeed. However, the VideoFormat is set to NV12 in the VCamFilter, which does not tickle down to OutputPin during construction in first place, but then NV12 is set as VideoFormat in VCamFilter constructor for the OutputPin. So finally OutputPin has a MediaFormat based on NV12.

And when it now comes to the media type negotation between the VCamFilter (or its OutputPin) and the downstream filter input pin, the Connect method of OutputPin only succeeds, if the input pin accepts NV12 during the ReceiveConnection call.

To overcome that, instead of trying just the NV12 based MediaFormat, OutputPin could iterate within the Connect method over its mtList, check if the requested pmt is in mtList and making the ReceiveConnection call with the matching MediaType.

If this succeeds, then I guess the SetVideoFormat should be called to update the OutputPin, so that this triggers.

[gottagofaster236]

FYI, opencv/opencv#23460 got merged. It probably should've used NV12 directly, though.

[RytoEX]

FYI, opencv/opencv#23460 got merged. It probably should've used NV12 directly, though.

Feel free to open another PR against OpenCV that does that, if you like. Given that the OpenCV Issue is closed, we will likely consider this Issue resolved and close it out as well.

[gottagofaster236]

FYI, opencv/opencv#23460 got merged. It probably should've used NV12 directly, though.

Feel free to open another PR against OpenCV that does that, if you like. Given that the OpenCV Issue is closed, we will likely consider this Issue resolved and close it out as well.

SetFormat() returning S_OK unconditionally (which does not comply to the DShow spec) still has to be fixed, that's the reason OpenCV broke in the first place.

[RytoEX]

FYI, opencv/opencv#23460 got merged. It probably should've used NV12 directly, though.

Feel free to open another PR against OpenCV that does that, if you like. Given that the OpenCV Issue is closed, we will likely consider this Issue resolved and close it out as well.

SetFormat() returning S_OK unconditionally (which does not comply to the DShow spec) still has to be fixed, that's the reason OpenCV broke in the first place.

Sure, I don't disagree. However, the scope of this Issue is "Built-in virtual camera does not work with OpenCV". That would seem to no longer the case once OpenCV 4.8.0 is released.

[gottagofaster236]

FYI, opencv/opencv#23460 got merged. It probably should've used NV12 directly, though.

Feel free to open another PR against OpenCV that does that, if you like. Given that the OpenCV Issue is closed, we will likely consider this Issue resolved and close it out as well.

opencv/opencv#23469 is merged now.
I guess that's about all that could be done from the OpenCV side of things.

[RytoEX]

Sounds like this is solved in OpenCV then.

[Avasam]

Once the next version of OpenCV-Python comes out I'll test their fix and report if it works for me.

Whilst I'm very glad that they did and accepted @gottagofaster236 's patch, I agree that OpenCV shouldn't have to special case the OBS Virtual Camera and that there's still an underlying issue that needs to be fixed here.

Assuming a working patch in OpenCV 4.8, it does make sense to close this issue given its scope and I'll be satisfied with that. As long as another is open for:

SetFormat() returning S_OK unconditionally (which does not comply to the DShow spec) still has to be fixed, that's the reason OpenCV broke in the first place.

[RytoEX]

Assuming a working patch in OpenCV 4.8, it does make sense to close this issue given its scope and I'll be satisfied with that. As long as another is open for:

SetFormat() returning S_OK unconditionally (which does not comply to the DShow spec) still has to be fixed, that's the reason OpenCV broke in the first place.

That is my stance, yes. This Issue as written has a resolution. Someone shouldn't have to wade through 40+ comments here to get to the information about SetFormat().

[DarkGamer209788]

I decided to use SplitCam instead, it works fine

[Avasam]

If you don't wanna have to downgrade OBS, checkout Avasam/obs-virtual-cam/releases . It is only a bandaid fix though, OBS needs to fix w/e is wrong with their DirectShow cam and/or support MediaFoundation. Somehow the old plugin works (whatever it did differently), even after porting for Qt6.

This workaround plugin is back to being broken since the 29.1 release.

Edit: Fixed again, it was due to the FFMPEG update :)

[Avasam]

Tested opencv-python using the latest build artefacts, looks like their patch works. Just need to wait for an official release now.

[agilebean]

Tested opencv-python using the latest build artefacts, looks like their patch works. Just need to wait for an official release now.

can you please specify which version of opencv you installed it?

[gottagofaster236]

OpenCV 4.8.0 with the fixes is out

[WizardCM]

Closing per the above. Feel free to comment with confirmation or disagreement based on experience. Thanks!

[Fenrirthviti]

As a note and final comment here, this issue is closed as the stated report of not working with OpenCV has been resolved by OpenCV.

However, our issue of not using SetFormat() properly is still ongoing, should anyone wish to open a new issue on that for tracking. It hasn't been fixed on our end just due to lack of time and resources for someone to take a look at it. PRs are, as always, welcome for fixes there.

[vietdoo]

OpenCV 4.8.0 with the fixes is out
I've been stuck for over 2 hours trying to upgrade it using "pip install --upgrade opencv-python==4.8.1.78."

[Avasam]

I've been stuck for over 2 hours trying to upgrade it using "pip install --upgrade opencv-python==4.8.1.78."

@vietdoo If you have a problem installing a library, ask around on help forums for your language/framework (Python/pip), or raise an issue in the appropriate repository. https://github.com/opencv/opencv-python
There's nothing the OBS team can do about having issues installing opencv-python on your machine.