Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows Graphics Capture support #2208

Merged
merged 12 commits into from
Feb 23, 2020
Merged

Conversation

jpark37
Copy link
Collaborator

@jpark37 jpark37 commented Nov 28, 2019

Description

Users will now have the option of legacy window capture via BitBlt, or
Windows Graphics Capture, which is new to Windows 10.

There are two annoyances with the new capture method though. One is that
there is a bright, yellow border added to the original window (but not
the OBS view of it). The other is that the mouse cursor is always
captured, and we won't be able to capture without cursor until a later
version of Windows 10 is released.

It should also be noted that DPI scaling is now applied, which may
result in blurrier images caused by Windows rescaling.

Motivation and Context

Works with hardware-accelerated Chrome. UWP apps also work with the new method.

How Has This Been Tested?

Scenarios tested:

  • Show/Hide
  • Multiple WGC captures
  • Multiple BitBlt captures
  • Toggle between WGC and BitBlt settings
  • WGC / BitBlt capture mix
  • Can't select WGC on Windows 7 and Windows 10 1809, OBS handles WinRT module load failure gracefully.
  • Can't select if OpenGL renderer is used.
  • BitBlt fallback if window has WGC setting somehow
  • Handle graphics thread recreation if video settings are touched with multiple WGC sources active
  • Handle D3D device loss
  • Handle window resize
  • Handle window close and latch on reopen
  • Hardware-accelerated Chrome (works on WGC [BitBlt implementation may be bugged])
  • Netflix UWP app (works on WGC [BitBlt implementation may be bugged])

Using QPF markers for CPU measurement, WGC tends to be a bit more expensive than BitBlt and wider variance, maybe 200-800 us vs 300-500 us. The new PeekMessage loop is very cheap when empty, about 0.2 microseconds.

WGC appears to be a bit cheaper than BitBlt on GPU just looking at Task Manager percentage, but it's hard to tell with the noise.

Since this feature is optional, and there's not much I can do to optimize these OS functions, I'm not too concerned about performance.

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist:

  • My code has been run through clang-format.
  • I have read the contributing document.
  • My code is not on the master branch.
  • The code has been tested.
  • All commit messages are properly formatted and commits squashed where appropriate.

@WizardCM WizardCM added Enhancement Improvement to existing functionality Request for Comments More feedback & discussion is requested Seeking Testers Build artifacts on CI Windows Affects Windows Work In Progress labels Nov 28, 2019
@jpark37 jpark37 force-pushed the screen-capture branch 4 times, most recently from b31cdf3 to b7d37a8 Compare November 28, 2019 01:27
@jpark37
Copy link
Collaborator Author

jpark37 commented Nov 28, 2019

Need to switch Windows SDK over to 10.0.18362.0 for this PR to build.

@jpark37 jpark37 force-pushed the screen-capture branch 3 times, most recently from 8911843 to 3304271 Compare November 28, 2019 09:18
@derrod
Copy link
Member

derrod commented Nov 28, 2019

It would appear that the old image Azure CI is currently using does not contain the latest Windows 10 SDK. Could you try changing https://github.com/obsproject/obs-studio/blob/master/azure-pipelines.yml#L64 and https://github.com/obsproject/obs-studio/blob/master/azure-pipelines.yml#L64 to windows-2019?

@jpark37
Copy link
Collaborator Author

jpark37 commented Nov 28, 2019

The windows-2019 image does not appear to have VS 2017 installed.

@jpark37 jpark37 force-pushed the screen-capture branch 3 times, most recently from 026a8a4 to 2de958b Compare November 28, 2019 10:46
@derrod
Copy link
Member

derrod commented Nov 28, 2019

I noticed that the new capture method stops working when changing OBS' frame rate in the video settings. The yellow border is still shown but the frame within OBS stops updating.

@jpark37
Copy link
Collaborator Author

jpark37 commented Nov 28, 2019

I hadn't realized OBS could recreate the graphics thread behind the sources backs until now. Added shenanigans handling, so changing video settings should work now.

@jpark37 jpark37 force-pushed the screen-capture branch 2 times, most recently from c30c15a to 5aa675b Compare November 28, 2019 19:25
@jpark37
Copy link
Collaborator Author

jpark37 commented Nov 28, 2019

Obviously, testing can have a never-ending list of permutations, but I am satisfied that I more or less have the basics covered.

The only thing holding me back from switching this out of draft status is the build situation. At a minimum, we need VS 2017 + Windows 10 SDK 10.0.18362.0 to compile this PR, but maybe we should move up to VS 2019 if that makes things easier for Azure.

@Gol-D-Ace
Copy link
Member

Getting the SDK available and being used by VS2017 is probably more pain than just switching to VS2019 which includes the newer SDK version.

@jpark37 jpark37 force-pushed the screen-capture branch 2 times, most recently from 84c086d to e7d9a0e Compare November 29, 2019 06:43
@derrod
Copy link
Member

derrod commented Dec 1, 2019

Not sure if we're going to retire AppVeyor as well but for now we'd have to update it to the vs2019 image as well.

@jpark37
Copy link
Collaborator Author

jpark37 commented Dec 2, 2019

Also, can we bump up to CMake 3.16? They recently added support for precompiled headers, and it reduces compile time for this PR from 6 seconds to less than a second.

@jpark37 jpark37 force-pushed the screen-capture branch 2 times, most recently from 111a4c2 to 036aadb Compare February 10, 2020 07:43
@derrod
Copy link
Member

derrod commented Feb 14, 2020

An issue I noticed is that this capture method appears to disable the hardware cursor, meaning cusor movement will feel somewhat unresponsive (input lag). While not necessarily a big deal - on my 144 Hz monitor this is barely noticeable, on my 60 Hz one it is very apparent - I think that might be something for others to test and confirm.

@jp9000
Copy link
Member

jp9000 commented Feb 20, 2020

If the CI works then the SDK issue is fine for the record.

For the hardware cursor thing, that's very likely Microsoft's doing, not ours, we probably aren't in control of that.

@jpark37
Copy link
Collaborator Author

jpark37 commented Feb 21, 2020

There's a commit in this PR to switch CI over to VS 2019. Is that fine?

@jp9000
Copy link
Member

jp9000 commented Feb 21, 2020

Yea, that's fine.

@jpark37
Copy link
Collaborator Author

jpark37 commented Feb 21, 2020

K, I'll see in a few hours if there's something I can do about hardware cursor before I remove draft status.

@jpark37
Copy link
Collaborator Author

jpark37 commented Feb 21, 2020

I couldn't repro the hardware cursor issue; it's always active for me.

@jpark37 jpark37 marked this pull request as ready for review February 21, 2020 05:28
@jp9000
Copy link
Member

jp9000 commented Feb 21, 2020

Although I don't have any lag either, I really doubt that it's anything in the source implementation anyway, I would expect it to be due to something happening internally as part of the capture in microsoft code. Unless you have some cursor stuff in your code that you think would be the cause, but I doubt it.

It doesn't really matter too much though, because this probably shouldn't be treated like a complete replacement for normal window capture. In my view, this is more important to us to capture windows that have been evading normal window capture, such as browser windows, microsoft store apps, office, or programs that run on chromium/cef/electon.

@jp9000
Copy link
Member

jp9000 commented Feb 21, 2020

This is just about perfect. I'm probably going to add an "auto" option and make it the default for the capture method dropdown though, so it can automatically capture windows we'd previously been having problems with. I feel like it should only be a few hours to make. I'm considering modifying the text for the items in the drop-down as well so the function is a bit more apparent.

jpark37 and others added 12 commits February 22, 2020 21:02
Don't make someone else define IUnknown.
Complex external systems using the D3D11 device may need to perform
their own device loss handling, the upcoming Windows Graphics Capture
support for example.
Necessary for upcoming Windows Graphics Capture support.
This provides a chance to clean up resources on the graphics thread in
the hide callback, which Windows Graphics Capture requires.
The top-level window appears to be preferred for upcoming Windows
Graphics Capture support, at least for UWP apps.
Currently wraps Windows Graphics Capture API.
Users will now have the option of legacy window capture via BitBlt, or
Windows Graphics Capture, which is new to Windows 10.

There are two annoyances with the new capture method though. One is that
there is a bright, yellow border added to the original window (but not
the OBS view of it). The other is that the mouse cursor is always
captured, and we won't be able to capture without cursor until a later
version of Windows 10 is released.

It should also be noted that DPI scaling is now applied, which may
result in blurrier images caused by Windows rescaling.
Reduces compile time from six seconds to less than a second.
The recent graphics capture changes require CMake 3.16 or above to
function on Windows, so add a requirement check for 3.16 if on Windows.
The graphics capture changes require Windows SDK version 10.0.18362.0 or
above to function, so check for that in CMake.
Adds an "Automatic" method to the capture method property of window
capture.  This allows the ability for window capture to automatically
determine the best capture method to use.  Primarily, it prefers BitBlt
in most cases, but will use WGC when it detects the window is a browser,
microsoft office, or a UWP program.

This was implemented because the new capture method has a number of
undesired effects associated with it -- the issue of the capture border
that we can do nothing about, the fact that we can't control the capture
of the cursor, and the fact that Microsoft designed it to switch the
cursor away from hardware cursor mode when the capture is active (there
was absolutely no reason to do this because even OBS can capture the
hardware cursor with no issue).  Until we get a new version of this API
that doesn't absolutely blow, we're stuck preferring BitBlt instead.

But hey, at least people will be able to capture browser windows now.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement Improvement to existing functionality Request for Comments More feedback & discussion is requested Seeking Testers Build artifacts on CI Windows Affects Windows Work In Progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants