feat(win): new capture method, Windows.Graphics.Capture #2580

tez011 · 2024-05-26T05:03:35Z

Description

This adds a capture backend on Windows that uses the Windows.Graphics.Capture API. This is a more recent API that's capable of capturing the Xbox Game Bar, which the DXGI Desktop Duplication API seemingly cannot do.

This pull request depends on C++20 compiler support (for the Microsoft WinRT headers, which define this API) as well as Universal CRT runtime support. These have already been integrated into Sunshine in #2322 and #2323.

Screenshot

Issues Fixed or Closed

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Dependency update (updates to dependencies)
Documentation update (changes to documentation)
Repository update (changes to repository files, e.g. .github/...)

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have added or updated the in code docstring/documentation-blocks for new or existing methods/components

Branch Updates

LizardByte requires that branches be up-to-date before merging. This means that after any PR is merged, this branch
must be updated before it can be merged. You must also
Allow edits from maintainers.

I want maintainers to keep my branch updated

tez011 · 2024-05-26T05:04:11Z

Right now, this pull request does not contain changes to support Sunshine's service model.
Linking #2149 (comment) for context.

.github/workflows/CI.yml

src_assets/common/assets/web/configs/tabs/Advanced.vue

.codeql-prebuild-cpp-Windows.sh

codecov · 2024-05-26T16:00:53Z

Codecov Report

Attention: Patch coverage is 7.39300% with 238 lines in your changes missing coverage. Please review.

Project coverage is 6.86%. Comparing base (2d706d3) to head (f77eb15).
Report is 153 commits behind head on master.

Files	Patch %	Lines
src/platform/windows/display_wgc.cpp	4.63%	138 Missing and 6 partials ⚠️
src/platform/windows/display_base.cpp	12.69%	47 Missing and 8 partials ⚠️
src/platform/windows/display_vram.cpp	7.89%	33 Missing and 2 partials ⚠️
src/platform/windows/display_ram.cpp	20.00%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #2580      +/-   ##
=========================================
- Coverage    7.03%   6.86%   -0.17%     
=========================================
  Files          87      88       +1     
  Lines       17698   17899     +201     
  Branches     8407    8514     +107     
=========================================
- Hits         1245    1229      -16     
- Misses      13717   14012     +295     
+ Partials     2736    2658      -78

Flag	Coverage Δ
Linux	`5.35% <ø> (ø)`
Windows	`2.62% <7.39%> (+0.06%)`	⬆️
macOS-12	`8.05% <ø> (ø)`
macOS-13	`7.98% <ø> (+0.01%)`	⬆️
macOS-14	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
src/platform/windows/display.h	`7.14% <ø> (-2.58%)`	⬇️
src/platform/windows/display_ram.cpp	`0.99% <20.00%> (-0.02%)`	⬇️
src/platform/windows/display_vram.cpp	`1.87% <7.89%> (+0.28%)`	⬆️
src/platform/windows/display_base.cpp	`12.50% <12.69%> (+0.62%)`	⬆️
src/platform/windows/display_wgc.cpp	`4.63% <4.63%> (ø)`

... and 29 files with indirect coverage changes

ReenigneArcher · 2024-05-27T03:38:26Z

I've quickly tested this locally.

CPU usage is < 4% while streaming with either capture method.
RAM usage is < 5mb while streaming with either capture method.
Can't use the new capture method when running it as a service, which I suppose is expected for now.
Can start Sunshine from start menu shortcut.

#2320

Was there anything else broken when we tried to merge this before?

tez011 · 2024-05-27T15:52:28Z

I seem to remember a couple other things:

Configuration UI comes up no slower than usual (was previously broken due to regex)
HDR may still not work (I don't have hardware capable of testing that)

ReenigneArcher · 2024-05-27T15:53:55Z

@tez011

UI seems to be coming up quickly for me.

HDR may still not work (I don't have hardware capable of testing that)

This was just with the new capture method, correct? The old one is unaffected?

tez011 · 2024-05-27T15:56:48Z

HDR may still not work (I don't have hardware capable of testing that)

This was just with the new capture method, correct? The old one is unaffected?

Yes. I could never reproduce it, but I recall this discussion from the old pull request. #2149 (comment)

ReenigneArcher · 2024-05-27T15:58:19Z

Alright, this will be merged soon. Thank you.

- Uses MinGW UCRT64 build environment - Bumps Windows compilation standard to C++20 to support WinRT - Refactors Windows display code to be agnostic towards capture API - Change can be enabled by setting 'capture = wgc' in sunshine.conf

#2149

)

Kobain-Seo · 2024-06-20T05:31:38Z

When i enabled this option, sunshine fails to start.

Logs:

[2024-06-20 14:24:29.019908] [0x00002e0c] [info] Gamepad x360 is disabled due to gamepads.vigem-not-available
[2024-06-20 14:24:29.021147] [0x00002e0c] [info] Gamepad ds4 is disabled due to gamepads.vigem-not-available
[2024-06-20 14:24:29.022127] [0x00002e0c] [info] Gamepad x360 is disabled due to gamepads.vigem-not-available
[2024-06-20 14:24:29.023114] [0x00002e0c] [info] Gamepad ds4 is disabled due to gamepads.vigem-not-available
[capture] -- [wgc]
[encoder] -- [nvenc]
[qp] -- [0]
[nvenc_twopass] -- [full_res]
[fec_percentage] -- [0]
[controller] -- [disabled]
[native_pen_touch] -- [disabled]
[nvenc_vbv_increase] -- [40]
[2024-06-20 14:24:29.024813] [0x00002e0c] [info] Gamepad x360 is disabled due to gamepads.vigem-not-available
[2024-06-20 14:24:29.025803] [0x00002e0c] [info] Gamepad ds4 is disabled due to gamepads.vigem-not-available
[2024:06:20:14:24:29]: Info: Sunshine version: v2024.620.13611
[2024:06:20:14:24:29]: Info: nvprefs: Opened undo file from previous improper termination
[2024:06:20:14:24:29]: Info: nvprefs: Restored OGL_CPL_PREFER_DXPRESENT for base profile
[2024:06:20:14:24:29]: Info: nvprefs: Restored global profile settings from undo file - deleting the file
[2024:06:20:14:24:29]: Info: nvprefs: No need to modify application profile settings
[2024:06:20:14:24:29]: Info: nvprefs: Changed OGL_CPL_PREFER_DXPRESENT to OGL_CPL_PREFER_DXPRESENT_PREFER_ENABLED for base profile
[2024:06:20:14:24:29]: Info: Compiling shaders...
[2024:06:20:14:24:29]: Info: System tray created
[2024:06:20:14:24:29]: Info: Compiled shaders
[2024:06:20:14:24:29]: Warning: Gamepad x360 is disabled due to gamepads.vigem-not-available
[2024:06:20:14:24:29]: Warning: Gamepad ds4 is disabled due to gamepads.vigem-not-available
[2024:06:20:14:24:29]: Warning: No gamepad input is available
[2024:06:20:14:24:29]: Info: Trying encoder [nvenc]
[2024:06:20:14:24:29]: Info: ddprobe.exe [1] [] returned: 0x00000000
[2024:06:20:14:24:29]: Info: Set GPU preference: 1
[2024:06:20:14:24:29]: Info:
Device Description : NVIDIA GeForce RTX 3060
Device Vendor ID : 0x000010DE
Device Device ID : 0x00002504
Device Video Mem : 12115 MiB
Device Sys Mem : 0 MiB
Share Sys Mem : 8113 MiB
Feature Level : 0x0000B100
Capture size : 1920x1080
Offset : 0x0
Virtual Desktop : 1920x1080
[2024:06:20:14:24:29]: Info: Active GPU has HAGS enabled
[2024:06:20:14:24:29]: Info: Using realtime GPU priority
[2024:06:20:14:24:29]: Info:
Colorspace : DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709
Bits Per Color : 8
Red Primary : [0.639648,0.330078]
Green Primary : [0.299805,0.599609]
Blue Primary : [0.149414,0.0595703]
White Point : [0.3125,0.329102]
Min Luminance : 0.5 nits
Max Luminance : 270 nits
Max Full Luminance : 270 nits

ReenigneArcher · 2024-06-20T11:11:18Z

You have to disable the service and start Sunshine manually.

…rdByte#2580)" This reverts commit 287ac4c.

Kobain-Seo · 2024-07-05T14:46:20Z

You have to disable the service and start Sunshine manually.

I think it is not relevant to running manually. I tested it on Windows 10 22H2 and sunshine failed to run, and It worked on Windows 11 22H2. Is it normal it does not work on Windows 10 ?

tez011 · 2024-07-05T15:22:30Z

Yes, that’s expected. This API is quite new.

Kobain-Seo · 2024-07-10T13:34:47Z

after patched #2825, It works on Windows 10, but have to be run manually, not by service.

ReenigneArcher · 2024-07-10T13:45:36Z

after patched #2825, It works on Windows 10, but have to be run manually, not by service.

For now, that's expected.

PieroPontra · 2024-07-10T15:25:32Z

Latest version of Win11 but the new beta WinGraphicsCapture that I switched to manually in configs says it can't find any encoder or display attached. I already tried to manually enter names from dxci command but same results, any hints?

WXZhao7 · 2024-07-15T02:08:32Z

I noticed from #2149 (comment) that WGC should be able to capture the UAC secure desktop. I have tested several pre-release versions, including v2024.714.230159. Unfortunately, Moonlight lose control when the UAC window pops up.
Is the capture of UAC prompts by WGC still incomplete, or is it not planned for implementation?

roob0 · 2024-07-22T07:51:53Z

@PieroPontra You have to close the service and launch Sunshine from the exe, but so you can't stream the lockscreen. However, I created a simple solution for switching automatically from service to exe and from ddx wo wgc (and viceversa) here:
#2149

…rdByte#2580)" This reverts commit 287ac4c.

ReenigneArcher requested changes May 26, 2024

View reviewed changes

.github/workflows/CI.yml Show resolved Hide resolved

src_assets/common/assets/web/configs/tabs/Advanced.vue Outdated Show resolved Hide resolved

src_assets/common/assets/web/configs/tabs/Advanced.vue Outdated Show resolved Hide resolved

ReenigneArcher reviewed May 26, 2024

View reviewed changes

.codeql-prebuild-cpp-Windows.sh Show resolved Hide resolved

ReenigneArcher mentioned this pull request May 26, 2024

fix(codeql): use ucrt64 for msys2 LizardByte/.github#336

Merged

11 tasks

ReenigneArcher approved these changes May 27, 2024

View reviewed changes

tez011 added 12 commits May 27, 2024 13:10

Address some review comments and fix CI pipeline for Windows.

a815105

#2149

Don't wait for new frames unless no frame is currently present.

c358b5c

Add more verbose logging to WGC initializer function.

07f9487

Restore functionality after blank image change in #2171

70975df

avoid using std::regex on windows

da80b87

Add description of new capture method.

54f4cd9

add some docstrings

97058c0

advertise automatic detection of capture method, default to DDAPI

6e46695

restore capture method selector on windows Advanced panel

0d8b3e5

don't break c++20

3ba952e

update build script/doc for cppwinrt

f77eb15

ReenigneArcher enabled auto-merge (squash) May 27, 2024 17:10

ReenigneArcher merged commit 287ac4c into LizardByte:master May 27, 2024
47 checks passed

Hazer mentioned this pull request May 30, 2024

ci: improved changelog generator #2610

Closed

11 tasks

KuleRucket pushed a commit to KuleRucket/Sunshine that referenced this pull request Jun 6, 2024

feat(win): new capture method, Windows.Graphics.Capture (LizardByte#2580

80d79b1

)

Black-Seraph added a commit to DuoStream/Sunshine that referenced this pull request Jun 27, 2024

Revert "feat(win): new capture method, Windows.Graphics.Capture (Liza…

070c8a2

…rdByte#2580)" This reverts commit 287ac4c.

TheMourningDawn mentioned this pull request Aug 7, 2024

Audio device keeps resetting to Streaming Speakers #2923

Closed

2 tasks

Black-Seraph added a commit to DuoStream/Sunshine that referenced this pull request Oct 3, 2024

Revert "feat(win): new capture method, Windows.Graphics.Capture (Liza…

dc32a2c

…rdByte#2580)" This reverts commit 287ac4c.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(win): new capture method, Windows.Graphics.Capture #2580

feat(win): new capture method, Windows.Graphics.Capture #2580

tez011 commented May 26, 2024

tez011 commented May 26, 2024

codecov bot commented May 26, 2024 •

edited

Loading

ReenigneArcher commented May 27, 2024

tez011 commented May 27, 2024

ReenigneArcher commented May 27, 2024

tez011 commented May 27, 2024

ReenigneArcher commented May 27, 2024

Kobain-Seo commented Jun 20, 2024 •

edited

Loading

ReenigneArcher commented Jun 20, 2024

Kobain-Seo commented Jul 5, 2024

tez011 commented Jul 5, 2024

Kobain-Seo commented Jul 10, 2024

ReenigneArcher commented Jul 10, 2024

PieroPontra commented Jul 10, 2024

WXZhao7 commented Jul 15, 2024

roob0 commented Jul 22, 2024

feat(win): new capture method, Windows.Graphics.Capture #2580

feat(win): new capture method, Windows.Graphics.Capture #2580

Conversation

tez011 commented May 26, 2024

Description

Screenshot

Issues Fixed or Closed

Type of Change

Checklist

Branch Updates

tez011 commented May 26, 2024

codecov bot commented May 26, 2024 • edited Loading

Codecov Report

ReenigneArcher commented May 27, 2024

tez011 commented May 27, 2024

ReenigneArcher commented May 27, 2024

tez011 commented May 27, 2024

ReenigneArcher commented May 27, 2024

Kobain-Seo commented Jun 20, 2024 • edited Loading

ReenigneArcher commented Jun 20, 2024

Kobain-Seo commented Jul 5, 2024

tez011 commented Jul 5, 2024

Kobain-Seo commented Jul 10, 2024

ReenigneArcher commented Jul 10, 2024

PieroPontra commented Jul 10, 2024

WXZhao7 commented Jul 15, 2024

roob0 commented Jul 22, 2024

codecov bot commented May 26, 2024 •

edited

Loading

Kobain-Seo commented Jun 20, 2024 •

edited

Loading