ScreenCaptureforHWND - Is it possible to downscale the resolution of the capture frame?

[gileli121]

Hello,
Is it possible to downscale the resolution of the capture frame from the API used in the example?

I want to reduce the resolution for example from 4K to 1080P without damaging performances too much.

I will appreciate any example or best instructions on how to do it efficiently.

I need to resize the actual pixels and not the way I display it on the screen.
I want at the end to get fewer pixels - instead of 4k, get 1920x1080 pixels

Thanks!

[robmikh]

Getting a scaled down buffer directly from the API is not available at the moment, but it's something we're investigating. In the mean-time, you'll need to scale down the buffer yourself using D3D/D2D. This won't give you any perf back on generating the frame, but if your pipeline is resolution dependent you may get something back there.

If you use D2D, there's a variety of interpolation modes.

[gileli121]

Thanks.
Do you know or can give an example of how to do it on ID3D11Texture2D ?

[gileli121]

I will try to ask you better what example I want:

In file SimpleCapture.cpp in function SimpleCapture::OnFrameArrived
The relevant code is:

auto frame = sender.TryGetNextFrame();
if (frameContentSize.Width != m_lastSize.Width ||
			frameContentSize.Height != m_lastSize.Height)
        {
        .....
        .....
        }

auto frameSurface = GetDXGIInterfaceFromObject<ID3D11Texture2D>(frame.Surface());

.....
.....

m_d3dContext->CopyResource(backBuffer.get(), frameSurface.get());

What I want to do is this:
before line

m_d3dContext->CopyResource(backBuffer.get(), frameSurface.get());

I know that I get ID3D11Texture2D
What I want to do here is this:

1. Reduce the texture size to 1920x1080 or any other resolution I want
2. I will do here my own processing algorithm (does not matter what)
3. Rescale it back to the original resolution
4. Do

m_d3dContext->CopyResource(backBuffer.get(), frameSurface.get());

To be clear: This line (unlike in the original example) actually copies the resized texture (original resolution but not the original frame)

You don't need to be care about step 2.
Please let me know how to: downscale it to another resolution --> rescale it back to the original resolution. After that, all steps are like in the example. You copy the resized frame instead of the original frame.

Thanks!

[gileli121]

If you can send me a modified version of the code in

void SimpleCapture::OnFrameArrived(
    Direct3D11CaptureFramePool const& sender,
    winrt::Windows::Foundation::IInspectable const&)

Choose any resolution you want.. for example, you can write that the code will always resize it to 1024x768 and then resize it to the original frame resolution

I will be very thanks to you.

[gileli121]

One more important note: I want that the resize will be done natively and not using the CPU.
In Direct3D/DirectX. I don't want to "Map" the texture and modify the buffer via CPU.
I want to resize it natively.

I did not found any docs about resizing the ID3D11Texture2D texture directly in the device.

[robmikh]

You need to setup a render target and texture a quad with the buffer (using shaders, a sampler, etc). Then you can use that render target (Or a copy of it with CopyResource) in the rest of your pipeline. I would look at D3D tutorials for that, that's outside the scope of these samples.

[gileli121]

I found that it will not help.
The GPU that scanning the frame is not so fast and this is the bottleneck.
At best, I get about 35 FPS on 4K, and then I have done my own processing using CUDA and it reduced to about 28FPS.. I don't have a very good start.

Please add support in the API to reduce the resolution of the scan so that at the very beginning I will not get the bottleneck.

If you add this support when it is expected to release?

[robmikh]

Unfortunately I can't provide a timeline at this time. I'll let you know when things change.

[gileli121]

OK, thanks.
I am watching this issue for updates about this.

[gileli121]

Hi @robmikh
Any update about this request?
The idea to scale it down myself did not solve any performance issues because I found that the bottleneck is on the OS level.

I need that the API will able to capture lower resolution.

I need it to solve this issue:
WindowTop/WindowTop-App#19

[robmikh]

There is no update at this time.

[gileli121]

OK thanks for the response.
Do you have some idea when will be update about the request?

[robmikh]

No

[gileli121]

@robmikh
Seems that I got it wrong all the time.
I see that what make it slow is not the capturing process.
It is the draw process.

I did some test with your ScreenCaptureforHWND example.
I captured 4k window in 2 ways:

• First time when I reduced a lot the size of ScreenCaptureforHWND window. To something small:
  
• When I increased the size of ScreenCaptureforHWND window

In the case that the ScreenCaptureforHWND window was very big, I noticed very big slowdown.
Is there a way to fix the example so it will work fast also when you draw the captured frame on much more pixels?
This seems to be the main issue or at least big part of it..

I was able to prove it also in another way

[gileli121]

@robmikh
Seems that I was right.
I tried to figure it out..
The first thing that I was thought about is this line:

Windows.UI.Composition-Win32-Samples/cpp/ScreenCaptureforHWND/ScreenCaptureforHWND/d3dHelpers.h

Line 167 in a59e758

desc.Scaling = DXGI_SCALING_STRETCH;

I have 0 knowledge about all of this stuff but my common sense told me that the flag DXGI_SCALING_STRETCH may be the issue that causing the slowdown in the rendering process.

So I tried to change it to DXGI_SCALING_NONE but without success.
I used to get the error DXGI_ERROR_INVALID_CALL
But I did not give up.

My strategy was to look for any code on the internet that creating swap chain with flag DXGI_SCALING_NONE
Then I found this code via google search
https://github.com/HadrianFinch/ScreenSnipper/blob/c77ec1ea499febf313f55cf892f2d52be303780e/packages/robmikh.common.0.0.7-beta/include/robmikh.common/d3dHelpers.desktop.h#L31

Looks like it is your code from another example so I thought to use it somehow.
So I ended up using your CreateDXGISwapChainForWindow function because it using DXGI_SCALING_NONE.
This time the function succeeds but I still ended up with an error here:

Windows.UI.Composition-Win32-Samples/cpp/ScreenCaptureforHWND/ScreenCaptureforHWND/App.cpp

Line 69 in a59e758

auto surface = m_capture->CreateSurface(m_compositor);

Then my common sense told me that maybe I should just remove these lines:

    auto surface = m_capture->CreateSurface(m_compositor);
    m_brush.Surface(surface);

And it just worked!
And now I see that this API is working 100% fast.
All the time the issue was in the draw and not in the capturing