Convert frame with FFMPEG

[polhaghverdian]

Hi @robmikh

One question: do you know how i can access the raw image data in order to encode it with FFMPEG? I assume this has to be done as soon as a new frame has arrived. Which will be in the function "OnFrameArrived". Do i get it from the backbuffer? Please guide me.

void SimpleCapture::OnFrameArrived(winrt::Direct3D11CaptureFramePool const& sender, winrt::IInspectable const&)
{
    auto swapChainResizedToFrame = false;

    {
        auto frame = sender.TryGetNextFrame();
        swapChainResizedToFrame = TryResizeSwapChain(frame);

        winrt::com_ptr<ID3D11Texture2D> backBuffer;
        winrt::check_hresult(m_swapChain->GetBuffer(0, winrt::guid_of<ID3D11Texture2D>(), backBuffer.put_void()));
        auto surfaceTexture = GetDXGIInterfaceFromObject<ID3D11Texture2D>(frame.Surface());
        // copy surfaceTexture to backBuffer
        m_d3dContext->CopyResource(backBuffer.get(), surfaceTexture.get());
        
    }

    DXGI_PRESENT_PARAMETERS presentParameters{};
    m_swapChain->Present1(1, 0, &presentParameters);

    swapChainResizedToFrame = swapChainResizedToFrame || TryUpdatePixelFormat();

    if (swapChainResizedToFrame)
    {
        m_framePool.Recreate(m_device, m_pixelFormat, 2, m_lastSize);
    }
}

[robmikh]

The swap chain is only to display the contents of the frame, it's not part of capturing the screen. The texture is held in the Direct3D11CaptureFrame object in its Surface property. You can see that the sample gets the ID3D11Texture2D from the Surface property here:

auto surfaceTexture = GetDXGIInterfaceFromObject<ID3D11Texture2D>(frame.Surface());

From there, you need to copy the texture to a staging texture (use D3D11_USAGE_STAGING and D3D11_CPU_ACCESS_READ flags during texture creation) and then map the staging texture. You'll be able to access the bits after that.

[polhaghverdian]

Hello @robmikh

Sorry to bother. I have spent some time trying to figure it out based on your explanation. Can you look at the code see if am on the right track?

void SimpleCapture::OnFrameArrived(winrt::Direct3D11CaptureFramePool const& sender, winrt::IInspectable const&)
{
        auto frame = sender.TryGetNextFrame();

        winrt::com_ptr<ID3D11Texture2D> backBuffer;
        winrt::check_hresult(m_swapChain->GetBuffer(0, winrt::guid_of<ID3D11Texture2D>(), backBuffer.put_void()));
        auto surfaceTexture = GetDXGIInterfaceFromObject<ID3D11Texture2D>(frame.Surface());

        /* Prepare for accessing data bits */
        // Get the device context
        ID3D11Device* d3dDevice;
        surfaceTexture->GetDevice(&d3dDevice);
        ID3D11DeviceContext* d3dContext;
        d3dDevice->GetImmediateContext(&d3dContext);

        // map the texture
        D3D11_MAPPED_SUBRESOURCE mapInfo;
        mapInfo.RowPitch;
        HRESULT hr = d3dContext->Map(
            surfaceTexture.get(),
            0,  // Subresource
            D3D11_MAP_READ,
            0,  // MapFlags
            &mapInfo);


        D3D11_TEXTURE2D_DESC desc;
        surfaceTexture->GetDesc(&desc);

        D3D11_TEXTURE2D_DESC desc2;
        desc2.Width = desc.Width;
        desc2.Height = desc.Height;
        desc2.MipLevels = desc.MipLevels;
        desc2.ArraySize = desc.ArraySize;
        desc2.Format = desc.Format;
        desc2.SampleDesc = desc.SampleDesc;
        desc2.Usage = D3D11_USAGE_STAGING;
        desc2.BindFlags = 0;
        desc2.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
        desc2.MiscFlags = 0;


        ID3D11Texture2D* stagingTexture = NULL;
        hr = d3dDevice->CreateTexture2D(&desc2, nullptr, &stagingTexture);
        if (FAILED(hr)) {
            // throw std::invalid_argument("received negative value");
            std::cout << "Failed to create staging texture";
        }

        // copy the texture to a staging resource
        d3dContext->CopyResource(stagingTexture, surfaceTexture.get());

        // now, map the staging resource
        hr = d3dContext->Map(stagingTexture, 0, D3D11_MAP_READ, 0, &mapInfo);

        if (FAILED(hr)) {
            // throw std::invalid_argument("received negative value");
            std::cout << "Failed to map staging texture";
        }

        // Frame data is here??
        mapInfo.pData;

}

[robmikh]

You're on the right track 🙂

Some things to consider:

• If you're doing this for more than one frame, I would suggest creating and caching 1 staging texture to use for this purpose. Then, every time a frame comes in you copy the frame to the staging texture and map it. You will know the dimensions ahead of time, because they will match what you used to set up the Direct3D11CaptureFramePool and will only change when you call Recreate. That way you only have to create a new staging texture when/if you call Recreate.
• Make sure you call Unmap after you're done with the bits. I would suggest copying the bits if you need to pass this off to some other processing pipeline.
• Textures may occupy more video memory than what you might expect given its dimensions and pixel format. Pay attention to the RowPitch field of the D3D11_MAPPED_SUBRESOURCE struct. Copying out the bits would look something like this:

D3D11_MAPPED_SUBRESOURCE mapped = {};
winrt::check_hresult(d3dContext->Map(bitmap.get(), 0, D3D11_MAP_READ, 0, &mapped));

std::vector<byte> bits(desc.Width * desc.Height * bytesPerPixel, 0);
auto source = reinterpret_cast<byte*>(mapped.pData);
auto dest = bits.data();
for (auto i = 0; i < (int)desc.Height; i++)
{
    memcpy(dest, source, desc.Width * bytesPerPixel);

    source += mapped.RowPitch;
    dest += desc.Width * bytesPerPixel;
}
d3dContext->Unmap(bitmap.get(), 0);

[polhaghverdian]

Thanks alot! 🙂