New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

is possible convert YUV surface to CanvasBitmap ? #198

Closed
PatrickSCLin opened this Issue Dec 8, 2015 · 15 comments

Comments

Projects
None yet
2 participants
@PatrickSCLin

PatrickSCLin commented Dec 8, 2015

I'm developing a video streaming app via Win2D and FFmpeg,

cus Win2D just work in D2D format, so I have to do sws_scale,

a worst case like a big resolution 4000 x 3000 h264 streaming

it's really too slow if convert it from yuv to bgra in FFmpeg

I just noticed a test "Direct3DSurfaceInteropTests",

so I don't know the detail, but is possible that I bind yuv data to a d3d surface, and convert yuv to bgra in GPU or even better if Win2D can draw the surface directly

@shawnhar

This comment has been minimized.

Show comment
Hide comment
@shawnhar

shawnhar Dec 8, 2015

Member

Win2D does not directly support YUV data formats. If you are willing to interop and include some native C++ code, you have several options:

  1. Use ID3D11VideoContext::VideoProcessorBlt to convert YUV surfaces to BGRA, prior to processing them with Win2D. This is a fast hardware accelerated conversion path.
  2. Use ID2D1DeviceContext2::CreateImageSourceFromDxgi to wrap an ID2D1ImageSource around your YUV surface(s), then interop the ID2D1ImageSource to a CanvasVirtualBitmap, which can be fed directly into any Win2D image processing operations. This is likely slightly more efficient than option 1, but requires Windows 10.
  3. Potentially the fastest but also by far the most work is to keep your Y data in a separate image from the UV, and process the two parts separately. This avoids ever having to expand the data into RGB format, which can allow UV processing to run at reduced resolution compared to the Y plane, but is only an option if your image manipulation algorithms allow the planes to be processed separately in this way (not an option if you are just using built-in effects like sepia or highlights and shadows). You can then use the YCbCr Effect (not exposed in Win2D, but part of native D2D) to recombine your Y and UV data into a single image, or (depending on what format you need the output in) just keep these as separate planar bitmaps.
Member

shawnhar commented Dec 8, 2015

Win2D does not directly support YUV data formats. If you are willing to interop and include some native C++ code, you have several options:

  1. Use ID3D11VideoContext::VideoProcessorBlt to convert YUV surfaces to BGRA, prior to processing them with Win2D. This is a fast hardware accelerated conversion path.
  2. Use ID2D1DeviceContext2::CreateImageSourceFromDxgi to wrap an ID2D1ImageSource around your YUV surface(s), then interop the ID2D1ImageSource to a CanvasVirtualBitmap, which can be fed directly into any Win2D image processing operations. This is likely slightly more efficient than option 1, but requires Windows 10.
  3. Potentially the fastest but also by far the most work is to keep your Y data in a separate image from the UV, and process the two parts separately. This avoids ever having to expand the data into RGB format, which can allow UV processing to run at reduced resolution compared to the Y plane, but is only an option if your image manipulation algorithms allow the planes to be processed separately in this way (not an option if you are just using built-in effects like sepia or highlights and shadows). You can then use the YCbCr Effect (not exposed in Win2D, but part of native D2D) to recombine your Y and UV data into a single image, or (depending on what format you need the output in) just keep these as separate planar bitmaps.

@shawnhar shawnhar added the question label Dec 8, 2015

@PatrickSCLin

This comment has been minimized.

Show comment
Hide comment
@PatrickSCLin

PatrickSCLin Dec 9, 2015

my application is windows 10 universal app, is option 2 more easier and better option for me ?

PatrickSCLin commented Dec 9, 2015

my application is windows 10 universal app, is option 2 more easier and better option for me ?

@shawnhar

This comment has been minimized.

Show comment
Hide comment
@shawnhar

shawnhar Dec 9, 2015

Member

Option 2 is probably the easiest, but these are all somewhat equivalent in that they require stepping outside of Win2D and writing some C++ D2D code to use alongside it.

Member

shawnhar commented Dec 9, 2015

Option 2 is probably the easiest, but these are all somewhat equivalent in that they require stepping outside of Win2D and writing some C++ D2D code to use alongside it.

@PatrickSCLin

This comment has been minimized.

Show comment
Hide comment
@PatrickSCLin

PatrickSCLin Dec 9, 2015

which D3D11_BIND_FLAG for this purpose ?

PatrickSCLin commented Dec 9, 2015

which D3D11_BIND_FLAG for this purpose ?

@shawnhar

This comment has been minimized.

Show comment
Hide comment
@shawnhar

shawnhar Dec 9, 2015

Member

I haven't actually tried this, but I'd expect these need to be shader resource (the same as to use a texture with D3D).

Member

shawnhar commented Dec 9, 2015

I haven't actually tried this, but I'd expect these need to be shader resource (the same as to use a texture with D3D).

@PatrickSCLin

This comment has been minimized.

Show comment
Hide comment
@PatrickSCLin

PatrickSCLin Dec 9, 2015

How to interop ID2D1ImageSource to CanvasVirtualBitmap?

I have already created ID2D1ImageSource but no idea how to do the last step.

PatrickSCLin commented Dec 9, 2015

How to interop ID2D1ImageSource to CanvasVirtualBitmap?

I have already created ID2D1ImageSource but no idea how to do the last step.

@PatrickSCLin

This comment has been minimized.

Show comment
Hide comment
@PatrickSCLin

PatrickSCLin commented Dec 9, 2015

I just commit the code on github https://github.com/PatrickSCLin/YUVSample

@shawnhar

This comment has been minimized.

Show comment
Hide comment
@shawnhar
Member

shawnhar commented Dec 9, 2015

@PatrickSCLin

This comment has been minimized.

Show comment
Hide comment
@PatrickSCLin

PatrickSCLin Dec 10, 2015

I just finish the interop, but the image which be drawn is not incorrect.

I guess the UV is wrong, I'm confused, since I have to interop Win2D & Direct2D

I should use the CanvasDrawSession and send it into C++ to do interop it to D2D1Context to do CreateImageSourceFromDxgi

or I should use the D2D1Device which I created in C++ and send it to C# , interop it to CanvasDevice ?

this sample is what I done for now, https://github.com/PatrickSCLin/YUVSample

maybe you give me some hint from the code

PatrickSCLin commented Dec 10, 2015

I just finish the interop, but the image which be drawn is not incorrect.

I guess the UV is wrong, I'm confused, since I have to interop Win2D & Direct2D

I should use the CanvasDrawSession and send it into C++ to do interop it to D2D1Context to do CreateImageSourceFromDxgi

or I should use the D2D1Device which I created in C++ and send it to C# , interop it to CanvasDevice ?

this sample is what I done for now, https://github.com/PatrickSCLin/YUVSample

maybe you give me some hint from the code

@PatrickSCLin

This comment has been minimized.

Show comment
Hide comment
@PatrickSCLin

PatrickSCLin Dec 10, 2015

I can draw YUV frame on win2d now, but ...

does option 2 only based on NV12 format ?

I mean, most h264 file is decoded to be yuv420p frame from FFmpeg,

but position of UV is kind different with NV12, can I use yuv420p format directly in option 2?

PatrickSCLin commented Dec 10, 2015

I can draw YUV frame on win2d now, but ...

does option 2 only based on NV12 format ?

I mean, most h264 file is decoded to be yuv420p frame from FFmpeg,

but position of UV is kind different with NV12, can I use yuv420p format directly in option 2?

@PatrickSCLin

This comment has been minimized.

Show comment
Hide comment
@PatrickSCLin

PatrickSCLin Dec 10, 2015

thanks for help :)

I just have to do a little function to swap UV by myself, and it display perfect.

PatrickSCLin commented Dec 10, 2015

thanks for help :)

I just have to do a little function to swap UV by myself, and it display perfect.

@shawnhar

This comment has been minimized.

Show comment
Hide comment
@shawnhar

shawnhar Dec 10, 2015

Member

Great to hear you got this working! How is the performance?

The incoming surfaces for CreateImageSourceFromDxgi don't have to be NV12. https://msdn.microsoft.com/en-us/library/windows/desktop/dn890791%28v=vs.85%29.aspx has a table documenting the various options - input can either be a single planar format texture, or a pair of textures containing separate Y and UV data.

Member

shawnhar commented Dec 10, 2015

Great to hear you got this working! How is the performance?

The incoming surfaces for CreateImageSourceFromDxgi don't have to be NV12. https://msdn.microsoft.com/en-us/library/windows/desktop/dn890791%28v=vs.85%29.aspx has a table documenting the various options - input can either be a single planar format texture, or a pair of textures containing separate Y and UV data.

@shawnhar shawnhar closed this Dec 10, 2015

@PatrickSCLin

This comment has been minimized.

Show comment
Hide comment
@PatrickSCLin

PatrickSCLin Dec 11, 2015

https://msdn.microsoft.com/en-us/library/windows/desktop/dn890791(v=vs.85).aspx

sorry, I'm trying to put yuv420p frame data directly to a CanvasVirturalBitmap by CreateImageSourceFromDxgi

since yuv420p is a planlar format, so in this case, I should create 3 textures for Y, U, V, all of them are R8 format, right ?

also I saw the document said

If multiple surfaces are provided, this method infers whether chroma planes are subsampled (by 2x) from the relative sizes of each corresponding source rectangle (or if the source rectangles parameter is NULL, the bounds of each surface). The second and third rectangle must each be equal in size to the first rectangle, or to the first rectangle with one or both dimensions scaled by 0.5 (while rounding up).
HRESULT CreateImageSourceFromDxgi(
  [in]                 IDXGISurface                        **surfaces,
  [in, optional] const D2D1_RECT_U                         *sourceRectangles,
                       UINT32                              surfaceCount,
                       DXGI_COLOR_SPACE_TYPE               colorSpace,
                       D2D1_IMAGE_SOURCE_FROM_DXGI_OPTIONS options,
  [out]                ID2D1ImageSource                    **imageSource
);


sourceRectangles [in, optional]
Type: const D2D1_RECT_U*
The regions of the surfaces to create the image source from.

the problem is, I can not see the argument sourceRectagles in my ID2D1DeviceContext2 interface

the function in my VisualStudio 2015 Update 1 is looks like this, and no other overload function:

HRESULT CreateImageSourceFromDxgi(
  [in]                 IDXGISurface                        **surfaces,
                       UINT32                              surfaceCount,
                       DXGI_COLOR_SPACE_TYPE               colorSpace,
                       D2D1_IMAGE_SOURCE_FROM_DXGI_OPTIONS options,
  [out]                ID2D1ImageSource                    **imageSource
);

what should I do ?

PatrickSCLin commented Dec 11, 2015

https://msdn.microsoft.com/en-us/library/windows/desktop/dn890791(v=vs.85).aspx

sorry, I'm trying to put yuv420p frame data directly to a CanvasVirturalBitmap by CreateImageSourceFromDxgi

since yuv420p is a planlar format, so in this case, I should create 3 textures for Y, U, V, all of them are R8 format, right ?

also I saw the document said

If multiple surfaces are provided, this method infers whether chroma planes are subsampled (by 2x) from the relative sizes of each corresponding source rectangle (or if the source rectangles parameter is NULL, the bounds of each surface). The second and third rectangle must each be equal in size to the first rectangle, or to the first rectangle with one or both dimensions scaled by 0.5 (while rounding up).
HRESULT CreateImageSourceFromDxgi(
  [in]                 IDXGISurface                        **surfaces,
  [in, optional] const D2D1_RECT_U                         *sourceRectangles,
                       UINT32                              surfaceCount,
                       DXGI_COLOR_SPACE_TYPE               colorSpace,
                       D2D1_IMAGE_SOURCE_FROM_DXGI_OPTIONS options,
  [out]                ID2D1ImageSource                    **imageSource
);


sourceRectangles [in, optional]
Type: const D2D1_RECT_U*
The regions of the surfaces to create the image source from.

the problem is, I can not see the argument sourceRectagles in my ID2D1DeviceContext2 interface

the function in my VisualStudio 2015 Update 1 is looks like this, and no other overload function:

HRESULT CreateImageSourceFromDxgi(
  [in]                 IDXGISurface                        **surfaces,
                       UINT32                              surfaceCount,
                       DXGI_COLOR_SPACE_TYPE               colorSpace,
                       D2D1_IMAGE_SOURCE_FROM_DXGI_OPTIONS options,
  [out]                ID2D1ImageSource                    **imageSource
);

what should I do ?

@PatrickSCLin

This comment has been minimized.

Show comment
Hide comment
@PatrickSCLin

PatrickSCLin Dec 14, 2015

the option3 only work when UV is packet format, right ?

PatrickSCLin commented Dec 14, 2015

the option3 only work when UV is packet format, right ?

@PatrickSCLin

This comment has been minimized.

Show comment
Hide comment
@PatrickSCLin

PatrickSCLin Dec 22, 2015

I'm using option2 for my project now, if the decoder runs well,

it should be fine to draw in 30fps, 4000 * 3000, on CPU i5 4460

thanks for help :)

PatrickSCLin commented Dec 22, 2015

I'm using option2 for my project now, if the decoder runs well,

it should be fine to draw in 30fps, 4000 * 3000, on CPU i5 4460

thanks for help :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment