D3D11 device removed in MediaPlayer with GPU textures

[richard42]

Description

I have written a Window Media Source object (registered DLL for Media Foundation framework) which parses omnidirectional media files, decodes and composites them with CUDA, and delivers Direct3D 11 media samples stored on the GPU to the Media Foundation player chain. The entire purpose of all my work was to display video in VR using Unity, SteamVR, and AVProVideo.

My decoder software works perfectly as a stand-alone windows application (without Media Foundation), and it works perfectly as a Media Foundation Media Source object using the MFPlayer2 sample application from Microsoft. It even works most of the time with AVProVideo in Unity. But sometimes, something blows up with Direct3D in AVProVideo or Unity during video playback, and I get D3D11 Device Removed errors. This typically happens after 100 to 10,000 frames have been decoded and displayed. Of course when this happens the playback stops and Unity prints error messages about Load or Decode errors.

My Setup

• Unity version: 2020.3.38f1
• AVPro Video version: v2.0.9f1-trial or core
• Operating system version: Windows 10
• Device model: Quadro RTX 4000, Cuda 11.3.1_465.89
• Video specs (resolution, frame-rate, codec, file size): 10560x5280 (or 3840x2160)

To Reproduce

1. Install OMAF Media Source plugin
2. Open Unity sample application with AVProVideo and MediaPlayer asset
3. Click 'Browse', select OMAF file to play, watch inspector playback until failure

Prior Debugging Steps

I typically debug by starting the Unity project and attaching to the unity player executable with MSVC 2019.

I have been working on debugging this problem for several weeks and have made many observations. The core problem is that something is going wrong with the player code in D3D11. Everything is rolling along perfectly fine in CUDA-land until D3D blows up. The only piece of my code which does anything with the D3D11Device is during the GPU setup function when the playback first begins. I create a pool of D3D11 texture objects (bound to CUDA resources). During playback I just map and unmap these objects to use them in CUDA and hand off completed D3D11 textures via the Media Foundation API.

I even made a temporary change to my Media Source object such that it would only decode the first 100 frames and thereafter it would just hand off the same D3D11 textures from the pool over and over again (doing no decoding or CUDA kernel processing). When running like this, the problem still occurs.

Logs

I enabled the D3D11 debug layer using Microsoft's "d3dconfig" tool and ran the player to observe the failure, and here are 3 different test runs which show the log messages right before and right after the problem occurs and the player dies.

Compositor: starting new output big frame with ulFrameHevcPTS=35034930.
Compositor::recycleOutputFrame(188afe6bd90) with MF 10MHz timestamp=33700333. 3 frames still out.
Compositor::RunOutputThread(): Delivering completed frame 188afe6b1c0 with PTS 35034930. 0 frames assembling and 4 out.
Compositor: starting new output big frame with ulFrameHevcPTS=35368596.
Compositor::recycleOutputFrame(188afe69630) with MF 10MHz timestamp=34034000. 3 frames still out.
Compositor: starting new output big frame with ulFrameHevcPTS=35702262.
Compositor::RunOutputThread(): Delivering completed frame 188afe6d140 with PTS 35368596. 1 frames assembling and 4 out.
Compositor: starting new output big frame with ulFrameHevcPTS=36035928.
D3D11 ERROR: ID3D11DeviceContext::Draw: Current Primitive Topology value (0) is not valid. [ EXECUTION ERROR #365: DEVICE_DRAW_INVALID_PRIMITIVETOPOLOGY]
D3D11 ERROR: ID3D11DeviceContext::Draw: A Vertex Shader is always required when drawing, but none is currently bound. [ EXECUTION ERROR #341: DEVICE_DRAW_VERTEX_SHADER_NOT_SET]
D3D11 ERROR: ID3D11DeviceContext::Draw: Rasterization Unit is enabled (PixelShader is not NULL or Depth/Stencil test is enabled and RasterizedStream is not D3D11_SO_NO_RASTERIZED_STREAM) but position is not provided by the last shader before the Rasterization Unit. [ EXECUTION ERROR #362: DEVICE_DRAW_POSITION_NOT_PRESENT]
Compositor: starting new output big frame with ulFrameHevcPTS=36369594.
Compositor::recycleOutputFrame(188afe68a60) with MF 10MHz timestamp=34367666. 3 frames still out.
Compositor::recycleOutputFrame(188afe68e50) with MF 10MHz timestamp=34701333. 2 frames still out.
Compositor::recycleOutputFrame(188afe6b1c0) with MF 10MHz timestamp=35035000. 1 frames still out.
Compositor::recycleOutputFrame(188afe6d140) with MF 10MHz timestamp=35368666. 0 frames still out.
The thread 0x43f8 has exited with code 0 (0x0).
D3D11: Removing Device.
D3D11 ERROR: ID3D11Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung. As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered. The current Device Context was executing commands when the hang occurred. The application may want to respawn and fallback to less aggressive use of the display hardware). [ EXECUTION ERROR #378: DEVICE_REMOVAL_PROCESS_AT_FAULT]
Compositor::RunOutputThread() error: cuGraphicsUnmapResources() failed with status 999.

Compositor: starting new output big frame with ulFrameHevcPTS=3002994.
Compositor::RunOutputThread(): Delivering completed frame 1ded39cc7f0 with PTS 2669328. 1 frames assembling and 1 out.
Compositor: starting new output big frame with ulFrameHevcPTS=3336660.
Compositor::recycleOutputFrame(1ded39cc7f0) with MF 10MHz timestamp=2669333. 0 frames still out.
The thread 0x4a3c has exited with code 0 (0x0).
Compositor: starting new output big frame with ulFrameHevcPTS=3670326.
Compositor: starting new output big frame with ulFrameHevcPTS=4003992.
Compositor::RunOutputThread(): Delivering completed frame 1ded39cef50 with PTS 3002994. 3 frames assembling and 1 out.
Compositor::recycleOutputFrame(1ded39cef50) with MF 10MHz timestamp=3003000. 0 frames still out.
Compositor::RunOutputThread(): Delivering completed frame 1ded39cb830 with PTS 3336660. 2 frames assembling and 1 out.
The thread 0x3f60 has exited with code 0 (0x0).
Compositor::recycleOutputFrame(1ded39cb830) with MF 10MHz timestamp=3336666. 0 frames still out.
The thread 0x3d70 has exited with code 0 (0x0).
Compositor: starting new output big frame with ulFrameHevcPTS=4337658.
Compositor::RunOutputThread(): Delivering completed frame 1ded39cff10 with PTS 3670326. 2 frames assembling and 1 out.
D3D11 ERROR: ID3D11DeviceContext::Draw: A Vertex Shader is always required when drawing, but none is currently bound. [ EXECUTION ERROR #341: DEVICE_DRAW_VERTEX_SHADER_NOT_SET]
D3D11 ERROR: ID3D11DeviceContext::Draw: Rasterization Unit is enabled (PixelShader is not NULL or Depth/Stencil test is enabled and RasterizedStream is not D3D11_SO_NO_RASTERIZED_STREAM) but position is not provided by the last shader before the Rasterization Unit. [ EXECUTION ERROR #362: DEVICE_DRAW_POSITION_NOT_PRESENT]
Compositor: starting new output big frame with ulFrameHevcPTS=4671324.
Compositor::recycleOutputFrame(1ded39cff10) with MF 10MHz timestamp=3670333. 0 frames still out.
The thread 0x3e24 has exited with code 0 (0x0).
D3D11: Removing Device.
D3D11 ERROR: ID3D11Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung. As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered. The current Device Context was executing commands when the hang occurred. The application may want to respawn and fallback to less aggressive use of the display hardware). [ EXECUTION ERROR #378: DEVICE_REMOVAL_PROCESS_AT_FAULT]
Compositor::submitInputFrame() error: cuGraphicsUnmapResources() failed with status 999.

Compositor::RunOutputThread(): Delivering completed frame 135ac073f20 with PTS 7340652. 2 frames assembling and 1 out.
Compositor: starting new output big frame with ulFrameHevcPTS=8341650.
Compositor::recycleOutputFrame(135ac073f20) with MF 10MHz timestamp=7340666. 0 frames still out.
The thread 0x14e4 has exited with code 0 (0x0).
Compositor: starting new output big frame with ulFrameHevcPTS=8675316.
Compositor::RunOutputThread(): Delivering completed frame 13831ce6260 with PTS 7674318. 3 frames assembling and 1 out.
Compositor::recycleOutputFrame(13831ce6260) with MF 10MHz timestamp=7674333. 0 frames still out.
The thread 0x49b0 has exited with code 0 (0x0).
Compositor: starting new output big frame with ulFrameHevcPTS=9008982.
Compositor::RunOutputThread(): Delivering completed frame 135ac072780 with PTS 8007984. 3 frames assembling and 1 out.
Compositor::recycleOutputFrame(135ac072780) with MF 10MHz timestamp=8008000. 0 frames still out.
The thread 0xef0 has exited with code 0 (0x0).
Compositor: starting new output big frame with ulFrameHevcPTS=9342648.
Compositor: starting new output big frame with ulFrameHevcPTS=9676314.
Compositor::RunOutputThread(): Delivering completed frame 135ac076680 with PTS 8341650. 4 frames assembling and 1 out.
Compositor::recycleOutputFrame(135ac076680) with MF 10MHz timestamp=8341666. 0 frames still out.
The thread 0x694 has exited with code 0 (0x0).
Compositor::RunOutputThread(): Delivering completed frame 135ac070fe0 with PTS 8675316. 3 frames assembling and 1 out.
Compositor: starting new output big frame with ulFrameHevcPTS=10009980.
Compositor::recycleOutputFrame(135ac070fe0) with MF 10MHz timestamp=8675333. 0 frames still out.
The thread 0x2bb0 has exited with code 0 (0x0).
Compositor: starting new output big frame with ulFrameHevcPTS=10343646.
Compositor::RunOutputThread(): Delivering completed frame 135ac075ea0 with PTS 9008982. 4 frames assembling and 1 out.
D3D11 ERROR: ID3D11DeviceContext::Draw: Current Primitive Topology value (0) is not valid. [ EXECUTION ERROR #365: DEVICE_DRAW_INVALID_PRIMITIVETOPOLOGY]
D3D11 ERROR: ID3D11DeviceContext::Draw: A Vertex Shader is always required when drawing, but none is currently bound. [ EXECUTION ERROR #341: DEVICE_DRAW_VERTEX_SHADER_NOT_SET]
D3D11 ERROR: ID3D11DeviceContext::Draw: Rasterization Unit is enabled (PixelShader is not NULL or Depth/Stencil test is enabled and RasterizedStream is not D3D11_SO_NO_RASTERIZED_STREAM) but position is not provided by the last shader before the Rasterization Unit. [ EXECUTION ERROR #362: DEVICE_DRAW_POSITION_NOT_PRESENT]
Compositor::recycleOutputFrame(135ac075ea0) with MF 10MHz timestamp=9009000. 0 frames still out.
Compositor::RunOutputThread(): Delivering completed frame 135ac073b30 with PTS 9342648. 3 frames assembling and 1 out.
The thread 0x4b1c has exited with code 0 (0x0).
Compositor::recycleOutputFrame(135ac073b30) with MF 10MHz timestamp=9342666. 0 frames still out.
The thread 0xd84 has exited with code 0 (0x0).
D3D11: Removing Device.
D3D11 ERROR: ID3D11Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung. As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered. The current Device Context was executing commands when the hang occurred. The application may want to respawn and fallback to less aggressive use of the display hardware). [ EXECUTION ERROR #378: DEVICE_REMOVAL_PROCESS_AT_FAULT]
Compositor::RunOutputThread() error: cuGraphicsUnmapResources() failed with status 999.

The Compositor log messages are being printed by my Media Source code. From these logs, it appears that the first D3D errors which are reported are state problems with the D3D11 device context.

Please help! Is there some kind of multithreaded texture / device sharing problem caused by something that I'm doing wrong in my decoder? Or has my Media Source object triggered a bug in AVProVideo or Unity?

[Chris-RH]

Hi @richard42,

Firstly, I'd suggest updating to the latest version of AVPro, as v2.0.9f1 is 18 months old and there have been a lot of improvements since then. I'd like to know if you're still getting the same errors with this version.
Secondly, could you please provide a few more details as to what sort of video files you are playing (container, codec, framerate etc).
Thank you

[richard42]

Okay, I have updated to v2.6.2 and now it fails before any frames are processed at all. Here's the relevant debug log from MSVC 2019:

'Unity.exe' (Win32): Loaded 'C:\Program Files\Panasonic Corporation\VR-Player\OmafMediaSource.dll'. 
'Unity.exe' (Win32): Loaded 'C:\Windows\System32\evr.dll'. 
'Unity.exe' (Win32): Loaded 'C:\Program Files\Panasonic Corporation\VR-Player\libmp4v2.dll'. 
'Unity.exe' (Win32): Loaded 'C:\Windows\System32\nvcuda.dll'. 
'Unity.exe' (Win32): Loaded 'C:\Windows\System32\nvcuvid.dll'. 
'Unity.exe' (Win32): Loaded 'C:\Program Files\Panasonic Corporation\VR-Player\cudart64_110.dll'. Module was built without symbols.
'Unity.exe' (Win32): Loaded 'C:\Windows\System32\sfc.dll'. 
'Unity.exe' (Win32): Unloaded 'C:\Windows\System32\sfc.dll'
'Unity.exe' (Win32): Loaded 'C:\Windows\System32\sfc.dll'. 
'Unity.exe' (Win32): Loaded 'C:\Windows\System32\sfc_os.dll'. 
'Unity.exe' (Win32): Loaded 'C:\Windows\System32\DriverStore\FileRepository\nv_dispui.inf_amd64_45b6550fdc35add8\nvcuda64.dll'. 
'Unity.exe' (Win32): Loaded 'C:\Windows\System32\nvapi64.dll'. 
OmafParser::CreateVideoMediaType(): Video RGB32 10560x5280 framerate 30000/1001
'Unity.exe' (Win32): Loaded 'C:\Windows\System32\avrt.dll'. 
'Unity.exe' (Win32): Loaded 'C:\Windows\System32\mfsvr.dll'. 
'Unity.exe' (Win32): Loaded 'C:\Windows\System32\msvproc.dll'. 
The thread 0x348c has exited with code 0 (0x0).
D3D11 ERROR: ID3D11DeviceContext::DrawIndexed: Current Primitive Topology value (0) is not valid. [ EXECUTION ERROR #365: DEVICE_DRAW_INVALID_PRIMITIVETOPOLOGY]
D3D11 WARNING: ID3D11DeviceContext::DrawIndexed: The Pixel Shader expects a Render Target View bound to slot 0, but none is bound. This is OK, as writes of an unbound Render Target View are discarded. It is also possible the developer knows the data will not be used anyway. This is only a problem if the developer actually intended to bind a Render Target View here. [ EXECUTION WARNING #3146081: DEVICE_DRAW_RENDERTARGETVIEW_NOT_SET]
D3D11 ERROR: ID3D11DeviceContext::DrawIndexed: Current Primitive Topology value (0) is not valid. [ EXECUTION ERROR #365: DEVICE_DRAW_INVALID_PRIMITIVETOPOLOGY]
D3D11 ERROR: ID3D11DeviceContext::DrawIndexed: A Vertex Shader is always required when drawing, but none is currently bound. [ EXECUTION ERROR #341: DEVICE_DRAW_VERTEX_SHADER_NOT_SET]
D3D11 WARNING: ID3D11DeviceContext::DrawIndexed: An Index Buffer is expected, but none is bound. This is OK, as reading from a missing Index Buffer is defined to return 0.However the developer probably did not intend to make use of this behavior.  [ EXECUTION WARNING #357: DEVICE_DRAW_INDEX_BUFFER_NOT_SET]
D3D11 ERROR: ID3D11DeviceContext::DrawIndexed: Rasterization Unit is enabled (PixelShader is not NULL or Depth/Stencil test is enabled and RasterizedStream is not D3D11_SO_NO_RASTERIZED_STREAM) but position is not provided by the last shader before the Rasterization Unit. [ EXECUTION ERROR #362: DEVICE_DRAW_POSITION_NOT_PRESENT]
D3D11 ERROR: ID3D11DeviceContext::DrawIndexed: Vertex Shader - Pixel Shader linkage error: Signatures between stages are incompatible. Semantic 'SV_POSITION' is defined for mismatched hardware registers between the output stage and input stage. [ EXECUTION ERROR #343: DEVICE_SHADER_LINKAGE_REGISTERINDEX]
D3D11 ERROR: ID3D11DeviceContext::DrawIndexed: Vertex Shader - Pixel Shader linkage error: Signatures between stages are incompatible. The input stage requires Semantic/Index (COLOR,0) as input, but it is not provided by the output stage. [ EXECUTION ERROR #342: DEVICE_SHADER_LINKAGE_SEMANTICNAME_NOT_FOUND]
D3D11 WARNING: ID3D11DeviceContext::DrawIndexed: The size of the Constant Buffer at slot 0 of the Vertex Shader unit is too small (192 bytes provided, 224 bytes, at least, expected). This is OK, as out-of-bounds reads are defined to return 0. It is also possible the developer knows the missing data will not be used anyway. This is only a problem if the developer actually intended to bind a sufficiently large Constant Buffer for what the shader expects.  [ EXECUTION WARNING #351: DEVICE_DRAW_CONSTANT_BUFFER_TOO_SMALL]
D3D11 ERROR: ID3D11DeviceContext::DrawIndexed: Current Primitive Topology value (0) is not valid. [ EXECUTION ERROR #365: DEVICE_DRAW_INVALID_PRIMITIVETOPOLOGY]
D3D11 WARNING: ID3D11DeviceContext::DrawIndexed: The Pixel Shader expects a Render Target View bound to slot 0, but none is bound. This is OK, as writes of an unbound Render Target View are discarded. It is also possible the developer knows the data will not be used anyway. This is only a problem if the developer actually intended to bind a Render Target View here. [ EXECUTION WARNING #3146081: DEVICE_DRAW_RENDERTARGETVIEW_NOT_SET]
D3D11: Removing Device.
D3D11 ERROR: ID3D11Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung. As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered. The current Device Context was executing commands when the hang occurred. The application may want to respawn and fallback to less aggressive use of the display hardware). [ EXECUTION ERROR #378: DEVICE_REMOVAL_PROCESS_AT_FAULT]
Exception thrown at 0x00007FFE60C44FD9 in Unity.exe: Microsoft C++ exception: _com_error at memory location 0x000000064E6FC5F0.
Exception thrown at 0x00007FFE60C44FD9 in Unity.exe: Microsoft C++ exception: _com_error at memory location 0x000000064E6FCC00.
Exception thrown at 0x00007FFE60C44FD9 in Unity.exe: Microsoft C++ exception: _com_error at memory location 0x000000064E6FCE48.
Exception thrown at 0x00007FFE60C44FD9 in Unity.exe: Microsoft C++ exception: _com_error at memory location 0x000000064E6FE288.
ID3D11Device::CreateTexture2D() failed.

You can see that Media Foundation loads my Media Source DLL and its dependencies, and then calls my code to retrieve the sample format metadata, and then something using D3D11 causes a bunch of errors. I think these errors are happening before any of my code even calls any CUDA or D3D11 functions. My GPU setup code runs when the media foundation player calls IMFMediaSourceEx::SetD3DManager().

Maybe something that's in my video sample format is causing a problem? That object is created by code which essentially looks like this:

    // retrieve parameters from demuxer and compositor
    uint32_t uiWidth = 0, uiHeight = 0;
    m_pCompositor->getOutputResolution(uiWidth, uiHeight);
    uint32_t uiVideoSampleDuration = 0, uiVideoTimescale = 0;
    m_pDemuxer->GetVideoSampleRate(uiVideoSampleDuration, uiVideoTimescale);

    // allocate an empty media type
    IMFMediaType *pType = NULL;
    HRESULT hr = MFCreateMediaType(&pType);
    hr = pType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
    hr = pType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_RGB32);
    hr = pType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);
    hr = MFSetAttributeSize(pType, MF_MT_FRAME_SIZE, uiWidth, uiHeight);
    hr = pType->SetUINT32(MF_MT_DEFAULT_STRIDE, uiWidth * 4);
    hr = pType->SetUINT32(MF_MT_SAMPLE_SIZE, uiWidth * uiHeight * 4);
    hr = pType->SetUINT32(MF_MT_FIXED_SIZE_SAMPLES, TRUE);
    hr = pType->SetUINT32(MF_MT_ALL_SAMPLES_INDEPENDENT, TRUE);
    hr = MFSetAttributeRatio(pType, MF_MT_FRAME_RATE, uiVideoTimescale, uiVideoSampleDuration);
    hr = MFSetAttributeRatio(pType, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);

The video files use a recent Mpeg standard called OMAF. It's an MP4 container with extra metadata to tell a decoder how to map multiple video streams into a single output projection. It contains multiple HEVC streams which I decode with NVDEC and composite with CUDA into a single output frame, which gets delivered to the player via Media Foundation.

[Chris-RH]

OK, OMAF isn't something that we are hugely familiar with, but it looks interesting.
Please could you send a reproducible project file to unitysupport@renderheads.com? Please include the issue number in the subject line. That way we can give it a proper debug. Thank you :)

[Chris-RH]

Hi @richard42

Please could you send a reproducible project file to unitysupport@renderheads.com? Please include the issue number in the subject line. That way we can give it a proper debug. Thank you :)

[richard42]

Yes, sorry for the delay. The work that I am doing with this project is actually under contract and proprietary, and I am working with the counterparty to figure out how I can share the Media Foundation plugin that I have written with you for testing. You will need to install the plugin, and have a CUDA-capable GPU, in order to run the test and reproduce the problem.

In the meantime I have also been doing more debugging, and I have narrowed down the scope of the problem some more. I progressively removed CUDA rendering code until the D3D error stopped occurring. And I found that the problem will happen if and only if I call cuGraphicsMapResources() and cuGraphicsUnmapResources() to acquire a lock on the D3D textures in order to do the compositing in CUDA.

The way that this Media Source works is that it allocates and owns a pool of big (10k or 12k wide) D3D11 textures. When segments from a new frame start decoding, one of these D3D textures is locked with cuGraphicsMapResources, and as each segment from an HEVC stream is decoded, its output is converted from NV12 to RGB and composited into the big locked output texture. When all segments for a particular frame have finished, the lock on that texture is released with cuGraphicsUnmapResources, and it is packaged up in a Media Foundation sample and sent down the line.

I have never done Cuda/D3D interop before, but I do have a lot of experience with other GPU interop technologies: OpenGL/OpenCL, OpenGL/Cuda, and OpenGL/Metal, so I have a good idea for how this stuff is supposed to work. As far as I understand it, D3D doesn't have any notion of a "current context" in a thread like OpenGL does, right? That is the source of many problems, but as I understand it, should not be a factor here with D3D11. Is it possible that something in the Media Foundation / Unity stack is using the D3D textures that I'm providing after they are released? I create them with the IMFTrackedSample interface, so when they are released by the other Media Foundation users, I get an Invoke() callback and I put them back into my pool for use with a future decoded frame.

[Chris-RH]

OK, thank you for the additional information. It would still really help if you could share the project file at some point.

[richard42]

Ok, here is a sample project file which they sent me which can be used to reproduce the bug. You can download it from here:
[redacted]
To run a test with this, I start the project then do the following:

1. Right-click in the Video tab and select Video --> AVPro Video - Media Player
2. In the Media Player section of the Inspector tab on the right, click on the folder icon (Browse)
3. Select an OMAF file and let it play until the D3D error occurs and the player stops with an error like "[AVProVideo] Error: Loading failed. File not found, codec not supported, video resolution too high or insufficient system resources". Sometimes it also gives a "Decoding Failed....".

Naturally you will not be able to load/play the OMAF files with my custom Media Source plugin installed on the system.

[Chris-RH]

Hi @richard42,
That file only contained AVPro? We would need your Media Source plugin as well as some OMAF files to test. If you can provide this, please email unitysupport@renderheads.com. Please do not post project files here.

[richard42]

Hi @Chris-RH, I just got approval to share the media source installer with you, and I have sent links to the installer and a test media file to the email address you provided.

[Chris-RH]

Thank you, we will look at those :)

[richard42]

From my testing, it seems that a ID3D11DeviceContext (in someone else's code) is getting corrupted occasionally (presumably a race condition) when I use the cuGraphicsMapResources() or unmap functions. This is very frustrating because I don't even have or use any ID3D11DeviceContext objects anywhere in my code. From my reading about D3D11, the ID3D11Device object is thread-safe (and only used for creating/destroying objects) while the ID3D11DeviceContext is not thread-safe and must be protected by external synchronization mechanisms. Is it possible that the CUDA cuGraphicsMapResources function is internally retrieving the immediate D3D11 device context corresponding with the ID3D11Device and using it during the map/unmap operation? If so, that seems like a bug in CUDA. Unfortunately in my past experience NVidia has been completely unhelpful in diagnosing and fixing bugs in their drivers and frameworks. Assuming that this is the cause of the problem, do you know of any way to retrieve any inherent synchronization object associated with the ID3D11DeviceContext and lock it around the map/unmap calls to prevent race conditions? I am not a D3D expert but it seems like MS does not give us any way to do this and requires us to use an external mutex.

[Chris-RH]

Thank you for sending the files and further information. It is in our To Do list :)

[RichRH]

Hi @richard42,

Apologies it's taken a while to get to this and thanks for sending over all of the relevant test files - I've had a quick look and am certainly seeing similar issues. I think it'll take a while to properly dig into the cause but a few thoughts/observations so far:

• I'm also seeing 'D3D11 ERROR: ID3D11DeviceContext::DrawIndexed' but AVPro doesn't use that function at all so if not in your code would presume we're seeing conflict with Unity itself
• More of the video seems to play if I don't have the inspector window/preview open
• It is possible to lock a 'ID3D11DeviceContext' - see here: https://learn.microsoft.com/en-us/windows/win32/api/d3d10/nn-d3d10-id3d10multithread (Yep, it's D3D10 targetted but still seems to work under D3D11!)
• (UPDATE 1) On my travels I have found a D3D11 version of the above which seems pretty identical: https://learn.microsoft.com/en-us/windows/win32/api/d3d11_4/nn-d3d11_4-id3d11multithread
• I've encountered crashes before with D3D11 textures shared/accessed between multiple contexts/threads - the solution to this generally has been to use keyed mutexes: https://learn.microsoft.com/en-us/windows/win32/api/dxgi/nn-dxgi-idxgikeyedmutex
• (UPDATE 1) I've found running Unity with '-force-d3d11-singlethreaded' seems to alleviate the crashing for me, which is hopefully a big clue and will keep investigating
• (UPDATE 2) Based on the above I think I've narrowed down the issue being related to the Unity D3D11 render context being used by the Media Foundation path (via https://learn.microsoft.com/en-us/windows/win32/api/mfobjects/nn-mfobjects-imfdxgidevicemanager) if it's created as multi-threaded, which it is by default, but this has operated this way for years without issue so we might need to dig a little deeper..

I'll update as/when I find more but maybe the above is of some use in the meantime.

Cheers,