Audio stops playing after stopping and restarting editor a number of times

[coreyx]

Using the latest commit 693233d and UE 4.14 Preview 2, if I play in editor and stop multiple times, eventually the audio will stop playing when the video starts. When this happens, the condition is permanent until the editor is restarted.

I have been able to reproduce this bug consistently. I had to play and stop anywhere from 2 times to 10 times to reproduce.

I also verified that another application was able to play the audio from the same NDI stream while the UE editor was in the failure condition, so the problem is isolated to the plugin.

Reloading the editor unloads and reloads the plugin, which alleviates the problem.

[coreyx]

A little update. I noticed the UE4Editor.exe stays running indefinitely in the Background Processes section of Task Manager in Windows 10 after this issue occurs, even after you close it and it disappears from the desktop. My theory is the thread locking is not allowing the program to terminate fully.

[gmpreussner]

Are you able to catch this in the Visual Studio debugger? Would be interesting to know which thread is getting stuck and where. I'll try to repro as well.

[coreyx]

I ran it with Visual Studio debugger attached. It looks like this is the culprit:

NdiMediaAudioSampler.cpp, line 10:
FNdiMediaAudioSampler::~FNdiMediaAudioSampler()
{
Thread->Kill(true); // <----- this is stuck
delete Thread;
Thread = nullptr;
}

So, the thread is still spinning for whatever reason and the Kill function waits and blocks.

bShouldWait If true, the call will wait infinitely for the thread to exit.
From https://docs.unrealengine.com/latest/INT/API/Runtime/Core/HAL/FRunnableThread/Kill/index.html

[gmpreussner]

Can you try the latest plug-in code? I wasn't able to reproduce, but Newtek fixed an issue with their thread synchronization.

[gmpreussner]

By the way, how are you able to play NDI audio? 4.14 does not have support for quadrophonic audio. That's something I added for 4.15. Are you using UE4 Github Master?

[gmpreussner]

Oh nevermind, I guess you're using your own test content. I was using the NDI Test Patterns app, which only outputs quadrophonic.

[coreyx]

Yep, you're right about that. I was streaming my own content which has stereo audio. I am working on testing the newest commit.

Edit: Fixed the issue I was having starting my project after upgrading NdiMedia plugin by doing a "Rebuild Solution" in Visual Studio.

[coreyx]

I upgraded my test project and tried out commit 1805260. I got the editor to hang on the 2nd or 3rd play/stop cycle in PIE mode. I'll see if this happens in a clean project.

I'm using UE4 4.14 Preview 3.

[gmpreussner]

Can you explain how to repro this? Do you open the NDI Media Source in your level blueprint's BeginPlay event, and then you simply play/stop PIE several times in a row?

[coreyx]

I do it just the way you describe sometimes, or I setup hotkeys using input mapping and open a source and close it using the NDI MediaPlayer multiple times.

[coreyx]

I reproduced the editor hang using commit #1805260 in a fresh UE4 4.14 Preview 3 project.

It's happening pretty reliably after 3 or so play/stop cycles as I described above.

I am opening a source on Begin Play and not closing it on End Play, although I can try closing it as well (I've tried both with previous builds--it did not seem to make a difference).

Edit: Tried closing the Media Player on End Play as well. It still hangs after a few PIE mode play/stop cycles.

[gmpreussner]

I still haven't been able to repro, but I suspect that it's because NDIlib_recv_destroy is called while NDIlib_recv_capture is in a waiting state. Newtek is investigating now.

[coreyx]

Ok, so I am running commit @#1805260 in UE4 4.14 Preview 3. I switched to DebugGame mode and ran the editor with Visual Studio's debugger attached. I'm opening a source on Begin Play and closing it on End Play.

It took me like 7 tries of playing and stopping in the editor, but I got the editor to freeze. No crash window appears. The editor just hangs, although it's still responding to Windows. I can't interact with the editor though, as it's not rendering or receiving input.

I broke into the debugger, and here's what immediately came up in WindowsPlatformProcess.cpp:

1053: bool FEventWin::Wait(uint32 WaitTime, const bool bIgnoreThreadIdleStats /= false/)
1054: {
1055: WaitForStats();
1056:
1057: SCOPE_CYCLE_COUNTER( STAT_EventWait );
1058: check( Event );
1059:
1060: FThreadIdleStats::FScopeIdle Scope( bIgnoreThreadIdleStats );
1061: return (WaitForSingleObject( Event, WaitTime ) == WAIT_OBJECT_0); <---- Thread is waiting here
1062: }

The crash reporting thread in UE4Editor-Core.dll's next statement to execute is in line 390 of WindowsPlatformCrashContext.cpp:

385: /** Main loop that waits for a crash to trigger the report generation */
386: FORCENOINLINE uint32 Run()
387: {
388: while (StopTaskCounter.GetValue() == 0)
389: {
390: if (WaitForSingleObject(CrashEvent, 500) == WAIT_OBJECT_0)
391: {
392: ResetEvent(CrashHandledEvent);
393: HandleCrashInternal();
394: ResetEvent(CrashEvent);
395: // Let the thread that crashed know we're done.
396: SetEvent(CrashHandledEvent);
397: break;
398: }
399: }
400: return 0;
401: }

MainThrd's next statement to execute is line 36 of ScopeLock.h:

26: /**
27: * Constructor that performs a lock on the synchronization object
28: *
29: * @param InSynchObject The synchronization object to manage
30: /
31: FScopeLock( FCriticalSection InSynchObject )
32: : SynchObject(InSynchObject)
33: {
34: check(SynchObject);
35: SynchObject->Lock();
36: }

[coreyx]

Repeated the same test as above but without closing the Media Player on End Play. Same results.

I'm going to try and step through the code that runs after End Play sequentially to see where exactly this chain of events results in a deadlock.

I may need symbols for either XAudio or the NDI SDK to see what's going on on a deeper level. I know there is an XAudio debug version, but do you know if NDI's SDK comes with symbols? I uninstalled it when you included the libraries as part of the plugin.

[gmpreussner]

There are no debug symbols in the SDK, unfortunately.

Can you copy/paste the callstack of both the Main thread and the Processing.NDI.Lib.x64 thread?

[coreyx]

Here is MainThrd's call stack:

[External Code] 

UE4Editor-NdiMedia.dll!FNdiMediaAudioSampler::SetReceiverInstance(void * InReceiverInstance) Line 32 C++
UE4Editor-NdiMedia.dll!FNdiMediaPlayer::Close() Line 212 C++
UE4Editor-MediaPlayerEditor.dll!FMediaPlayerEditorModule::HandleEditorEndPIE(bool bIsSimulating) Line 218 C++
UE4Editor-MediaPlayerEditor.dll!TBaseRawMethodDelegateInstance<0,FMediaPlayerEditorModule,void cdecl(bool)>::ExecuteIfSafe(bool <Params_0>) Line 639 C++
UE4Editor-UnrealEd.dll!TBaseMulticastDelegate<void,bool const >::Broadcast(const bool <Params_0>) Line 922 C++
UE4Editor-UnrealEd.dll!UEditorEngine::EndPlayMap() Line 218 C++
UE4Editor-UnrealEd.dll!UEditorEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 1670 C++
UE4Editor-UnrealEd.dll!UUnrealEdEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 371 C++
UE4Editor.exe!FEngineLoop::Tick() Line 2859 C++
UE4Editor.exe!GuardedMain(const wchar_t * CmdLine, HINSTANCE * hInInstance, HINSTANCE__ * hPrevInstance, int nCmdShow) Line 152 C++
UE4Editor.exe!WinMain(HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, char * __formal, int nCmdShow) Line 191 C++
[External Code]

There are three threads named Processing.NDI.Lib.x64.dll with different ID's and locations in the Processing.NDI.Lib.x64.dll DLL and/or its dependencies.

• 15528 Processing.NDI.Lib.x64.dll!00007ffa6f3299f9
• 6364 Processing.NDI.Lib.x64.dll!00007ffa6f32a7b0
• 1088 mswsock.dll!00007ffad5448033

Screenshot:
http://i.imgur.com/NhjjIqW.png

Call stacks are as follows:

15528 Processing.NDI.Lib.x64.dll!00007ffa6f3299f9:
[External Code]

Processing.NDI.Lib.x64.dll!00007ffa6f3299f9() Unknown
Processing.NDI.Lib.x64.dll!00007ffa6f328919() Unknown
[External Code]

6364 Processing.NDI.Lib.x64.dll!00007ffa6f32a7b0:
[External Code]

Processing.NDI.Lib.x64.dll!00007ffa6f32a7b0() Unknown
Processing.NDI.Lib.x64.dll!00007ffa6f328939() Unknown
[External Code]

1088 mswsock.dll!00007ffad5448033:
[External Code]

Processing.NDI.Lib.x64.dll!00007ffa6f32a173() Unknown
Processing.NDI.Lib.x64.dll!00007ffa6f328929() Unknown
[External Code]

All stack traces were captured right after the editor froze. I simply went to Visual Studio and gave it a Break All command, then examined the threads.

[gmpreussner]

Do you have the callstack of the "FNdiMediaAudioSampler" thread?

[gmpreussner]

Please try the following change:

void FNdiMediaPlayer::Close()
{
    {
        FScopeLock Lock(&CriticalSection);

        if (ReceiverInstance != nullptr)
        {
            NDIlib_recv_destroy(ReceiverInstance);
            ReceiverInstance = nullptr;
        }

        CurrentState = EMediaState::Closed;
        CurrentUrl.Empty();

        LastAudioChannels = 0;
        LastAudioSampleRate = 0;
        LastVideoDimensions = FIntPoint::ZeroValue;
        LastVideoFrameRate = 0.0f;

        SelectedAudioTrack = INDEX_NONE;
        SelectedVideoTrack = INDEX_NONE;
    }

    UpdateAudioSampler();

    MediaEvent.Broadcast(EMediaEvent::TracksChanged);
    MediaEvent.Broadcast(EMediaEvent::MediaClosed);
}

[coreyx]

Here is the callstack of the FNdiMediaAudioSampler thread taken under the same conditions as above:

[External Code] 

UE4Editor-NdiMedia.dll!FNdiMediaPlayer::HandleAudioSamplerSample(const NDIlib_audio_frame_t & AudioFrame) Line 787 C++
UE4Editor-NdiMedia.dll!TBaseRawMethodDelegateInstance<0,FNdiMediaPlayer,void __cdecl(NDIlib_audio_frame_t const & __ptr64)>::ExecuteIfSafe(const NDIlib_audio_frame_t & <Params_0>) Line 639 C++
UE4Editor-NdiMedia.dll!FNdiMediaAudioSampler::Run() Line 49 C++
UE4Editor-Core.dll!FRunnableThreadWin::Run() Line 74 C++
UE4Editor-Core.dll!FRunnableThreadWin::GuardedRun() Line 23 C++
[External Code]

I will try your changes to FNdiMediaPlayer::Close() right now and report back.

[coreyx]

I haven't been able to reproduce the issue after I made the change you suggested above. It's looking good so far!

Edit: Annnnnnd... a few minutes later. Still unable to reproduce the issue. Looks like you fixed it! Great work!

[gmpreussner]

Ok, the old code deadlocked when shutting down while audio samples were being returned.