Playback produces unexpected results in Unity

[wyskoj]

I've integrated DWM into my Unity project (2019.1.0f2) for MIDI file playback and parsing. I've called playback.Start() and the MIDI file will not play. I have tried several MIDI files and nothing will play.

// Use the first available OutputDevice (Microsoft GS Wavetable Synth)
var outputDevice = OutputDevice.GetAll().ToArray()[0];
// The MIDI file is copied to this location earlier
var playback = MidiFile.Read(Application.dataPath + @"/Scripts/in.mid").GetPlayback(outputDevice);

Later, in a coroutine (so the MIDI file starts playing at a specific time):

playback.Start();

Playback doesn't want to start, even on the main thread.

Also, playback.Play() works, but it freezes the thread which I don't want, and it also drops a lot of notes, even on really simple MIDI files.

No exceptions are thrown.

What am I missing? And is there some fix to the dropped notes?

[melanchall]

Hi,

Please give me a MIDI file you see issue on. I'll test playback on my side.

[wyskoj]

Here are a couple, one simple and one complex.
midis.zip

[melanchall]

I don't see problems with Start method regarding starting playback. I suppose your program exits before playback started or something like that.

But I confirm the problem with some notes aren't played in the complex file. Thank you a lot for reporting the issue! I'll fix it as soon as possible.

[wyskoj]

Can you send me your script that you used to play these?

[melanchall]

I use this simple program:

using Melanchall.DryWetMidi.Devices;
using Melanchall.DryWetMidi.Smf;

namespace Issue31
{
    class Program
    {
        static void Main(string[] args)
        {
            var outputDevice = OutputDevice.GetByName("Microsoft GS Wavetable Synth");

            Console.WriteLine("Press any key to start 'Never Look Back.mid' playback...");
            Console.ReadKey();

            var midiFile = MidiFile.Read("Never Look Back.mid");
            var playback = midiFile.GetPlayback(outputDevice);
            playback.Start();

            Console.WriteLine("Press any key to stop playback...");
            Console.ReadKey();
            playback.Stop();

            Console.WriteLine("Press any key to start 'percussion test.mid' playback...");
            Console.ReadKey();

            midiFile = MidiFile.Read("percussion test.mid");
            playback = midiFile.GetPlayback(outputDevice);
            playback.Start();

            Console.WriteLine("Press any key to stop playback...");
            Console.ReadKey();
            playback.Stop();

            Console.WriteLine("Press any key to exit...");
            Console.ReadKey();
        }
    }
}

[melanchall]

I've fixed bug with some notes aren't played. Please get latest sources from develop branch.

[wyskoj]

Okay, getting somewhere. I opened up a new project and ran parts of that code you provided. The file now plays for a few seconds before this is thrown:

NullReferenceException: Object reference not set to an instance of an object
Melanchall.DryWetMidi.Devices.MidiClock.get_IsRunning () (at Assets/DryWetMidi/Devices/Clock/MidiClock.cs:59)
Melanchall.DryWetMidi.Devices.MidiClock.OnTick (System.UInt32 uID, System.UInt32 uMsg, System.UInt32 dwUser, System.UInt32 dw1, System.UInt32 dw2) (at Assets/DryWetMidi/Devices/Clock/MidiClock.cs:133)
UnityEngine.UnhandledExceptionHandler:<RegisterUECatcher>m__0(Object, UnhandledExceptionEventArgs)

If I try to hit play again, nothing plays and I have to restart Unity for it to play again.

[melanchall]

I don't see any issues on my side. I suppose it's related to Unity and how it works. I'm testing in Visual Studio with .NET (not Mono used by Unity). So let's investigate what can be wrong.

• Do you use completely the same code as I've provided above for testing?
• Are you sure calling thread doesn't exit after those few seconds?
• Do you run your code in debug or release? If in debug, please run it in release without attaching a debugger.
• Can you perform test in Visual Studio with "true" .NET instead of Unity?

The line where you get the exception is:

public bool IsRunning => _stopwatch.IsRunning;

So the only thing that can give NRE is _stopwatch. But I never set it to null. So my assumption is your environment disposes clock object after some time.

[wyskoj]

Getting closer. The thread was indeed closing a short while after starting the playback. I've fixed that so the thread does not stop. But, stopping and running the program again produces no sound. However, playback.IsRunning returns true. Then stopping the program crashes Unity entirely (it only crashes on the second time). Is this something to do with the output device now? (the note dropping issue is gone)

I also brought the exact code you provided into a new, separate .NET solution and it seems to work fine, except that any note being played when stopping playback is continued to be held, even after starting the next MIDI file.

[melanchall]

But, stopping and running the program again produces no sound

Do you mean stopping playback or program? Can you provide code you're performing tests with?

I also brought the exact code you provided into a new, separate .NET solution and it seems to work fine, except that any note being played when stopping playback is continued to be held

It's OK. To stop currently playing notes on Stop you should set InterruptNotesOnStop to true.

[wyskoj]

I mean stopping the program, then rerunning the program. Here's my code (omitting extraneous animation code):

/* other private vars */
private Playback _playback;
private void Start()
    {
        /* Get MIDI file and copy to assets folder */
        var midiFilePath = EditorUtility.OpenFilePanel("Open MIDI file", "%USERPROFILE%/Desktop", "mid");

        File.Delete(Application.dataPath + @"/Scripts/in.mid");

        File.Copy(midiFilePath, Application.dataPath + @"/Scripts/in.mid");

        var midiFile = MidiFile.Read(Application.dataPath + @"/Scripts/in.mid");
        Global.CurrentTempoMap = midiFile.GetTempoMap();
        
        var outputDevice = OutputDevice.GetByName("Microsoft GS Wavetable Synth");

        _playback = midiFile.GetPlayback(outputDevice);
        
        /* ui and note handling */

        // start coroutine
        StartCoroutine(StartMusicAndAnimation(/*extraneous arguments*/));
    }


private IEnumerator StartMusicAndAnimation( /*extraneous arguments*/ )
    {
        yield return null;
        // Accomodate for panel open
        Global.SettleTime += Time.unscaledTime;
        var played = false;
        while (!played)
        {
            
            if (Time.unscaledTime >= Global.SettleTime - 0.1)
            {             
                // Begin instrument and music playback
                _playback.Start();

                /* animation handling */
     
                played = true;
            }

            yield return null;
        }

        // Prevent the thread from closing
        while (true)
        {
            yield return null;
        }
    }

[melanchall]

Instead of while (true) you can use while (_playback.IsRunning) which looks better since you don't need the thread after a file played (or you do?).

Playback and OutputDevice are implement IDisposable and you should always dispose instances of these classes when you're done with them.

So put output device to field as you do with playback and try to change your last-loop part to:

while (_playback.IsRunning)
{
    yield return null;
}

_playback.Dispose();
_outputDevice.Dispose();

[wyskoj]

Okay, I've added the dispose methods to the end of the script just like you provided. Now Unity won't even play the second time, Unity itself just freezes. This may be some bug in Unity or something. I'm gonna poke around with the settings and see if I can get something to work.

[melanchall]

OK, please let me know if you solve the problem.

[melanchall]

@wyskoj Any news?

[wyskoj]

I'd like to say I have. I tried changing a bunch of stuff trying to get it to work. Nothing worked so I'm taking a hiatus... I'll get back to it a different day.

[Teafuu]

I'm also experiencing the same issue whereas playback.IsRunning() returns true, although no sound is playing.

The same applies to the midi files being played for a few seconds then cut short as the thread closes. Let me know when you have a solution, I saw that you managed to open the playback on a separate thread, however it won't work when restarting.

[melanchall]

@wyskoj @Teafuu Can you create a minimal Unity project to reproduce the problem and give it to me? I'm going to install Unity to try to solve the issue. I'm not familiar with developing on Unity so I hope you will help me with it if I have questions :)

[wyskoj]

DWM Test2.zip
Here's a simple test. To reproduce the exact problem:

1. Open and load the Unity project
2. Edit Line 19 of test.cs to a MIDI file on your computer
3. Press play. Allow the full MIDI file to playback. When it's done, "disposed!" will appear in the console.
4. Press the play button to end the game. Press it again to restart, and Unity will crash.
5. If you press the play button to stop the game before the file is finished, Unity will not crash, but will not playback the second time.

Let me know if you have issues or questions.

[melanchall]

Thank you a lot!

Please say exact version of Unity you use so I can be in the same conditions as you.

[wyskoj]

Unity 2019.1.0f2

[melanchall]

@wyskoj I confirm the issue. I've spent several hours to find the root of the behavior, but I found nothing. Seems like there is a deadlock inside of Unity/Mono on objects cleanup.

I've tried to follow this article and removed all finalizers but it didn't solve the problem.

Do you mind if I create thread on Unity forum or create a support request attaching provided Unity project?

[wyskoj]

Please do! Thanks for all your help.

[melanchall]

Unity freezes on second play after objects disposing

[melanchall]

Answer from Unity tech support:

We successfully reproduced this issue and have sent it for resolution with our developers.

[melanchall]

https://issuetracker.unity3d.com/issues/unity-freezes-when-entering-play-mode-after-the-object-disposing

@wyskoj @Teafuu Please vote for the bug to bring Unity team attention to the problem.

[melanchall]

Answer from Unity tech support:

I've been looking into this hang/freeze and it looks like there is a winmm thread that is persisting even after the dispose occurs. Due to this when we enter another playmode (or close the editor) and a domain unload occurs it gets stuck waiting for this thread to end. Which it appears to never do so. I hope this information helps. If you could look into getting this thread to close/end on dispose and let me know if it fixes the freeze that would be excellent!

I'll determine if winmm thread can be terminated on dispose. Probably I hold some references that must be destroyed.

[melanchall]

Problem with using winmm timers in Unity/Mono. Unity team is working on a solution. Waiting for news from them.

[melanchall]

Unity tech support:

Unfortunately it seems that the issue that is causing the hang in mono will require a fairly involved change and therefore will not be getting fixed anytime in the short term. There are several other bugs in our database that are of similar types where mono's domain unload is waiting on a thread indefinitely so this effort will be undertaken at some point.

It seems the bug will not be fixed in nearest future :(

[Teafuu]

I'll take it for granted that there is no current workaround regarding this issue?

[melanchall]

@Teafuu Right now there are no workarounds.

BUT I'm going to implement kind of ticking mode for playback/clock to choose between:

• high-precision timer (current approach which fails on Unity/Mono)
• regular .NET timer (can cause latency of about 15 ms or higher)
• manual ticking (useful for coroutines in Unity, you need to call "tick" method in coroutine body to play next portion of data).

I'll implement this API as soon as possible.

[melanchall]

I've added MidiClockSettings parameter to all playback creation methods. It has CreateTickGeneratorCallback which lets to specify tick generator that will be used for playback.

By default HighPrecisionTickGenerator will be used. But it cause Unity to hang as we know. You can use either RegularPrecisionTickGenerator or manual ticking playback's clock. Let me show both ways.

RegularPrecisionTickGenerator

All you need for your project is to get playback with this code:

_playback = _midiFile.GetPlayback(_outputDevice, new MidiClockSettings
{
    CreateTickGeneratorCallback = interval => new RegularPrecisionTickGenerator(interval)
});

RegularPrecisionTickGenerator uses System.Timers.Timer to drive playback's clock so it can be possibly inaccurate.

Manual ticking

Playback creation:

_playback = _midiFile.GetPlayback(_outputDevice, new MidiClockSettings
{
    CreateTickGeneratorCallback = interval => null
});

In coroutine:

_playback.Start();
while (_playback.IsRunning)
{
    _playback.TickClock();
    yield return null;
}

I suppose the second approach will be the most accurate. Both ways work perfectly in Unity project. You can even create your own tick generator and use it.

@wyskoj @Teafuu Please download sources of the library from develop branch and replace all your DryWetMidi folder since there are a lot of changes in the library structure (nearest release will be major one including breaking changes).

Plaese confirm the issue is resolved so I can close it.

[wyskoj]

I have tried both of the methods you have provided, but each drops or sticks on a considerable amount of notes. But I assume this is just a limitation of lower fidelity timers. It also seems to not playback the first second or so of the MIDI file.

[melanchall]

Let's create custom tick generator that will be super accurate (but it consumes a lot of CPU so it's not recommended to implement timers in this way):

private sealed class LoopTickGenerator : ITickGenerator
{
    public event EventHandler TickGenerated;

    private readonly Thread _thread;
    private bool _disposed;

    public LoopTickGenerator()
    {
        _thread = new Thread(() =>
        {
            for (var i = 0; ; i++)
            {
                if (i % 1000 == 0)
                    TickGenerated?.Invoke(this, EventArgs.Empty);
            }
        });
    }

    public void TryStart()
    {
        if (_thread.IsAlive)
            return;

        _thread.Start();
    }

    public void Dispose()
    {
        Dispose(true);
    }

    private void Dispose(bool disposing)
    {
        if (_disposed)
            return;

        if (disposing)
        {
            _thread.Abort();
        }

        _disposed = true;
    }
}

So playback should be created like this:

_playback = _midiFile.GetPlayback(_outputDevice, new MidiClockSettings
{
    CreateTickGeneratorCallback = interval => new LoopTickGenerator()
});

And remove manual ticking (_playback.TickClock();).

@wyskoj Please try this approach just for test. While you're trying it, I'll investigate current problems. But at now it seems they are indeed related with low accuracy (in case of RegularPrecisionTickGenerator) and how often Unity asks for next frame (in case of manual ticking within coroutine).

[melanchall]

I've checked the following approaches on Never Look Back.mid file:

• high precision tick generator
• regular precision tick generator
• manual ticking within loop on separate thread
• custom tick generator that uses thread from manula ticking approach.

All events are went through output device, I've dumped delays for all events for all approaches to files:

CheckFilePlayback_RegularPrecisionTickGenerator.txt
CheckFilePlayback_ManualTicking.txt
CheckFilePlayback_HighPrecisionTickGenerator.txt
CheckFilePlayback_CustomTickGenerator.txt

Files contain strings like this:

[9] [+009 ms]: 00:00:00 -> 00:00:00.0088251

It means:

• 9th event
• event was sent 9 ms later than it should be sent
• event should be sent after 00:00:00 from the start of a file
• but it was sent after 00:00:00.0088251 (~9 ms) from the start of a file

I don't see any catastrophic delays. Maybe output device is unable to process some events in time due to a lot of events are incoming at almost same time.

I'll continue investigation of possible issues.

[haosizheng]

oh,met this issue and finally find this page.

hoping solving soon.

[melanchall]

@hoszeching I've contacted support recently and unfortunately the priority of the issue is very low so it seems it will not be fixed in near future.

[Teafuu]

I'll be waiting patiently, been putting it on hold until it gets fixed 👍

[wyskoj]

I'm taking a crack at this again...
And this seems to be working (as a basic example).

using System;
using System.Collections;
using System.Collections.Generic;
using Melanchall.DryWetMidi.Core;
using Melanchall.DryWetMidi.Devices;
using UnityEngine;

public class DWMHandle : MonoBehaviour
{
    private Playback _playback;
    private OutputDevice _outputDevice;

    // Start is called before the first frame update
    void Start() {
        var midiFile = MidiFile.Read("Assets/GROOVE.MID");
        _outputDevice = OutputDevice.GetByName("Microsoft GS Wavetable Synth");
        _playback = midiFile.GetPlayback(_outputDevice, new MidiClockSettings
        {
            CreateTickGeneratorCallback = interval => new RegularPrecisionTickGenerator(interval)
        });
        _playback.InterruptNotesOnStop = true;
        StartCoroutine(StartMusic());
    }

    private IEnumerator StartMusic() {
        _playback.Start();
        while (_playback.IsRunning) {
            yield return null;
            _playback.TickClock();
        }
        _playback.Dispose();
        
    }

    private void OnApplicationQuit() {
        _playback.Stop();
        _playback.Dispose();
    }
}

No crashes, so far.... and notes don't seem to drop or have any inconsistencies. Not sure what's different except unity version = 2018.4.15f1. Could possibly be a bug sometime after this Unity release?

[melanchall]

Hm, really strange that this approach didn't work for you when I wrote about it last time and now it works :) Maybe it's because of combination of RegularPrecisionTickGenerator and manual ticking (TickClock call). Just for test can you comment _playback.TickClock(); and check again?

@Teafuu Please try approach shown by @wyskoj. Does it work for you?

[melanchall]

It seems the bug will not be fixed :( I've contacted Unity tech support again:

Me

Hi,
Sorry that I'm writing you again, but I see that original issue (https://issuetracker.unity3d.com/issues/editor-freezes-when-updating-a-nativearray-on-the-net-4-dot-x-scripting-runtime-and-entering-play-mode-a-second-time) marked as Won't fix. Since my issue (https://issuetracker.unity3d.com/issues/unity-freezes-when-entering-play-mode-after-the-object-disposing) is duplicate of that, does it mean that my problem will never be fixed?

Unity

Hello,
Unfortunately, that is correct = we will not be able to fix this in the near term because it probably requires rewriting of internal threading functionality, which might introduce new issues. The main case has been tagged for a revisit internally, but it will probably be months until the case is re-valuated again.

[melanchall]

Miracle happened, the bug is fixed now! Message from Unity tech support:

The bug has been fixed and the fix has been backported across several Unity releases! Here is the issuetracker link for the bug that the fix was made under: https://issuetracker.unity3d.com/issues/commandbuffer-native-plugin-events-hang-in-the-editor

So I'm finally closing the issue. (more than 2 years, heh)

[wyskoj]

Awesome!!