MIDI: native midiout system #2302

tyrone-sudeium · 2020-08-25T02:38:27Z

Todo

Readd checks whether native API was initialized
Ordering of Midi: FluidSynth, WildMidi, Native, FmMidi
Fix WildMidi
Resampler does not forward certain api calls 👎

What do we get?

RPG_RT-like looping, fading, and tempo support with the native synthesizer.
Also attempted to implement the "skip silence at start" behaviour from RPG_RT. I've tested it to work OK with RTP2000, and some MIDI tracks from some games I have on hand, such as Legion Saga III and my own private games.
Lets us ditch SDL_mixer 2 and all of its problems. We mainly used it for its ability to utilise the OS MIDI synthesizer, this PR lets us do that without it.

Known problems / discussion points (all resolved by now)

Duplicated code

There's a lot of very similar code between GenericMidiOut and GenericMidiDecoder. Both these features were developed in parallel, and I was rapidly changing the structure of my work at the same time that work on GenericMidiDecoder was changing. They do pretty different things but it should still be possible to factor out a common base class, or to have some kind of "MIDI Sequencer Controller" class that both of these utilise. It's also possible that most of the code in these could live directly in the MIDI sequencer itself.

Off by default

Currently, this PR shouldn't change behaviour of a production build, which will still use SDL_mixer 2 where it can, thus disabling the midi out. If you want to try out this stuff, it automatically becomes active when you disable SDL_mixer 2, so just set PLAYER_WITH_SDL2_MIXER to False and it should become active, if you're on a supported OS. Speaking of supported OS...

Mac and Windows only

I've written midiout device implementations for Windows and macOS. It should theoretically be possible for iOS, Linux, and libretro, but I haven't written these myself.

Ticks

There's also quite a lot of duplicated code to support ticks. Honestly this is just my C++ skills letting me down. I also don't completely understand how RPG Maker's GetTicks works and don't have any games that use it. See the MidiTempoData struct and related nastiness in audio_midiout.cpp, which is suspiciously similar to stuff in decoder_midigeneric.[h,cpp].

src/audio_sdl.cpp

src/audio_generic.cpp

src/audio_midiout.cpp

src/audio_midiout_device.h

src/platform/macos/midiout_device_coreaudio.h

src/audio_midiout_device.h

src/midisequencer.cpp

src/audio_sdl.h

src/audio_midiout.h

src/audio_midiout.cpp

src/audio_midiout.h

src/audio_midiout.cpp

src/audio_generic.cpp

fmatthew5876 · 2020-08-25T03:53:43Z

We should think about if it's possible for MidiOut to just inherit from AudioDecoder. Then a lot of my review nits go away as the audo_generic.cpp code mostly won't change.

Ghabry · 2020-08-25T07:58:25Z

Thanks for opening the PR. Will think about ways to unify the logic but your code is a good base 👍 (and there shouldn't be conflicts for a while as my audio work is finished)

To explain the std::thread and std::mutex situation: They are broken on 3ds, Wii, emscripten and (I think?) Vita. So for the non-major targets only the Switch implements them. The good news is here that on the slow platforms (emscripten can't support it because SharedArrayBuffers are off for spectre mitigations) this isn't useful anyway. Sooo tif you want to use std::thread the realtime MIDI stuff must be properly abstracted to avoid a ifdef-mess in AudioGeneric.

The motivation for spawning a thread here is that you need a resolution of 1ms. With my FillBuffer approach I use for the Midi Decoders the resolution is the buffer size, which is ~4ms. So for fast MIDI tracks the notes are off by up to 3 ms. This PR fixes this for the "real Midi devices" but it would be also very useful for the softsynths FluidSynth and FmMidi (not useful for WildMidi but wouldn't hurt - is unaffected by this issue).

Ghabry · 2020-08-25T13:37:58Z

When we agree on the general implementation and style here the next steps are:

Decide how to do threading best
"Merge" MidiOut and AudioDecoder (common base class) to reduce code duplication. Imo the only real difference is "Update" vs "FillBuffer", otherwise the interface is almost the same. I could do this as I'm more familiar with the codebase.

tyrone-sudeium · 2020-08-28T09:49:36Z

One thing I haven't handled yet is: If there's an easyrpg.soundfont present, I would assume we should make the highest priority MIDI renderer Fluidlite, right? At the moment on Windows and macOS, if we don't compile sdl_mixer, it'll always use MidiOut. I think eventually we'd want to deprecate sdl_mixer entirely so we'll need to handle this. Does this order of precedence sound reasonable?

Fluidlite (if easyrpg.soundfont available, or otherwise configured)
MidiOut (if on supported platform)
WildMidi (if patches available)
FmMidi (last resort)

src/midisequencer.cpp

… a tempo-command was skipped

Remove unnecessary "loop_position" assignment (duplicated code)

…fying the code-base

Ghabry · 2021-07-15T16:02:27Z

There was a bug in the silence skipping when there was tempo-data encountered. This broke the tick and the sample calculation.

Also WildMidi learned now how to skip initial silence and I can confirm that the looping works (again) 👍

tyrone-sudeium · 2021-07-16T11:06:13Z

I learned something fairly cursed today about win32 midi out. According to the documentation, whether midiOutLongMsg is synchronous or asynchronous is apparently left up to the MIDI driver for the current device, and if it's asynchronous, then what I'm doing here (calling midiOutUnprepareHeader immediately after midiOutLongMsg) will always result in a failure and small memory leak. When it's asynchronous you have to poll the midiOutUnprepareHeader call until it results in MMSYSERR_NOERROR 🤦.

No idea how relevant this is in 2021, since I'm reasonably sure these days you can't even provide your own MIDI drivers any more, everything goes through Microsoft's built-ins now, and the behaviour in Windows 10 is these calls act synchronous.

Ghabry · 2021-07-16T18:18:45Z

More to fix: Fluidsynth changed the function parameter in there VIO interface from long to fluid_long_long_t

Add WildMidi fast-path and fix bug introduced in the WildMidi silence skip. Address review comments

Ghabry · 2021-07-17T11:24:25Z

no idea how I calculated the 512 for the samples. This number is totally wrong. It must be around 64 for ~1.5 ms. Fixes this now.

Ghabry · 2021-07-17T11:25:21Z

This should finally address everything...

…al_synth unique_ptr is depending on the compiler. Add a shutdown flag to prevent accessing a deleted pointer on Windows

CMakeLists.txt

carstene1ns

Works for me.

Ghabry · 2021-07-19T08:52:15Z

Jenkins: test this please

carstene1ns · 2021-07-19T21:59:50Z

Unfortunately 3a19bb6 was not enough, there are still traces of SDL_mixer around. Is only dead code, since the HAVE_SDL_MIXER definition should be missing, however, should still be deleted.

https://github.com/EasyRPG/Player/search?q=have_sdl_mixer

carstene1ns · 2021-07-20T21:15:32Z

Another thing broken: fmmidi fallback mode (autotools build). Will prepare a patch for that.