External Audio Cursor API

[Danielku15]

Motivation

Most details have been discussed here: #1935

The plan of alphaTab is to support eventually the integration with external audio and video sources in an easy-to-use fashion (e.g. by linking to an audio or video element, or other player frameworks like youtube).

Until such a fully fledged integration exists, it is currently hard to build something outside of alphaTab due to the tight integration between the synthesizer and the cursor+highlighting logic.

With this feature it should become possible easier to integrate the cursor and highlighting with an external audio source.

Proposed Feature

alphaTab should provide an API for placing and animating the cursor via an external time provider.

The approach of Guitar Pro should act as a base for the design:

A list of sync points define how the time axis of an external audio, and the alphaTab cursor placement should align.

One possible approach to handle the alphaTab time axis, is to still generate a Midi file.

Then an external audio provides "timePositionChanged" events into alphaTab defining isSeek, currentTime and endTime in millis.

alphaTab translates the time position into the respective midi tick position respecting the sync point information. A mapping logic exists in the synth already and could be refactored to something reusable: https://github.com/CoderLine/alphaTab/blob/develop/src/synth/AlphaSynth.ts#L489
https://github.com/CoderLine/alphaTab/blob/develop/src/synth/MidiFileSequencer.ts#L325

Without sync points the logic could almost be the same by just linearly mapping the time axis. With the sync points the whole time axis is extended with a relative time factor which has to be taken into account.

A non-optimized lookup is needed for seek scenarios where we freshly need to derive the position. This can happen like today by starting at 0 and then fast forwarding in the required steps defined by sync points and tempo changes.

An optimized lookup is needed during playback. We should internally keep a state of the current place in the time axis the upcoming tempo changes and sync points. Assuming the external audio is playing normally, it is possible to "no-op" in many time update calls as nothing changes from the cursor and highlighting situation. Only when the next point-of-change occurs (change in played beats, sync point hit, tempo change..) the logic triggers.

Speed Trainer

It is quite common for media players to provide a playback speed option. The provided APIs should be able to handle this like alphaSynth today.

Metronome

Handling additional the playback of a metronome (hybrid audio) is out-of-scope for this item.

User Interactivity

Selecting playback ranges is disabled when operating in this mode. (maybe a future extension to allow looping, playback ranges etc.).

Player APIs

Most player related APIs will no-op like today (e.g. volume change or play/pause).

The following list describes whether APIs are available when the alphaTab is operating in this mode.

Methods

• changeTrackMute - not available
• changeTrackSolo - not available
• changeTrackTranspositionPitch - not available
• changeTrackVolume - not available
• enumerateOutputDevices - available
• getOutputDevice - available
• loadSoundFont - not available
• loadSoundFontFromUrl - not available
• pause - not available
• play - not available
• playBeat - not available
• playNote - not available
• playPause - not available
• resetSoundFonts - not available
• scrollToCursor - available
• setOutputDevice - not available
• stop - not available

Properties

• countInVolume - not available
• isLooping - not available
• isReadyForPlayback - not available
• masterVolume - not available
• metronomeVolume - not available
• midiEventsPlayedFilter - not available
• playbackRange - not available
• playbackSpeed - not available
• player - not available
• playerState - not available
• tickCache - available
• tickPosition - available
• timePosition - available

Events

• activeBeatsChanged - available
• beatMouseDown - available
• beatMouseMove - available
• beatMouseUp - available
• midiEventsPlayed - not available
• playbackRangeChanged - available (see note above)
• playedBeatChanged - available
• playerFinished - available
• playerPositionChanged with mapped midi tick information. Additionally a "source" information for the event should be added to differenciate:
  • changes because of user interaction
  • changes because of internal synth playback
  • changes because of internal seeking (looping, playback ranges)
  • changes because of external audio time update
• playerReady - not available
• playerStateChanged - not available
• soundFontLoad - not available
• soundFontLoaded - not available

Implementation Details (initial thoughts)

Inside alphaTab we need an abstraction handling aspects between the different players:

• Today we have IAlphaSynth
• This new mode needs also some internal player keeping the state but allowing external updates.
• In future we will have more modes where we use an external audio/video element (provided externally, or via Guitar Pro embedded file).

Most logic for the cursor should not change.

The new APIs should "map" the external time axis to the alphaTab time axis. And then we work like today in alphaTab in our axis (midi) to lookup and match things. We definitely need the "sync points" which additionally modify the speed (compared with the current logic only having midi tempo changes). Otherwise the cursor animations would be too fast/slow.

Sync points can likely be achieved by generating "additional" tempo changes. The BPM changes also seems to be the strategy of GuitarPro (according to the UI).

Guitar Pro handles sync points as automations on masterbar level:

<Automation>
<Type>SyncPoint</Type>
<Linear>false</Linear>
<Bar>1</Bar>
<Position>0</Position>
<Visible>true</Visible>
<Value>
<BarIndex>1</BarIndex>
<BarOccurrence>0</BarOccurrence>
<ModifiedTempo>129.07317</ModifiedTempo>
<OriginalTempo>110</OriginalTempo>
<FrameOffset>70800</FrameOffset>
</Value>
</Automation>
<Automation>
<Type>SyncPoint</Type>
<Linear>false</Linear>
<Bar>2</Bar>
<Position>0</Position>
<Visible>true</Visible>
<Value>
<BarIndex>2</BarIndex>
<BarOccurrence>0</BarOccurrence>
<ModifiedTempo>134.31473</ModifiedTempo>
<OriginalTempo>110</OriginalTempo>
<FrameOffset>132300</FrameOffset>
</Value>
</Automation>

While we should store them also in the data model (for file format compatiblity), we need ways of changing/adapting the sync points easy.