Skip to content

Technical design

Jump to bottom
LenweSaralonde edited this page Oct 12, 2021 · 23 revisions

Play music with the WoW API

The basic principle of Musician consists in playing audio recordings of instrument notes in time with a musical score or manually by the player using the computer keyboard.

The inspiration was taken from the Mellotron, an electro-mechanical keyboard instrument from the 1960’s on which each key plays a magnetic tape of the recorded note. This instrument and its very distinctive sound was made popular by artists such as The Beatles and David Bowie.

Musician relies on 2 core functions of the WoW API: PlaySoundFile and StopSound.

willPlay, soundHandle = PlaySoundFile(soundFile, channel)

PlaySoundFile accepts a path to an OGG file and a game audio channel. The cool feature of the PlaySoundFile() function is the sounds played are affected by the environmental audio effects (i.e. reverb when inside the Stormwind Cathedral etc.) which provides a more immersive experience.

The note keeps playing until StopSound is called or when the note exceeds the actual sound file duration. There is no way to add loop points in the sound file so if the note duration exceeds the sound file duration, it can only be restarted from the beginning.

StopSound(soundHandle, fadeoutTime)

StopSound accepts as parameter the sound handle provided by PlaySoundFile, but also a fadeoutTime. The fade out is essential in the case of playing musical notes to simulate the decay of the instrument and avoiding the note to end abruptly which would not sound natural at all.

Unfortunately, there is nothing more we can do with the WoW API. There is no way to change the volume, the panning, to set loop points or an envelope (except for the decay as explained above). Everything has to be pre-recorded in a sound file with a limited duration.

Creating a sample library

To be able to play music, we first need to create a sample library, with as much sound files as there are notes and instruments.

This can be achieved by recording each individual note of each instrument but since it requires the help of an actual musician, a full recording setup and a lot of time, the easy way is to generate the samples out of virtual instruments using a DAW such as FL Studio.

The maximum duration of each note is set to 6 seconds which covers 99% of the needs, without taking too much disk space. The note range start from the MIDI C0 (12 @ 16.3515 Hz) to C8 (108 @ 4186 Hz).

A MIDI file containing all the notes is rendered using the virtual instrument we want to use in WoW into a single WAV file, that is then splitted into individual OGG files using a Node.js script and FFmpeg.

The slicer script also performs some normalization for each individual note sample file to ensure each instrument has a constant volume level, is loud enough to be properly heard in game among the other instruments and avoid clipping when many simultaneous notes are playing.

The normalization settings usually applied are :

  • -10dB peak normalization for plucked instruments (harp, guitar etc.)
  • -20dB RMS normalization for continuous instruments (violin, flute, voice etc.)
  • -4dB peak normalization for drums (they need to be louder to preserve the dynamics)

File caching

Musician plays OGG files in time with the music using PlaySoundFile. The main limitation of this function is it's synchronous: the entire WoW client process has to wait for the sound file to be loaded to continue.

Loading a single sample file takes approximately 5 ms from an SSD but can take up to 30 ms from a 5400 RPM HDD. To avoid situations such as the game freezing while playing music, especially on computers running WoW from an HDD, the samples files should be preloaded in RAM.

Operating systems such as Windows and macOS keep a copy of frequently accessed files in the unused portion of RAM: the page cache. Loading a sample file that present in the page cache only takes a few microseconds which does not affect game performance at all.

To ensure the sample files are already present the page cache while playing music, the Preloader first loads all the ~2000 samples when the player logs into the game (it just calls PlaySoundFile then StopSound right away so the sample file is loaded but is not actually played). The loading rate is calculated in real time to avoid FPS drop while loading for the first time after a cold start. If music starts playing during the process, the samples that have not been loaded yet will not play.

When all the samples are loaded, the process starts again and keeps running over and over at a slower rate to ensure the OS keeps them inside the page cache.

On computers having less RAM than recommended by Blizzard for WoW, some issues may occur if there is no enough free RAM to keep all the samples in page cache.

MIDI converter

The WoW UI does not allow to open files except assets that belong to the add-on or the game itself. The only way to import an external file is to copy-paste its contents as text in a text field.

MIDI files, aka "SMF" format is quite complex and it's a binary format. They can't safely be opened in a text editor and copy-pasted. The only way to import a MIDI file in a WoW add-on is to convert it into text.

The idea of a basic web application that converts MIDI files into text quickly appeared to be the most obvious solution. The web page can be hosted online and accessed from a shortened easy-to-remember URL. In addition, it can be bundled with the add-on as it runs entirely on client side in the browser.

The converter was initially made from a forked version of the web MIDI to JSON converter from Tonejs. It was then rewritten using midi-file to work at a lower level, closer to the initial MIDI format.

Some processing has to be made to the song prior to importing it into the add-on. The note-on / note-off events are adjusted accordingly to sustain pedal events. In addition, due to the fact that WoW's sound API doesn't allow pitch variation, the notes affected by pitch-bend events are sliced to the closest semitone, as an autotune would do.

Finally, a new binary music file is generated in a format that is designed for Musician. The binary file is then converted into text using base64 encoding so it can be copy-pasted in WoW.

Song import

Musician accepts songs in MUS format. Songs can be encoded in a base64 string or compressed using the Deflate algorithm. The MUS file structure reflects the way songs are stored in memory in game.

A song has 2 possible ways of storing the note information:

  • Using note on / note off events, as a MIDI file a.k.a. "Live mode"
  • Using note events with a fixed duration a.k.a. "Duration mode"

The "Live mode" is only used for live play (see below) since the note duration is unknown while playing. The "Duration mode" is used for importing the song from MIDI.

MUS file structure

Header

The file starts with a header.

  • Format identifier (4) MUSX where X is the version number of the file format
  • Song title (2 bytes for the title length + title string)
  • Song mode (1) 0x10 for "Duration mode" and 0x20 for "Live mode"
  • Duration (3) Song duration in seconds (rounded to the closed second)
  • Number of tracks (1) Songs can't have more than 256 tracks

Track information

There is as many track information blocks than Number of tracks.

  • MIDI instrument (1) Original MIDI instrument ID (0-127). For drums, 128 is added. This value can be overridden in the Track options.
  • MIDI channel (1) Only used as reference for display
  • Note count (2) Tracks can't have more than 65536 notes

Note information

All track notes are arranged end-to-end.

  • MIDI key (1)
    • MIDI note key (first 7 bits, mask 0x7F) from 12 (C0) to 108 (C8)
    • Note type (last bit, mask 0x80). In "Live mode", 1 is a note on and 0 is note off. In "Duration mode", 1 indicates the Duration is encoded in 16-bit instead of 8 which allows long notes lasting up to 1536 seconds.
  • Time interval from the previous note (2) in 1/240th of a second. If the interval exceeds the 16-bit limit, a 0xFF byte is inserted before the note and the resulting time is subtracted from the current note interval. Several 0xFF bytes can be added if needed.
  • Duration (1 or 2) in 1/255th of 6-second which is the maximum duration of an instrument sample. Only used in "Duration mode". Can be encoded on 2 bytes if the Note type is set to 1.

Track names

There is as many track name blocks than Number of tracks.

  • Track name (2 bytes for the track name length + track name string)

Song settings

This part is optional. It's added by Musician when the song is saved for export but it's not included by the web MIDI converter.

  • Crop from (4) in 1/100th of second.
  • Crop to (4) in 1/100th of second.

Track options

Mandatory if Song settings are present. There is one block per track.

  • Track options (1) Bitmask of options
    • has instrument (first bit, mask 0x1) 1 When an instrument is selected in the song editor
    • is muted (second bit, mask 0x2) 1 When the track is muted in the song editor
    • is solo (third bit, mask 0x4) 1 When the track is solo in the song editor
  • Instrument (1) MIDI instrument ID selected in the song editor (0-255). Values above 127 are for drums.
  • Transpose (1) In semitones + 127

Registry

The Registry is a key component in Musician. It has 2 purposes:

  • Determine who are the other players who run a compatible version of Musician we can interact with.
  • Track the position of the active musicians to simulate the "hearing range" of the music being played.

When a player logs into the game, they first request the list of players inside the communication channel then all players are added to the registry. The message "There are X other musicians around!" is displayed in the chat window with the number of players in the communication channel/Registry. If nameplates are enabled, a musical note icon shows up next to the player names who are in the Registry, indicating they also have Musician.

Every 5 minutes, each player sends a "Hello" message with their add-on and protocol version numbers. This information is used to determine if the player runs a compatible version and to issue a notification or a warning if their add-on is outdated.

The add-on version is also shown in the player tooltip. If the other player has no version information in the Registry yet, it will be requested via a whisper add-on message then the tooltip is updated accordingly.

When another player is playing a song, their position in the 3D world has to be known to calculate the distance from them and determine if we are able to hear them.

The hearing range is set to 40 yards which corresponds to the same range as a healing spell and the UnitInRange API function.

The UnitPosition and UnitInRange API functions do not work for players who don't belong to our party or group. Each active musician has to broadcast their own position in the communication channel along with the music data of the song being played.

The player GUID and position is stored into the registry and is updated every 1 to 2 seconds as long as a song is playing. There is no way to know or request the other players' position when they're not playing music.

The Z position can't be determined and is always set to 0 so proximity calculations are actually in 2D, not 3D. In addition, player coordinates can't be determined in instances so the proximity will be determined only for other group members using the UnitInRange API function.

The active musician position is also used to show their position on the map.

Communication

The music is streamed in chunks, sent periodically over the communication channel.

Depending on the song type, the periodicity and the length of song chunk differ:

  • 2-second chunks (representing 2 seconds of the song), sent every 2 seconds for imported MIDI files with notes and durations.
  • 1-second chunks for song played live with note on/note off events.

The chat system allows only 1 message per second in a CHANNEL, which represents a bandwidth of 256 bytes/second. To avoid the song data to be throttled, the chunks are compressed using the Deflate algorithm.

Due to limitations and poor implementation from Blizzard in an attempt to break group finding add-ons in WoW Classic and BC Classic, the song data can't be sent over the communication channel in the Classic versions of WoW. Instead, it's sent locally over the YELL method that requires the data to be encoded in base 64, resulting in lower bandwidth and increased the risk of chunk drop.

When the player is in a party or raid, the song data is also sent in the group channel since the other group members may not be from the same realm as the player.

In addition, a module for the CrossRP add-on allows to send and receive song data with players from the other faction and/or another realm.

Song chunks structure

A song chunk consists in a header and song data.

  • Header (24) The header is decoded by every player to keep the Registry up to date, show the musician's position on the world map and to determine if the player is in loading range (~100 units/yards). If the player is not within the loading range, only the header is decoded, track and note data is discarded.
    • Version and mode (1)
      • Version (first 4 bits, mask 0x0f) Chunk version
      • Mode (last 4 bits, mask 0xf0) Song mode, with note durations (0x10) or "live" with note on/note off events (0x20)
    • Chunk duration (1) in 1/10th of a second
    • Song ID (1) It's not a real identifier but only a way to distinct 2 different songs that could be played one after the other by the same player. This ID is initialized randomly, is incremented every time a new song is created by the player and is reset when the limit of 255 is reached.
    • Playtime left (2) Number of seconds left until the end of the song. This is used to correctly position the chunk data within the song in case some chunk were dropped due to lag or exceeded bandwidth. For live song, it's fixed to 10 seconds and the resulting song is extended for every chunk until it stops.
    • Player position (18) No Z position since it's not supported by Blizzard's API.
      • X position in 3D world (4) rounded-integer unit precision with 0x7fffffff offset
      • Y position in 3D world (4) rounded-integer unit precision with 0x7fffffff offset
      • Instance ID (4)
      • Player GUID (6) 16-bit Realm ID and 32-bit player ID.
    • Number of tracks (1)
  • Song data The song data consists in track headers, followed by the track notes.
    • Track headers (4)
      • Track ID (1) Track index
      • Instrument (1) MIDI instrument ID
      • Note count (2) Number of track notes.
    • Notes
      • Time interval from the previous note (1) in 1/240th of a second. Using time intervals instead of absolute times results in a much better compression since intervals are often repeated in music. If the interval exceeds the 8-bit limit, a 0xFF byte is inserted before the note and the resulting time is subtracted from the current note interval. Several 0xFF bytes can be added if needed.
      • Note key and type (1)
        • Note key (first 7 bits, mask 0x7F) from 0 (C0) to 96 (C8)
        • Note type (last bit, mask 0x80). In "Live mode", 1 is a note on and 0 is note off. In "Duration mode", 1 indicates the Duration is encoded in 16-bit instead of 8 which allows long notes lasting up to 1536 seconds.
      • Duration (1 or 2) in 1/255th of 6-second which is the maximum duration of an instrument sample. Only used in "Duration mode". Can be encoded on 2 bytes if the Note type is set to 1.

Live play

The live play mode is very straightforward. It consists in adding note on and note off events in a Musician.Song object while it's streaming.

Live play songs use mode 0x20.

Each note event has a "layer" which is similar to a MIDI channel and an instrument. There is one track per used layer/instrument combo.

Playing a note on event while there is already a note on ongoing for the same key, instrument and layer will first initiate a note off. There is no note on stacking.

Live keyboard

Music can be played live using the computer keyboard. Since the physical keyboard layout can't be determined automatically, the player has to configure it manually by pressing all the keyboard keys one by one.

The sustain pedal is supported using the spacebar. The Musician.Live module automatically converts sustained notes into proper note on/note off events.

Some latency may occur between the key is pressed and the sound starts playing because WoW doesn't use low latency sound drivers such as ASIO.

Band live play mode

While using the live keyboard, the player hears instant feedback but the other players will hear the music after a delay corresponding to the streaming buffer length (1 second) + a delay to deal with lag (0.5s).

The band live play mode allows players within the same party or raid to - almost - play live together. All note on/note off events are first sent individually to the party or raid group then the notes are played based on the events received from the group chat. The latency is high but it's still usable to play songs that do not require fast paced notes. Players outside the group will hear the music with the usual 1.5s delay.

MIDI keyboard

There is no way to use a MIDI device in WoW. Instead, we use a MIDI translator such as Bome MIDI translator to convert MIDI note events into PC keyboard keystrokes that can be decoded in WoW. This feature is implemented in the MusicianMIDI plugin.

Visual feedback

Events for visual feedback

The visual feedback consists in event handlers on the note on/note off activity. Check the API of the events below for more information:

VolumeMeter module

The Musician.VolumeMeter mimics a volume meter activity based on the instrument specs. It's used in the song editor and for the glowing effect in the live keyboard.

Nameplates module

When nameplates are enabled, animated notes sprites show up above the character model while playing music. This animation is handled by the nameplates module. In addition, the module adds a note icon next to the player name who also have Musician.

Table of contents

Discord / MusicianList / Patreon / Donate

Clone this wiki locally