Add interactive music support

[reduz]

This PR adds 3 types of audio streams used for interactive music support.

• AudioStreamInteractive: Allows setting several clips as sub-streams and edit the transition table between them.
• AudioStreamPlaylist: Allows sequential or shuffled playback of a list of streams.
• AudioStreamSynchronized: Allows synchronous playback of several streams, the volume of each can be controlled.

These three stream types can be combined to create complex, layered interactive music and transitions between them, similar to software such as WWise, FMOD or Elias.

As an example:

• AudioStreamPlaylist insde of AudioStreamInteractive resembles how WWise does interactive music.
• AudioStreamSynchronized inside of AudioStreamInteractive resembles how Elias does interactive music.

In Godot style, these components are presented stand alone, and can be mixed and matched by the user to achieve the desired function. This supersedes #62896 and #32769.

Screens:
Interactive music, list of clips:

Transitions:

NOTE: Functionality is missing in Tree so it cant be expanded and have the rightmost column always visible, hopfully this will be added/fixed later on.
Playlist:

Synchronizer:

Preview video:

interactive_music_take3.mp4

Sample project:

interactive_music.zip

[Mickeon]

A new PROPERTY_HINT_FLAG_LIST? Where is it used on in here?

[reduz]

@Mickeon Its used for the transitions

[fire-forge]

After this is merged, it would be nice to add an @export_flag_list annotation to GDScript (and a C# equivalent) for exporting flags with the new PROPERTY_HINT_FLAG_LIST.

[MJacred]

First few findings (will test more)

Case 1
reproduction: add AudioStreamOggVorbis stream to AudioStreamPlayer or its stream playlist (did not test other stream types)
what happens: spams modules/vorbis/audio_stream_ogg_vorbis.cpp:474 - Condition "packet_sequence.is_null()" is true. Returning: 0 to console

Case 2
Open an ogg file in advanced import window: eats 100% CPU (i.e. 1 core), normally ~8%
Set beat count and bar beats to 4, then play preview: after playing the stream to the fake ending, CPU jumps over 200%
Both high consumptions mentioned before are only released on quit to project list or quit app
maybe related warning: modules/vorbis/audio_stream_ogg_vorbis.cpp:324 - Burning negative samples doesn't make sense. Check seek algorithm.
may also make godot unresponsive (killing with SIGTERM signal with terminal still works)
EDIT: upon re-testing, the 100% cpu by just opening advanced import window is not reproducible, but the latter high consumption with incorrect beat settings persists.

Proposals

• Personally, I find it odd to enter the total beat count and bar beats instead of entering time signature, e.g. 4/4
• rename beat count to total beat count to better differentiate from bar beats, or add fitting Editor description Total beat count within the audio stream.

[MJacred]

Case 3
Create a new AudioStreamPlayer and set stream type to e.g. Playlist, then add your first stream and drag and drop in an ogg-file. First preview in "Stream" is black (which is fine) and once it's done scanning the stream, it shows a squished preview

Manual fix: select a different node in the scene and then back to the AudioStreamPlayer

Case 4
Deleting a stream/clip in Playlist, Interactive, … does not clean up the stream/clips list:
not-deleted.webm

Proposal 1
It would be nice to be able to change the stream position in the Inspector (i.e. seeking), makes testing faster/easier
For reference, this is how I implemented it in my adaptive music addon (see at top the jump button):

Proposal 2
Add option to copy Audio Streams in Playlist & Co., in case you want to switch from Synchronized to Interactive, otherwise you have to re-drag & drop all files. And then paste on new stream type

NOTE

• while playing preview in Inspector, manipulating volume db, pitch scale, playing, stream paused, mix target, bus and (un)plugging headphones worked without issue

[MJacred]

Case 5
Audio pop -> created Issue: #64775

Case 6
Ok, I get the modules/vorbis/audio_stream_ogg_vorbis.cpp:324 - Burning negative samples doesn't make sense. Check seek algorithm warning sometimes, when seeking in the advanced importer, while the stream is playing. (NOTE: BPM was disabled, but loop point was defined.)

Proposal 3
In Interactive, you can define a clip name, which is auto-generated on drop (if no clip name defined). But it is not updated in case it could; reproduction

1. drag&drop file my_test_song.ogg and results in clip name My Test Song
2. don't change name -> drag&drop file rock_song.ogg -> clip name stays unchanged

So in case name is not adjusted by user, re-generate clip name using audio file name.

[MJacred]

Case 7 (might also be a security feature to avoid breaking everything.)
Create Interactive AudioStreamPlayer, then drag&drop in clips.
Rename a Clip.
Result: Clip name references in Transitions and "initial clip" and "switch to" stay outdated.
Only workaround that works: re-opening the scene.

will test more another day…

Until then, what does AUTO_ADVANCE_RETURN_TO_HOLD do?

[reduz]

@MJacred thanks so much for testing! Transitions have the hold option so they remember which clip they come from. Return to hold just returns to that clip. This is useful for when you have the main theme playing and you want to transition to an incidental music clip, battle clip, etc. then go back to wherever you were before. The PreviousPosition option also helps to return to the previous saved state in case you want to continue from where it was (very common for the main theme).

[MJacred]

Glad to be of help and thanks for the explanation! Will check it out another day and do more testing 😪 💤

Ah, one last thing today: Another feature I really missed in the Inspector: the current TIME, i.e. the progress in seconds:milliseconds of the currently playing clip(s). A must have, especially if you add a seek helper-field (mentioned in #64488 (comment) under proposal 1)

---

EDIT2
NOTE: testing will continue on August, 23nd (did lots of hterrain testing today…)

[ghost]

Will it be available on 3.5?

[Calinou]

Will it be available on 3.5?

No, as this PR relies on backwards-incompatible changes to AudioServer.

[IntangibleMatter]

Is there any way to build this for Windows? I've been trying since the pull request came out and it consistently returns Error 1 in the last stage of compilation.

[fire-forge]

@IntangibleMatter This pull request seems to be a work in progress right now. If you look at the checks it's failing for almost all platforms.

[TokageItLab]

@reduz I provide some feedback.

---

Is it possible to set a property like Global BPM in AudioStreamInteractive and set a switching point for such like every bar or every 4 bars?

Especially in music with a clear pattern of beats, there are several points suitable for switching, like in a DJ Mix. It is better to keep the current Stream until the point after the switching command, and switch when the point is reached.

---

Can you implement something like AudioStreamRoundRobin as a variation of AudioStreamPlaylist?

I wrote a Proposal on this in godotengine/godot-proposals#5231. If we want to randomize the playback of footsteps with patterns, AudioStreamPlaylist will result in a long AudioStream. To make it more flexible and versatile, it would be excellent if you could integrate that proposal into this PR if possible.

There is already exist as AudioStreamRondomizer.

[MJacred]

Is it possible to set a property like Global BPM in AudioStreamInteractive and set a switching point for such like every bar or every 4 bars?

The time signature / bpm can be different per audio file, so having one for all of them would be detrimental in other cases…
See this wonderful lecture by Chance Thomas: https://youtu.be/q4CYUfgRdos?t=1680
Having the transition/swap on the next smallest common denominator bar would be better

so this would be a new
Proposal 4: for Transitions, add

• From Time: next common bar
• To Time: Same Position (as we wait for next common bar, this should be fine)

[MJacred]

proposal 5: mass loader
in the addon (it's coming), there's a mass loader. In this example the there are only 3 streams in that track, but we have other tracks with 10 streams/files. and so on. becomes tiresome. If you want some inspiration:

mass-loader.mp4

[MJacred]

case 8: Auto Advance enabled ignores looping -> might have found root cause, see code review
After clip 0 finishes, it auto-advances to clip 1, but it does not jump to clip 1 loop point, it jumps to START.
Therefore, Same Position should mean loop point, if stream changes clip on file end. An alternative would be to add new To Time option Loop Point. The latter would be confusing though, because we already have a To Time option Start

[IntangibleMatter]

Is it possible to set a property like Global BPM in AudioStreamInteractive and set a switching point for such like every bar or every 4 bars?

I honestly think what a good approach would be (that I've been hesitating to suggest because it would require changing some of the fundamental code of this pr, as far as I can tell) would be to be able to set what multiple of beats/bars is transitions on. I.E. a song could be able to transition on every OTHER bar because it would better match the pattern of the song.

[MJacred]

case 9: Hold Previous reduces volume dB

1. click Playing in inspector to play interactive stream player -> plays clip 0
2. switch to clip 1 in inspector

click Hold Previous (on transition for clip 0 to clip 1) at any time BEFORE starting to play or BEFORE manually switching to clip 1

This happens only on FADE_IN and FADE_CROSS

What happens: stream continues playing and reduces volume dB to reeeeally quiet. no error/warning

BUT! If you do a step 3 and switch back from clip 1 to 0, the volume dB is reset. This does only work if step 3 is a switch back to the clip where volume dB was working correctly. Switching to any other clip and then sometime back to clip 0 (where volume dB was working correctly at play start), the volume dB won't be fixed. EXCEPT you let it play on, wait for its auto-advance to trigger to another clip, then it suddenly plays volume dB correctly again. Confusing, I know.
So in short: if your clip switch mutes volume, fix it either by a) going back to previous clip where volume was fine or b) play any other clip, live with the mute, then auto-advance to another clip and volume dB is back…

[MJacred]

case 10: ReturnToHold only works AFTER transitioning at least once to a clip with no ReturnToHold

1. Play clip 0
2. let it (auto-)advance to clip 2
3. let it (auto-)advance to clip 1
4. jumps back to held clip 0 -> but this sounds like a bug 🐛 , because before clip 1, clip 2 was playing… right?

this way does NOT work, see code highlighted below in my review:

1. Play clip 0
2. let it (auto-)advance to clip 1
3. will never jump back to held clip 0

Might have found root cause, see code review

But it works, if you immediately manually switch from clip 0 to 1.

I foresee loads of misunderstandings concerning ReturnToHold and Hold Previous

Will continue testing in the coming days…

still leftover

• fading (mode, beats)
• filler
• test playing utilities with wav and mp3
• more?

[akien-mga]

Read the pinned issue ;)

But you should apply the suggestion I gave you, there's more than just running clang-format, module includes should be relative (../audio_stream_interactive.h).

[KoBeWi]

Synchronized I think its doing what its supposed to

The buzzer sound at the end is not intended I assume. It's not part of the sample, something is glitching.

[reduz]

@KoBeWi oh yeah I think I fixed this too, but feel free to test again just in case.

[akien-mga]

Looks good to me!

Should be fine to merge and give it wider testing in the next dev snapshot, so we have time to iron out the feature before 4.3 stable.

[KoBeWi]

shuffle in AudioStreamPlaylist is still bugged:

_2D.tscn-ReproNeo-GodotEngine2024-03-1216-25-32-ezgif.com-crop-video.mp4

If the playback starts with element other than the first element, it will be silent.

[AThousandShips]

Suggested change

	void _validate_property(PropertyInfo &property) const;
	void _validate_property(PropertyInfo &p_property) const;

(This isn't virtual, it's internal)

[AThousandShips]

Suggested change

	void _validate_property(PropertyInfo &property) const;
	void _validate_property(PropertyInfo &p_property) const;

[reduz]

Thanks for checking it out! will give it a last spin tonight so we can merge tomorrow.

[reduz]

Alright, shuffle is now working and applied the suggested changes. Can't believe this feature will finally get merged after 5 years in the works!

[akien-mga]

Thanks! Great work Juan, and @DanielMatarov for the initial work on this feature during GSoC 2019!

[caimantilla]

The editor AudioStreamInteractive editor window doesn't work properly on high-DPI displays. editor_scale.h was already included, so it was probably just an oversight. Here is a PR that fixes the issue: #89592

[GeminiSquishGames]

This may not be the right place or time but I've been thinking about this for a while, it's pretty important to me and maybe I should propose this somewhere else but I didn't have as much time to prepare for the interactive music support train as I thought. After playing with the new features in beta, I feel they are a bit lacking when an open source solution has existed for quite some time, but doesn't seem to get the love it deserves IMO.

Proposal for Enhancing Godot's Interactive Audio Capabilities

Current Limitations:

-Multiple AudioStreamInteractive Tracks: Can we have these play in sync? It seems more functional to have simultaneous tracks with independent buses for effects.
-Speed and Pitch Control: Can we adjust song speed without altering pitch, and transpose without affecting tempo? (I know these are not possible with the stem/individual wav file playback method, but worth highlighting.)

Suggestions:

-FMOD Studio-like Concept, good: While this might attract modern developer workflow users, Godot's past support for tracker formats was a missed opportunity. Embracing something like the libopenmpt addon could be more beneficial for interactive music.
-libopenmpt Integration, even better: This addon (https://github.com/Dudejoe870/godot-openmpt) has been invaluable in my projects recently. It uses the .mptm format, offering flexibility in music and sound effects, with everything handled by a single AudioPlayer stream resource and has GDScript functions for manipulating and getting song data and timing.
--Internal Features: It manages samples, instruments, speed, transposing, channel pan and volume within its .dll-based player and file format. Of course speed and transposition of single long sample playback still cannot be achieved without DSP of somekind, which is hard on overhead if you want it to sound good.
--Effect Buses: It supports its own effects bus and OPL soft synth instruments designed in the tracker software, though creating new bus effects is tricky if you wanted Godot mixer like effects as well. Would love to have a tracker channel be patched into Godot's mixer channels and individual effects buses.

Potential Benefits:

-Enhanced Interactivity: Official support for tracker files would simplify syncing and transitioning music tracks in Godot.
-Seamless Integration: With integration, Godot could fully leverage tracker formats, enhancing the engine's appeal to game developers. Like patching into Godot's mixer or more novel ways of playing back instruments in realtime.

Conclusion:

Leveraging open-source, free solutions like libopenmpt would offer Godot a unique, interactive audio solution born from game music concepts, aligning better with developers' needs.

For reference on licensing: https://lib.openmpt.org/libopenmpt/license/ and have a nice day.

[fire]

Microsoft Foundation Classes (MFC) are required to build OpenMPT.

I am not sure we can distribute Microsoft Foundation Classes for OpenMPT and also support our 5-10 platforms.

[GeminiSquishGames]

Microsoft Foundation Classes (MFC) are required to build OpenMPT.

I am not sure we can distribute Microsoft Foundation Classes for OpenMPT and also support our 5-10 platforms.

Pretty sure that is only for OpenMPT application, the actual tracker/daw used to make the songs. The libopenmpt library is standalone from that. Rather OpenMPT uses libopenmpt and MFC, but libopenmpt is a separate software library. I changed OpenMPT to read libopenmpt in my post.

Dependancies for libopenmpt are here: https://lib.openmpt.org/doc/dependencies.html#md_doc_libopenmpt_dependencies and I think it's pretty basic for cross-platforms, console systems aside, but they always want you to buy their DevKits as well.

The plugin at DudeJoe's repo is the best source to see and test an extension implementation with code. He was concerned that it wouldn't be accepted to the asset library due to only having a windows implementation so far, but I think the community could help with that if enough people knew what it was and what it did. https://github.com/Dudejoe870/godot-openmpt

Also, to avoid any confusion OpenMPT (the tracker that uses MFC libs) and libopenmpt are maintained together in the same repo to avoid project fragmentation, so if you look for the repo it does include the OpenMPT project as well, but the libopenmpt library source is just maintained along with it and free from other licenses other than it's own open source.

PS, also wanted to be clear that since this works as a stream resource, it can exist parallel to the new implementations which I would consider higher level simplified interactive music and sound, while libopenmpt would be a lower level alternative to something like midi that would require both midi and a soundfont reader or General Midi of the platform. Everything is self-contained in the one file type and handled by the library.

[IntangibleMatter]

@GeminiSquishGames the reality is that most people composing music for games don't use tracker software and are unlikely to want to switch. Additionally, most devs these days aren't used to using trackers for music. While there are some benefits to the approach, its applications are far less broad than the ones that have been introduced in this PR.

I think tracker support makes more sense to be implemented as a GDExtension/Module than as core to the engine.

[GeminiSquishGames]

@GeminiSquishGames the reality is that most people composing music for games don't use tracker software and are unlikely to want to switch. Additionally, most devs these days aren't used to using trackers for music. While there are some benefits to the approach, its applications are far less broad than the ones that have been introduced in this PR.

I think tracker support makes more sense to be implemented as a GDExtension/Module than as core to the engine.

I don't think the current perception reflects reality, just the popular opinion. Do most game music composers know they could be using tracker software? Have they been presented with it as an option and learned about its features? They should have the chance to try it. Tracker software scales well with modern tech and is user-friendly once you get the workflow down. If Godot made it an official resource, it would give the engine a unique feature and appeal to retro developers who love its standalone 2D rendering engine. It could lead to unique developments and innovations, which would be great for Godot. But just bringing up some points may help to make what we have here better, if it doesn't find a way to convince getting libopenmpt more support officially, and maybe simply just drumming up support for the plugin.

I've been testing beta 2, and while the features are nice, they're not groundbreaking. They're things that people have probably been doing in GDScript already. I'd certainly like to use these features and built-in support will be convenient, but it mainly organizes playlists, syncs without manual math, and transitions between stems. While pretty fundamental benefits, I don't think I would say they are broad.

I can't send each stem's sequence through its own effects bus. When I master a final song, it’s fine because the music is linear and polished, but with stems, I need effects for each track to master during dynamic playback and game. Otherwise, it sounds bad. The editors are less than basic, and I can't drag and drop multiple resources into the transition or sync stem places, or organize them well. The inspector's data layout isn't conducive to a good composition workflow.

In a tracker format, the resource type or file is the actual file the DAW creates. It can contain multiple songs, shared stems, samples, or instruments with many samples and has subtractive/FM-type synthesis via OPL. A composer can make precise changes, control panning and volume per channel, and adjust playback speed and instrument transposition without recording more tracks for that effect. They would just change the tuning or tempo without affecting each other, though long samples don't get the benefit that instruments do.

The current PR lets devs/composers use stems in very nice ways, but doesn't allow them to create procedural melodies or music on the fly with instruments. It's more like DJing than composing or allowing a composer who also has programming skills to set their imagination free. How it is can work sometimes or be made to work regardless, but the editor isn't ready for a good composition workflow or DJing without spending time making specific tools. I am having to make my own exports to help make it easier to look at and keep track of my composition. I've got the cart but have to make the pony. Trackers have established methods for creation and are easy to use, and converting MIDI to tracker format is straightforward too.

I know my passion might be hard for others to share, especially without the same exposure. I’m working on a tutorial and presentation on using the plugin DudeJoe made, which could help others see my perspective. I also need to test beta 2 more and compare its features and overhead with the official methods here and also show how they play well together. DudeJoe’s plugin works well and complements the PR so far. I just want to see tracker software thrive and be used to create amazing things in Godot and see it do more outside of the box than what major corporate engines do. Godot is a special thing and an opportunity to show what can really be done.

Please support DudeJoe if you can. His work has motivated me to chase my dreams and be ambitious again. Thanks for your feedback and time; it means a lot to me. Also I hope GDExtention gets easier to work with. We both had trouble getting things to compile at times.

[IntangibleMatter]

I don't think the current perception reflects reality, just the popular opinion. Do most game music composers know they could be using tracker software? Have they been presented with it as an option and learned about its features? They should have the chance to try it. Tracker software scales well with modern tech and is user-friendly once you get the workflow down. If Godot made it an official resource, it would give the engine a unique feature and appeal to retro developers who love its standalone 2D rendering engine. It could lead to unique developments and innovations, which would be great for Godot. But just bringing up some points may help to make what we have here better, if it doesn't find a way to convince getting libopenmpt more support officially, and maybe simply just drumming up support for the plugin.

I'm sorry to say, but I think you're too deep in the open source community. Happens to the best of us. While there are benefits to the tracker approach, very few composers, even when presented with it as an option, have an interest in changing from their current software. Programmers are rather unique in how much they change the tech they use. Most people have very little interest in ever learning new software. Ever tried to get an artist to change art programs? It's a difficult task. Additionally, Godot emphasizes keeping things as minimal as possible. QOI support was first proposed all the way back in 2021, and it's only getting added now because it's only started to be a relevant format, and because of its advantages for the movie writer feature. Trackers just aren't popular enough in the modern day, nor do they solve enough general problems for them to be implemented as part of the engine itself, sorry to say.

I've been testing beta 2, and while the features are nice, they're not groundbreaking. They're things that people have probably been doing in GDScript already. I'd certainly like to use these features and built-in support will be convenient, but it mainly organizes playlists, syncs without manual math, and transitions between stems. While pretty fundamental benefits, I don't think I would say they are broad.

Yeah, I've been following this PR since before it was officially created, and I've been advocating for a more FMOD-like system to work for general parametric audio. It's nice that it's added by default, but it's only the most basic functionality.

I can't send each stem's sequence through its own effects bus. When I master a final song, it’s fine because the music is linear and polished, but with stems, I need effects for each track to master during dynamic playback and game. Otherwise, it sounds bad. The editors are less than basic, and I can't drag and drop multiple resources into the transition or sync stem places, or organize them well. The inspector's data layout isn't conducive to a good composition workflow.

Yeah, this wouldn't be solved if trackers were added as an AudioStream, it's part of the core design of the Audio Engine.

I know my passion might be hard for others to share, especially without the same exposure. I’m working on a tutorial and presentation on using the plugin DudeJoe made, which could help others see my perspective. I also need to test beta 2 more and compare its features and overhead with the official methods here and also show how they play well together. DudeJoe’s plugin works well and complements the PR so far. I just want to see tracker software thrive and be used to create amazing things in Godot and see it do more outside of the box than what major corporate engines do. Godot is a special thing and an opportunity to show what can really be done.

Nah, I entirely get it. I've also got technologies that I love and wish were better supported! There's no harm in being enthusiastic, but part of being part of an Open Source community in my experience is not only recognizing what your needs are, but also recognizing the goals of the project, and how well those align with any given thought. If you think it would be really beneficial, create a proposal! Though I suspect that one may already exist :p

[loteque]

I was able to find a proposal for a dynamic music feature that includes OpenMPT support here: godotengine/godot-proposals#10077. The proposal seems to be a good place to make this discussion highly visible to interested parties.

godotengine/godot-proposals#10077

[GeminiSquishGames]

I'm sorry to say, but I think you're too deep in the open source community...

Heh, yeah probably. I have been for some time and I get it.

Yeah, I've been following this PR since before it was officially created, and I've been advocating for a more FMOD-like system to work for general parametric audio. It's nice that it's added by default, but it's only the most basic functionality.

FMOD is pretty great and having used it before I know there is a plugin for Godot also. Having a nice compact resource type with a DAW like editor would seem a really good idea. I also used FMOD before with tracker formats since the Core features support it but not the mptm format ATM. Edit: But also, having less external/3rd party dependencies can be good. Part of my love of the tracker, midi, or generated synth is having small data footprints.

Yeah, this wouldn't be solved if trackers were added as an AudioStream, it's part of the core design of the Audio Engine.

I think my point was that I can put effects on each channel in the tracker that are supported by libopenmpt, at least with MPTM files, so I have some amount of mastering rather than just one master track's effects. It might help if the actual Audio
StreamPlayers could sync across each other and communicate the transitions, then using the player's bus for collections or individual stems. That's how I was doing it with just timing offset manually with the buffer properties like in the docs. But there aren't as many effects in libopenmpt and it's a proprietary effect type. Making new ones would be a hassle and not as accessible to the Godot community when they already exist fine in Godot's Audio Engine.

Nah, I entirely get it. I've also got technologies that I love and wish were better supported! There's no harm in being enthusiastic, but part of being part of an Open Source community in my experience is not only recognizing what your needs are, but also recognizing the goals of the project, and how well those align with any given thought. If you think it would be really beneficial, create a proposal! Though I suspect that one may already exist :p

I probably should have split up my ideas between what I think would work better here and in that other proposal, and I see that one existsts...

I was able to find a proposal for a dynamic music feature that includes OpenMPT support here: godotengine/godot-proposals#10077. The proposal seems to be a good place to make this discussion highly visible to interested parties.
... here:
godotengine/godot-proposals#10077

Thank you for taking time to address my ideas in length. Hopefully, we can work more on the plugin and get some more features into it that make it unique. I've been a dev/composer off and on for 20 years and have done some audio direction work with indies that make retro games and I keep trying to evangelize for Godot at conventions, and have converted a few devs for neo-retro projects. A lot of the work is with old 8 and 16 bit consoles though and getting them on cartridges. That's a whole other world that probably has a bit of influence on me too. ;) Thanks again.