Add an absolute time (DSP time) feature to play audio effects at specific intervals

[rezgi]

Describe the project you are working on:
A rhythm game where all events and animations are synced to MIDI data

Describe the problem or limitation you are having in your project:
I struggle to track and trigger events based on musical data (tempo). Since I can only execute logic in _process() and _physics_process(), I depend on the framerate while I need to trigger an event at exactly 2 seconds for example. But it's never 2s precisely and I constantly compensate the delay when the tempo (bar or beat) happens between 2 frames.

Other engines have APIs like dspTime (Unity) or TimeSynth (Unreal) to manage that and I'm wondering if it's achievable in Godot. I don't know about the complexity of implementation, from what I understand these precise timing methods seem to run on their own thread I guess ?

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
By having access to an API that can trigger events on precise timings like 2s (00:02:0000) which will allow easier programming for musical mechanics. Game logic will still run on main thread but at least we'll have another parallel logic that we can rely on to track the progression of the tempo for example.

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
if get_absolute_time() == 2 print("Bar 2 reached")

If this enhancement will not be used often, can it be worked around with a few lines of script?:
There's no way to get absolute time and we need to check for approximations, which is ok for music playback and simplistic rhythm games but becomes hard when we want to deeply sync rhythm and game mechanics.

Is there a reason why this should be core and not an add-on in the asset library?:
I think this kind of implementation really depends on the engine low-level process and I guess it needs an alternative thread to run this implementation. Maybe a C++ plugin could do that ? I'm not advanced enough to know.

[Jummit]

There is OS.get_ticks_msec(), if that works for you.

[Calinou]

Related to godotengine/godot#32382.

[rezgi]

There is OS.get_ticks_msec(), if that works for you.

Indeed but where do I call it and check it ? We can run logic only in process() functions so we stay dependent on delta duration, which means that when I'll check if OS.get_ticks_msec() equals 2s for example, in _process(), time may already be 2.001 for example.

Related to godotengine/godot#32382.

Indeed this is quite related. It's about finding a way to trigger events at an absolute given time even though the game runs on an irregular tick (delta). Unity provides dspTime and we have to manually code the logic to preload sounds when it gets close to this absolute time for example. And Unreal provides TimeSynth that tackles this issue more precisely and out of the box.

I guess using OS.get_ticks_msec() and anticipate it should be enough, having events trigger on absolute time maybe isn't the right approach for a game I guess ?

[starry-abyss]

Isn't it already available?

https://github.com/godotengine/godot-demo-projects/tree/master/audio/bpm_sync
https://www.youtube.com/watch?v=kXG1jXc8k2k

[kerskuchen]

I think @starry-abyss is correct in that what @rezgi wants to do is already available (you can look at https://docs.godotengine.org/de/stable/tutorials/audio/sync_with_audio.html for the corresponding tutorial).

The problem is though, that as soon as @rezgi decides to play a sound at an exact moment in time (i.e. at a specific beat), it may be played out of sync by up to n milliseconds (where n is the audio buffer size??) and it may sound awful. Here is a GDC talk about this exact problem https://gdcvault.com/play/1017877/The-Audio-Callback-for-Audio.

The solution for this would be a new API like Unity's AudioSource.PlaySheduled. @rezgi you may get away with godotengine/godot#32382 (comment) from the related issue @Calinou mentioned.

[rezgi]

Thanks @kerskuchen for the info. As I was working on a DAW-like metronome in Godot, I found it was quite constraining to depend on process() to check for the exact timing, so I was thinking of APIs like Unity's PlayScheduled or Unreal's TimeSynth that try to solve this specific problem.

Seems like the other issues have pointed that problem too, as I'm not technically advanced in the matter, I don't know how this feature could be implemented aside from having another thread only for audio.

[benjarmstrong]

I'm also working on a music/rhythm game with events synced to MIDI data and have experienced the same problem.

There is OS.get_ticks_msec(), if that works for you.

This is exactly what I use, but alone it wasn't enough, because...

The problem is though, that as soon as @rezgi decides to play a sound at an exact moment in time (i.e. at a specific beat), it may be played out of sync by up to n milliseconds (where n is the audio buffer size??) and it may sound awful. Here is a GDC talk about this exact problem https://gdcvault.com/play/1017877/The-Audio-Callback-for-Audio.

...so a "close enough" solution for my game was to:

• Reduce the buffer size (yes, this doesn't exactly solve the problem but it was good enough for my use case).

• Modify the engine source for improved audio latency; here's what I did:

  • For Godot 4.0 see my PRs 38280 and 38210. These PRs are still pending review so use them at your own risk.
  • For Godot 3.2 you can grab a copy of the build I'm using here; it's just my 4.0 PRs backported to 3.2. It should be relatively trivial to rebase my commits to a newer 3.2 version (I may do this eventually at that repo as my game gets closer to launch). This isn't as well tested as the 4.0 PRs because I'm assuming these PRs won't be backported to 3.2, but it appears to compile and work on Windows 10, Windows 7 and Linux.

This was enough to solve the problem in my use case - I hope it's enough to solve yours.

The solution for this would be a new API like Unity's AudioSource.PlaySheduled. @rezgi you may get away with godotengine/godot#32382 (comment) from the related issue @Calinou mentioned.

I like this idea.

[rezgi]

I'm also working on a music/rhythm game with events synced to MIDI data and have experienced the same problem.

There is OS.get_ticks_msec(), if that works for you.

This is exactly what I use, but alone it wasn't enough, because...

The problem is though, that as soon as @rezgi decides to play a sound at an exact moment in time (i.e. at a specific beat), it may be played out of sync by up to n milliseconds (where n is the audio buffer size??) and it may sound awful. Here is a GDC talk about this exact problem https://gdcvault.com/play/1017877/The-Audio-Callback-for-Audio.

...so a "close enough" solution for my game was to:

• Reduce the buffer size (yes, this doesn't exactly solve the problem but it was good enough for my use case).

• Modify the engine source for improved audio latency; here's what I did:

  • For Godot 4.0 see my PRs 38280 and 38210. These PRs are still pending review so use them at your own risk.
  • For Godot 3.2 you can grab a copy of the build I'm using here; it's just my 4.0 PRs backported to 3.2. It should be relatively trivial to rebase my commits to a newer 3.2 version (I may do this eventually at that repo as my game gets closer to launch). This isn't as well tested as the 4.0 PRs because I'm assuming these PRs won't be backported to 3.2, but it appears to compile and work on Windows 10, Windows 7 and Linux.

This was enough to solve the problem in my use case - I hope it's enough to solve yours.

The solution for this would be a new API like Unity's AudioSource.PlaySheduled. @rezgi you may get away with godotengine/godot#32382 (comment) from the related issue @Calinou mentioned.

I like this idea.

Thank you very much for the detailed answer. I also reduced the audio buffer and even set the FPS to 120. It mainly solves the issue but I also wanted to suggest adding a playScheduled feature to Godot for improvement.

[benjarmstrong]

Thank you very much for the detailed answer. I also reduced the audio buffer and even set the FPS to 120. It mainly solves the issue but I also wanted to suggest adding a playScheduled feature to Godot for improvement.

I second the need for a PlayScheduled feature. Even after all my changes I'd still prefer better audio synchronisation in my game.

Another point is my solution doesn't completely solve the problem - it merely reduces it's effects. In projects with complex mixer arrangements it may not be feasible to lower the audio latency below a certain point, in which case my solution isn't very helpful.

[aleksfadini]

I also second better audio synchronization with scheduling. Had to cave in and move to unity for one specific app requiring that.
For musical apps those work-arounds are insufficient.

[fire]

I think "scheduled" audio play makes more sense than getting absolute time. Do you think it should be a separate proposal and can you write it?

[mortarroad]

Ah, I see this here is a bit more elaborate.

Any thoughts on AudioStreamPlayer::get_playback_position() missing synchronization? (As mentioned over here: #2603)

[fire]

Here are my working notes. https://gist.github.com/fire/b9ed7853e7be24ab1d5355ef01f46bf1

[Calinou]

As a workaround, you can disable V-Sync to make it possible to run per-frame events with more accurate timing. On simple 2D games, it should be possible to reach hundreds of FPS on modern mid-range desktop hardware. This has many caveats however:

• The slower the system, the less accurate timings will be. Window size and graphics settings will affect the framerate, and therefore timer precision.
• Timing precision will be inconsistent unless you enforce a FPS cap with Force Fps and ensure the system can keep up with it.
• If V-Sync is forced by the operating system (which is the case on Android, iOS and HTML5), this will not benefit timings at all.

[NoodleSushi]

A PlayScheduled feature would've been useful and save my time the last time I needed it. I've developed a rhythm game where I needed audio to start playing at a specific later time. Since I couldn't input negative values on AudioStreamPlayer.play() or use a PlayScheduled alternative for Godot, I had to add 100ms of silence at the beginning of every audio clip I want to play later, then calculate the offset and play at an earlier time.

[gmamekudz]

Hi, I'm a Unity refugee and am currently reviewing how to transfer my audio library into Godot. I am surprised that there is no way to play audio files with Sustian, Loop and Release. In Unit I have my own functions which then realize this with SchedulePlay. As already written here, such functions would be very desirable, either direct DSP functions or at least an audio function where at a LoopStop the sound is still played until the end of the file. Even better would be a support of sound atlases, where then the start, LoopStart, LoopEnd and SoundEnd positions are defined.

[kotx]

This would be awesome :)
Something like an audio callback for when chunks are played (like OnAudioFilterRead in Unity) would help with variations in _process too.