MIDI audio is delayed after suspending the server

[daveyarwood]

I get pretty much immediate responses from the server* after first running it, but after leaving the server running for a while (e.g. if I leave my computer and come back hours later), it may take several seconds or more before I hear anything. Restarting the server fixes it.

*meaning, if I run alda play -r -c 'piano: c', I hear the note immediately

No idea what the issue is at this point, but figured I would report it in case anyone has any ideas.

[daveyarwood]

This might be fixed now that we've switched to JSyn for event scheduling. I'll do some testing to see if I can reproduce this bug while running the latest version of Alda on the master branch.

[daveyarwood]

Yep, problem solved after switching to JSyn!

[daveyarwood]

Nope -- I spoke too soon. This still seems to happen, possibly related to closing and later re-opening my Macbook.

[daveyarwood]

Just a stab in a dark, but this might be related to some of the top-level things I've defined, like *midi-synth*, getting def'd multiple times without being closed. Should try using defonce wherever it makes sense.

[daveyarwood]

I'm still reproducing the issue on my Macbook by doing this:

1. Start an Alda server, play a simple score like alda play -c 'piano: c8 e'. I hear the two notes immediately.
2. Close laptop.
3. Re-open laptop. Play the same score. Now there is a noticable delay of seconds before hearing the two notes.

[firesofmay]

So here's what I did:

• Forked Alda
• Ran boot dev -a server --port 27713 --alda-fingerprint to start the alda server
• Ran boot dev -a repl to start the repl
• Played this simple score piano: d d d a g a f g a+ a | r | d d d a g a f g f e | r | d e c d e c d f e | r | d e c d e c f e d
• It loaded immediately
• Closed the laptop, opened it again, played the score, it took a second or two to play. So I was able to reproduce it.
• Closed both repl and server
• Searched for :dynamic keywords and made them defonce.
• Started server and repl again like above.
• Played the score
• Closed the laptop and started again.
• Played the score again, same lag.

Looks like defonce is not the issue.
Am I doing this right? Did I miss anything?
Btw I am totally new to boot, (Only used lein so far) so bear with me.
Let me know if you have any other ideas on what could be wrong.

Thanks.

[firesofmay]

I did what you did as well,

• Played a note without using repl via: alda play -c 'piano: c8 e'
• It also has a lag after closing the laptop.

Also I noticed that if you keep playing the same note after closing the laptop, after a while the lag goes away.

[daveyarwood]

@firesofmay Yep -- you're doing it right. It does sound like defonce is not the issue.

That's very interesting that the lag goes away after a while. It almost seems like either the JSyn SynthesisEngine or the Java MIDI Synthesizer instances are "falling asleep" or dying when the laptop is closed... It does take a few seconds for a MIDI synth to start up, so I wonder if maybe that's related.

[firesofmay]

@daveyarwood Yeah could be. But how I am not sure how to confirm that?

[daveyarwood]

Hmm... this gets a bit complicated, but I think you would need to run the Alda server in a way that you can access it via a Clojure REPL. There are two ways you could do that:

1. Modify the boot dev task in build.boot so that it also starts up a REPL server, then connect to it via boot repl -c -p $PORT (where $PORT is the port on which the REPL server is running)
2. Run boot dev with a socket server REPL and connect to it using telnet.

The second way is probably easier -- that link has a good walkthrough on how to use a socket server REPL in Clojure 1.8.0 (the version we're using for Alda).

Once you're in a REPL, you'll be able to inspect and interact with anything in any of Alda's Clojure namespaces.

One thing to check would be whether the MIDI synthesizer instances in the *midi-synth-pool* are still open after closing the laptop. These MIDI synthesizer instances are the ones that the Alda server grabs each time it plays a new score, so if the first one it grabs is not open, then that might explain the delay.

[firesofmay]

Okay. Will check this on weekend and get back to you.

On Wed, Jun 1, 2016 at 8:00 PM, Dave Yarwood notifications@github.com
wrote:

Hmm... this gets a bit complicated, but I think you would need to run the
Alda server in a way that you can access it via a Clojure REPL. There are
two ways you could do that:

Modify the boot dev task in build.boot so that it also starts up a
REPL server, then connect to it via boot repl -c -p $PORT (where $PORT
is the port on which the REPL server is running)
2.

Run boot dev with a socket server REPL
http://clojure.org/reference/repl_and_main#_launching_a_socket_server
and connect to it using telnet.

The second way is probably easier -- that link has a good walkthrough on
how to use a socket server REPL in Clojure 1.8.0 (the version we're using
for Alda).

Once you're in a REPL, you'll be able to inspect and interact with
anything in any of Alda's Clojure namespaces.

One thing to check would be whether the MIDI synthesizer instances in the
midi-synth-pool
https://github.com/alda-lang/alda/blob/master/server/src/alda/sound/midi.clj#L21
are still open
https://docs.oracle.com/javase/7/docs/api/javax/sound/midi/MidiDevice.html#isOpen()
after closing the laptop. These MIDI synthesizer instances are the ones
that the Alda server grabs each time it plays a new score, so if the first
one it grabs is not open, then that might explain the delay.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#160 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AA2SwgKDv3JXHowpWdz7VSanCiG9QHaXks5qHZeKgaJpZM4G7g4r
.

[firesofmay]

Sorry, Didn't get time to investigate this. Had some office work left.
I'll see if I can get some time out for this. Will update when I make some progress.

[daveyarwood]

I had an interesting observation:

• Last night, I started up an Alda server in debug mode, so I could see when the MIDI notes are playing. When playing notes, I see debug messages like this appear at the exact same time as I hear the notes:

  16-Aug-14 10:37:38 skeggox.local DEBUG [alda.sound.midi] - Playing note 60 on channel 0.
  16-Aug-14 10:37:38 skeggox.local DEBUG [alda.sound.midi] - MIDI note off: 60
  16-Aug-14 10:37:38 skeggox.local DEBUG [alda.sound.midi] - Playing note 62 on channel 0.
  16-Aug-14 10:37:38 skeggox.local DEBUG [alda.sound.midi] - MIDI note off: 62
  16-Aug-14 10:37:39 skeggox.local DEBUG [alda.sound.midi] - Playing note 64 on channel 0.
  16-Aug-14 10:37:39 skeggox.local DEBUG [alda.sound.midi] - MIDI note off: 64
  

• When I went to bed, I left the server running and closed the laptop lid.

• This morning, I re-opened it, then after a while, I made a few requests to the server to play a few notes on piano.

• The behavior is the same as what we've been seeing -- there is an unexplained delay -- but what I noticed was that the debug messages were still appearing, in rhythm, right away! Then I would hear the audio seconds later. So it seems like the MIDI events are happening, but the audio is delayed for some reason.

I'm really scratching my head over this. I'm tried googling for some explanation as to how this can happen, but I can't seem to find anything about it. I think I may ask this question on StackOverflow, and see if there is someone out there who happens to know what happens when you have a MIDI synthesizer on a running JVM and you leave the program running and close your laptop lid.

[daveyarwood]

I've asked this question on StackOverflow: http://stackoverflow.com/questions/38944107/java-midi-audio-is-delayed-after-laptop-comes-out-of-hibernation

I also put together a simple Java program that demonstrates the issue: https://github.com/daveyarwood/java-midi-delayed-audio-example

So it's definitely not anything Clojure-specific. If we can figure out how to fix the problem in Java, then we can use that knowledge to fix it in Clojure for Alda.

[jgkamat]

I messed around with your example code and I wanted to let you know the issue wasn't so much in the computer going to sleep, it's the process being 'suspended' for a period of time. You can reproduce this without sleeping (on debian, running in bash) by:

1. Running your example program
2. Pressing ctrl-z to 'suspend' the process playing notes
3. wait a couple of seconds
4. run the fg command to bring it back into the foreground
5. Press enter to hear delayed notes

I guess the java midi synth just isn't happy when it's suspended for any reason. Sounds like this should be submitted as a java bug and fixed upstream? I'm not sure if it's possible to workaround this (perhaps you could somehow intercept waking up and restart the synth?)

Glad to know it's not an issue in alda though 😄

[daveyarwood]

Ah, that's good to know! I think we're getting closer to figuring this out.

I've filed a bug at http://bugreport.java.com -- currently awaiting review (EDIT: here it is)

In my example code, there is a single MidiSynthesizer instance being reused, which I think is part of the problem. I did try changing my example so that each time you press Enter, a new MidiSynthesizer instance is created and used, and that does seem to help. Here is the alternate version on a branch.

It's inconvenient to do it this way though, because it takes each MidiSynthesizer instance at least a few seconds to initialize and be ready to play. If you try and use it immediately after calling .open() on it, the timing of the notes ends up being choppy/stuttered. This is the reason I would prefer to have a pool of MidiSynthesizer instances open and ready in the background, so the Alda server can just grab one and use it immediately whenever you ask it to play a score.

As a workaround until this gets fixed upstream (assuming that ever happens), I wonder if there is any way for the server to quickly check to see if it is in this state, and if so, it can return a helpful explanation message to the client and then start up a fresh pool of MidiSynthesizer instances before playing the score? Better yet, the server could periodically check for this scenario in the background and quietly fix itself.

[daveyarwood]

A possible solution has been posted on SO: http://stackoverflow.com/questions/38944107/java-midi-audio-is-delayed-after-laptop-comes-out-of-hibernation/38948413#38948413

TODO: try this out and see if it works... I'm also curious how many people have the Sun implementation of the MIDI Synthesizer. I'm not even sure if I do or not.

[daveyarwood]

I just tried the solution above with my Java example (here's the modified code on a branch) and it seems to be a great workaround!

Next steps:

• Translate it to Clojure and use it as a replacement for this code that opens a Synthesizer instance.
• Test thoroughly to make sure there aren't adverse side effects to disabling jitter correction.

[daveyarwood]

I have this change on a branch and I tried it out. It does solve the issue of delayed audio after suspending and bringing back the server process, but I'm noticing an obvious instability in the timing of the notes, which I find unacceptable. I think jitter correction is something we need... it's just a shame that it carries this bug with it.

Even if the bug I filed with Java is fixed, we won't reap the benefit until it makes it into a new version of Java, and then we can't expect everyone using Alda to be on that new version, so I think we need to explore another tactic.

I've been playing around with a "one server, multiple workers" architecture for #243 using ZeroMQ. Once we have that in place, I'm thinking the solution for this issue might be to watch for workers getting suspended, and quietly replace them. Each worker could send a heartbeat to the server every X seconds, and if the server doesn't get a heartbeat from a worker within a certain timeframe, then it could assume that the process was suspended or something else went wrong, and replace that worker.

[jimcheetham]

You might want to look into the style of Erlang, which is a language
especially suited for multiple workers, with a 'die early' philosophy to
keep real-time goals (i.e. telephone exchange software).

On Mon, Aug 22, 2016 at 2:00 PM, Dave Yarwood notifications@github.com
wrote:

I have this change on a branch
https://github.com/alda-lang/alda/blob/disable-jitter-correction/server/src/alda/sound/midi.clj#L17-L27
and I tried it out. It does solve the issue of delayed audio after
suspending and bringing back the server process, but I'm noticing an
obvious instability in the timing of the notes, which I find unacceptable.
I think jitter correction is something we need... it's just a shame that it
carries this bug with it.

Even if the bug I filed with Java is fixed, we won't reap the benefit
until it makes it into a new version of Java, and then we can't expect
everyone using Alda to be on that new version, so I think we need to
explore another tactic.

I've been playing around with a "one server, multiple workers"
architecture for #243 #243
using ZeroMQ. Once we have that in place, I'm thinking the solution for
this issue might be to watch for workers getting suspended, and quietly
replace them. Each worker could send a heartbeat to the server every X
seconds, and if the server doesn't get a heartbeat from a worker within a
certain timeframe, then it could assume that the process was suspended or
something else went wrong, and replace that worker.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#160 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAd7ixUXqQowJj-c6LoUdlBDfHGYU1Zpks5qiQLXgaJpZM4G7g4r
.

[daveyarwood]

I think 1.0.0-rc35 inadvertently fixed this issue 😄

As of 1.0.0-rc35, a server will "lay off" a worker process if it hasn't gotten a heartbeat from it in 3+ seconds. The worker process stops receiving heartbeats from the server and shuts itself down. Meanwhile, the server maintains the desired number of worker processes by spawning new ones, so the server will notice that it's low on workers and create more.

I left my Alda server running at the end of the day yesterday and shut my laptop (suspending all the processes). When I opened it this morning, there was a brief period of sluggishness caused by there being "11/4" worker processes available (yikes!), but then the stale workers shut themselves down and I had 4/4 workers available again. I ran an alda play command repeatedly, and got immediate playback each time.

I'll leave this issue open for now -- we can improve this by teaching the server to recognize when it's been suspended (easily done*), and when that happens, kill and replace all the workers. That way we can avoid having a bunch of stale workers lingering at the same time the server is trying to start new ones.

* The server can keep a reference to the last time it did its message-handling loop (this happens constantly). At the end of the loop it can update the "last run" time. At the beginning of the loop, it can check the "last run" time against the current system time. If it's been more than say, 10 seconds, we can conclude that the server and workers may have been suspended.

[daveyarwood]

Fixed-ish in 1.0.0-rc38.

Now the server will cycle out the workers if it detects it hasn't been active in 10+ seconds. Workers will also shut down if they detect they haven't been active in 10+ seconds.

I say "fixed-ish" because the problem still may be observable if you do something like close your laptop lid and then re-open it in under 10 seconds. If you ever observe this problem you can fix it by running alda downup to restart the server and workers.

[daveyarwood]

Unfortunately, I'm not seeing the worker restart happening anymore after upgrading to macOS Sierra (from Mavericks). If I close my laptop and reopen it hours later, the delayed audio problem is still there. I suspect that since the upgrade, my OS is doing something differently re: suspending processes when it goes into sleep mode.

I think maybe a better workaround would be to implement a life span (maybe 15-30 mins? we could randomize it so that the workers don't all quit at the same time) for worker processes, after which time the worker quits and the server spawns a new one to replace it. This would at least ensure that if your computer is awake for long enough (i.e. 15+ minutes), you are guaranteed to have a worker available that isn't stale from before your computer woke up. I'll file a new issue in alda-server-clj for this idea.