Hydrogen crashes with "Can not reroute empty drumkit paths"

[emilaxelsson]

Hydrogen version * : 1.2.2
Operating system + version : Ubuntu 23.04
Audio driver + version : Jack

---

Error message:

(E) Filesystem::rerouteDrumkitPath Can not reroute empty drumkit paths
(E) ::void handleFatalSignal(int) Fatal signal 11
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogrEfY4a/AppRun.wrapped() [0x68636a]
(E) ::void handleFatalSignal(int) /lib/x86_64-linux-gnu/libc.so.6(+0x3c4b0) [0x7fd60743c4b0]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogrEfY4a/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core4ADSR9applyADSREPfS1_iif+0x6b1) [0x7fd60a9eaf7d]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogrEfY4a/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core7Sampler20renderNoteNoResampleESt10shared_ptrINS_6SampleEEPNS_4NoteES1_INS_17SelectedLayerInfoEES1_INS_19InstrumentComponentEES1_INS_16DrumkitComponentEEiiffff+0x674) [0x7fd60ab59848]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogrEfY4a/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core7Sampler10renderNoteEPNS_4NoteEj+0x17cb) [0x7fd60ab576f9]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogrEfY4a/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core7Sampler7processEj+0x704) [0x7fd60ab5473c]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogrEfY4a/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core11AudioEngine12processAudioEj+0x104) [0x7fd60a9b178e]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogrEfY4a/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core11AudioEngine19audioEngine_processEjPv+0x7d9) [0x7fd60a9b1487]
(E) ::void handleFatalSignal(int) /lib/x86_64-linux-gnu/libjack.so.0(+0x14a0d) [0x7fd60af6ca0d]
(E) ::void handleFatalSignal(int) /lib/x86_64-linux-gnu/libjack.so.0(+0x146d8) [0x7fd60af6c6d8]
(E) ::void handleFatalSignal(int) /lib/x86_64-linux-gnu/libjack.so.0(+0x31650) [0x7fd60af89650]
(E) ::void handleFatalSignal(int) /lib/x86_64-linux-gnu/libc.so.6(+0x8f18a) [0x7fd60748f18a]
(E) ::void handleFatalSignal(int) /lib/x86_64-linux-gnu/libc.so.6(+0x11dbd0) [0x7fd60751dbd0]

My song plays fine on Hydrogen 1.1.1. The problem has been for all versions after that (though I'm not sure whether the error message has been the same).

I attach my song attack.txt (changed to .txt so that GitHub would accept it). I've done manual edits to the drumkit XML, and it's possible that I've screwed something up.

Unfortunately, I can't include the samples. They add up to 147 MB and contain copyrighted samples.

[theGreatWhiteShark]

AFAICS the error

(E) Filesystem::rerouteDrumkitPath Can not reroute empty drumkit paths

is not doing any harm at all. (In version 1.2.0 Hydrogen introduced a replaced the element drumkitLookup with drumkitPath. The latter stores the absolute path to a kit which allows to have an arbitrary number of kits sharing the same name in your system. But as the system kits are mounted when using the AppImage, their paths differ between individual runs. Thus Hydrogen attempts to reroute them but is not able to find the element mentioned.) I have to check whether this message is just a false positive or if drumkit loading is indeed affected.

I can not reproduce the crash though.

When does the crash occur? During startup or while running playback? Does it also occur when using a different audio driver (since the stacktrace suggests the crash might happen in the JACK server)?

I've done manual edits to the drumkit XML, and it's possible that I've screwed something up.

Maybe. But Hydrogen must never crash regardless of what you enter in there. Could you try saving (as) the song (doing so with version 1.1.1 would be fine too)? This should smooth out erroneous parameters introduced during manual edit.

Unfortunately, I can't include the samples. They add up to 147 MB and contain copyrighted samples.

Attaching the drumkit.xml without samples would be fine too. But we should rule out a problem with JACK first (as it seems more likely to me).

Could you run

jackd --version

[emilaxelsson]

When does the crash occur? During startup or while running playback?

During playback. It always crashes at the same place when the song is played from the beginning. I tried to skip forward to just before where the crash happens and then it manages to play through it, but crashes a bit later.

Does it also occur when using a different audio driver (since the stacktrace suggests the crash might happen in the JACK server)?

Interesting -- PulseAudio seems to do better, though I hear a small pause where the crash usually happens. After some repeating at the problematic part I got a crash with PulseAudio too (at the same place in the song). Here is the log:

(E) Filesystem::rerouteDrumkitPath Can not reroute empty drumkit paths
(E) ::void handleFatalSignal(int) Fatal signal 11
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/AppRun.wrapped() [0x68636a]
(E) ::void handleFatalSignal(int) /lib/x86_64-linux-gnu/libc.so.6(+0x3c4b0) [0x7f9ea663c4b0]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core4ADSR9applyADSREPfS1_iif+0x6b1) [0x7f9ea9beaf7d]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core7Sampler20renderNoteNoResampleESt10shared_ptrINS_6SampleEEPNS_4NoteES1_INS_17SelectedLayerInfoEES1_INS_19InstrumentComponentEES1_INS_16DrumkitComponentEEiiffff+0x674) [0x7f9ea9d59848]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core7Sampler10renderNoteEPNS_4NoteEj+0x17cb) [0x7f9ea9d576f9]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core7Sampler7processEj+0x704) [0x7f9ea9d5473c]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core11AudioEngine12processAudioEj+0x104) [0x7f9ea9bb178e]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core11AudioEngine19audioEngine_processEjPv+0x7d9) [0x7f9ea9bb1487]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core16PulseAudioDriver21stream_write_callbackEP9pa_streammPv+0xa0) [0x7f9ea9ce7450]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libpulse.so.0(+0x2bd8e) [0x7f9ea5a2bd8e]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libpulsecommon-8.0.so(pa_pdispatch_run+0xe2) [0x7f9e9f83b422]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libpulse.so.0(+0xedbe) [0x7f9ea5a0edbe]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libpulsecommon-8.0.so(+0x3dd3f) [0x7f9e9f83dd3f]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libpulsecommon-8.0.so(+0x40474) [0x7f9e9f840474]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libpulse.so.0(pa_mainloop_dispatch+0xc7) [0x7f9ea5a23e77]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libpulse.so.0(pa_mainloop_iterate+0x3c) [0x7f9ea5a2427c]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libpulse.so.0(pa_mainloop_run+0x20) [0x7f9ea5a24320]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core16PulseAudioDriver11thread_bodyEv+0xf0) [0x7f9ea9ce710c]
(E) ::void handleFatalSignal(int) /tmp/.mount_HydrogAuLnpW/usr/bin/../lib/libhydrogen-core-1.2.2.so(_ZN6H2Core16PulseAudioDriver13s_thread_bodyEPv+0x20) [0x7f9ea9ce6fca]
(E) ::void handleFatalSignal(int) /lib/x86_64-linux-gnu/libc.so.6(+0x8f18a) [0x7f9ea668f18a]
(E) ::void handleFatalSignal(int) /lib/x86_64-linux-gnu/libc.so.6(+0x11dbd0) [0x7f9ea671dbd0]

Could you try saving (as) the song (doing so with version 1.1.1 would be fine too)?

Tried that and it didn't change the file. Actually, the song file has never been manually edited, only the XMLs for the drumkits I pulled in when I first created it.

Could you run

jackd --version

It prints: jackdmp version 1.9.21 tmpdir /dev/shm protocol 9

[emilaxelsson]

I've managed to isolate the problematic part. I attach a bundle with the entire data directory (Hydrogen_data.zip) including a few samples. Hopefully you can reproduce the bug from that.

With this shortened song, I sometimes have to repeat the patterns a couple of times before the crash happens. It usually happens at the end of the third pattern, but sometimes in other places as well.

[emilaxelsson]

I noticed something else: It sounds a bit different in 1.2.2 compared to 1.1.1. You can hear the difference in the files in
exported_sounds.zip. At around 4.5 seconds, you can hear two overlapping notes in attack_stripped_1.2.2.ogg. This overlap does not occur in attack_stripped_1.1.1.ogg. Incidentally, this is at the place where Hydrogen usually crashes.

[theGreatWhiteShark]

Hopefully you can reproduce the bug from that.

Hmm. Unfortunately not. Using your specific version of JACK didn't do it either. Did you by any chance try to compile Hydrogen from source on the 1.2.2 tag? Maybe it's the AppImage that is not working properly.

But thanks a lot for all these resources!

I noticed something else: It sounds a bit different in 1.2.2 compared to 1.1.1. You can hear the difference in the files in
exported_sounds.zip. At around 4.5 seconds, you can hear two overlapping notes in attack_stripped_1.2.2.ogg. This overlap does not occur in attack_stripped_1.1.1.ogg. Incidentally, this is at the place where Hydrogen usually crashes.

That's quite interesting. I can hear this overlap as well. I'll check what is going on there. (I rewrote most parts of the audio engine for 1.2.0 to kill a number of long standing bugs. Maybe something went wrong)

[theGreatWhiteShark]

I was able to reproduce the issue using a Ubuntu 23.04 live system. It will need a couple of days as I need to setup an image, toolchain etc. but I will look into it.

[emilaxelsson]

That's great, thanks for looking into it!

[cme]

Does this make use of Hydrogen's ADSR enveloping? The implementation of that changed (from an odd polynomial approximation, to a standardish logarithmic implementation) which would result in audible differences. It's a rarely-used feature so might be less well-tested than others, though having said that I'm not sure there's much scope for it to actually cause a SEGV.

[cme]

Does this make use of Hydrogen's ADSR enveloping? The implementation of that changed (from an odd polynomial approximation, to a standardish logarithmic implementation) which would result in audible differences. It's a rarely-used feature so might be less well-tested than others, though having said that I'm not sure there's much scope for it to actually cause a SEGV.

Aha, yep, it has maximum sustain. I've reproduced some bad behaviour on Mac which I'll look into if I have spoons (having a deliberate day off from regular work for the first time in aeons). I guess one side effect of the new implementation is that notes fade out longer so we're probably seeing a pile up of notes which is hitting some otherwise-unexercised conditions around max polyphony.

Hmm. Although since you have auto note off on the instruments they shouldn't release longer than a single pattern anyway. Hmm.

Thank you for an excellent test case!

[theGreatWhiteShark]

Does this make use of Hydrogen's ADSR enveloping?

No, it doesn't look like it.

I think the crash might be an Ubuntu-specific issue. It's the only distro I'm aware of that does not ship the libfuse version 2 required by AppImage, so you have to extract the contained SquashFS and run the contained app manually. Maybe there are more things which have to be treated differently. (Evil tongues whisper such issues might be related to Canonical trying to push their own snap tool)

The overlap around second 4,5, however, is probably an issue introduced while rewriting the audio engine (of which we had surprisingly little yet :D )

[cme]

Yeah my theory about this was based on just trying it without thinking too much, PLUS forgetting that I had a bunch of debug tracing in my local build that messed things up :D

I have however reproduced a crash on current master with macOS, letting this run over and over for a few minutes, it eventually crashes out in the sampler trying to access a Note through a null pointer. Which is pretty odd for a shared_ptr.

It does seem to happen around the notes with explicit length set at the end of the third pattern. At a guess, maybe the Notes are being destroyed at the end of the "note length" even though the Release period hasn't expired?

[theGreatWhiteShark]

That's indeed possible. Very long samples in combination with custom note lengths are an edge case we might not have tested yet.

I'll still go down the Ubuntu route. At least for a little. It bothers me that the same AppImage runs flawless on my local Devuan but crashes immediately on the Ubuntu live image.

[cme]

I spent a little time on this on Tuesday and found that renderNote[No]Resample sometimes calculates a negative note length because computeFrameFromTick and Note::getNoteStart are inconsistent somehow. This gets passed to the ADSR code which immediately moves to Release (because it's past the end of the note... very far past the end of the note!) which doesn't consider this possibility and tramples past the beginning of the buffer, with predictably unpredictable results.

[theGreatWhiteShark]

I spent a little time on this on Tuesday and found that renderNote[No]Resample sometimes calculates a negative note length because computeFrameFromTick and Note::getNoteStart are inconsistent somehow. This gets passed to the ADSR code which immediately moves to Release (because it's past the end of the note... very far past the end of the note!) which doesn't consider this possibility and tramples past the beginning of the buffer, with predictably unpredictable results.

Ah, nice lead! The notes in both the sampler and the audio engine note queue do indeed "time travel" in case tempo or song size changes during playback. This meant to keep their rendering consistent whenever the playback position has to be adjusted internally while from the audio rendering point of view "nothing" changed.

These offset calculations also depend on both sample rate and buffer size of the audio driver. If we are lucky the somewhat inconsistent occurrences of crashed is just due to different audio engine configurations. I'll have a look

[theGreatWhiteShark]

Alright. I was just able to reproduce the crash on my usual developing machine by setting my sample rate to 44100. (And I learned that we can change audio driver, sample rate, and buffer size while playback is rolling. Wow)

[theGreatWhiteShark]

@emilaxelsson could you check whether this version fixes your issues? (both the crash and the audible glitch should be gone)

[emilaxelsson]

It works!! Thanks a lot for the quick fix -- now I can finally switch to the new UI.

[theGreatWhiteShark]

Glad to hear!

[theGreatWhiteShark]

Hey @emilaxelsson. We found some rendering issue that caused audible artifacts when defining the length note in the aftermath of your issue. This version of Hydrogen should fix them

[emilaxelsson]

Sorry, for the delay. The linked version is no longer available, but I don't think it matters because everything sounds fine in the version I'm using (from Oct 6).