Issue receiving large quantity of MIDI messages

[mariobuoninfante]

When Pd receives a large quantity of SysEx messages, the MIDI IN port 'gets stuck' and then you need to go to 'Media->MIDI settings...' and re-select the port, otherwise no MIDI messages will be received anymore.
Here's a video that shows the issue (pass: sysexissue):
https://vimeo.com/316624285
attached the Pd and Max patches used to repro.
This has been tested on a MacBookPro 2.7 GHz Intel Core i5 16 GB of ram using Pd 0.49-1
Similar results found on Windows 10 and Ubuntu Studio 16.04.

Pd_SysEx_issue_repro.zip

[mariobuoninfante]

this happens also with different MIDI messages, not only with SysEx

[mariobuoninfante]

Pd 0.47 doesn't show the problem

[umlaeute]

could you also try Pd-0.48?

[umlaeute]

hmm, i cannot really reproduce this with a plain Pd-setup (see attached patches), using Pd-0.49-0 as shipped with Debian/buster (amd64), using the alsamidi backend.

at 2ms, i haven encountered any problems.

going down to 1ms, packages start getting dropped, but they keep getting in.

sysex.zip

[Lucarda]

@umlaeute

Using a "Virtual loopback MIDI cable" (https://www.tobias-erichsen.de/software/loopmidi.html) i can send using your "sender.pd" to http://www.midiox.com/ @ 1ms with no problem.

When using your "receiver.pd" I cant go faster than 8ms. Then it hangs and I have to reset the midi-in driver as in @mariobuoninfante video.

Win8.1
Pd 0.49.0 64bit.

[Spacechild1]

FWIW, I have recently encountered situations on Windows 7 where MIDI input would sometimes stop working. I wasn't sure about the cause of the problem and had no time to debug it. I think this happened after I switched from 0.48 to 0.49. I used a keyboard which sends a lot of realtime clock messages, so maybe it's the same issue which Mario is experiencing (but might as well be a problem with the MIDI interface).

[mariobuoninfante]

double checked Pd 0.48.0 and that works fine. the issue is not there.
Tested on MacBookPro 2.7 GHz Intel Core i5 16 GB MacOS Sierra

[umlaeute]

iirc @danomatika fixed a few things in the portmidi implementation. that's the only occasion where i think the problem could have crept into the code (and it would explain why the linux builds are not affected)

[mariobuoninfante]

@umlaeute On Linux also there are 'bandwidth' issues. If you try to deal with the same amount of data, something would get lost for sure. Do you think that's due to ALSA?

[umlaeute]

well, as i said: i couldn't reproduce this on my linux machine (using a "virtual" midi connection between two Pd instances on the same machine).

[mariobuoninfante]

Yap, but you said messages got dropped. And I experienced the same in other occasions (receiving FW via SysEx from HW devices)

[umlaeute]

but this issue is about MIDI-IN "getting stuck" (?)

[mariobuoninfante]

fair enough ;)
I'll open another one after some further testing

[mariobuoninfante]

also if, reading the title, here we're talking about dealing with large quantity of messages. Would it be ok to put here info about Linux performances?

[umlaeute]

sure.
i've been running my patch for >5minutes now, sending 21-byte SysEx messages (including the start and stop byte) at a rate of 2ms, without a single dropout.

after that i raised the message length to 101 bytes, and let it run for another 5 minutes. still no dropout.

[mariobuoninfante]

Hi,
I just updated the title of this bug, as it's been proven that the issue is not only with SysEx but with all the MIDI msg.
is there any update on this? I'm simply curious to know what's the issue :)

[danomatika]

I've not had any time to look at it. It would be nice if there was more than one developer interested in MIDI...

[mariobuoninfante]

I totally understand.
Unfortunately I'm not a C/C++ programmer, but I can try to have a look at it and see if I can spot anything. The area where to look seems to be clear anyway, since we've been able to track down when the issue has been introduced (0.48.1)

[Spacechild1]

I can reproduce this problem on Windows by "connecting" two Pd instances with a MIDI loopback (https://www.tobias-erichsen.de/software/loopmidi.html) and sending a simple MIDI message from one instance to the other at an interval of 1 ms. after about 2 seconds, the [midiin] object stops outputting anything (while the loop back program itself still shows traffic) and I have to reset the MIDI device in the MIDI settings. since @umlaeute can't reproduce this with alsamidi, it seems to be a problem with portmidi. note that this happens both with Pd 0.48 and Pd 0.49. I can have a look but not in the next few days I'm afraid...

[Spacechild1]

ok, I think I've fixed this in the PR above, the error was only in the Pd code.

@mariobuoninfante here are binaries containing the bugfix:
pd-0.49.0-midi-fix.zip

can you test and tell me if this fixes your problems?

[porres]

@Spacechild1 , you're on fire :)

[mariobuoninfante]

Hi @Spacechild1,

really sorry I've seen this just now, I'm getting the build and test it. Max tomorrow I'll get back to you.
sorry for the late reply

[mariobuoninfante]

@Spacechild1 ,
is the build you attached here, the same I can find here: #581?
cause I tried that one (so that I didn't have to compile anything) and still gets stuck when I use the Max Patch attached above

[Spacechild1]

no, that's a totally different issue. use the one above ("pd-0.49.0-midi-fix.zip")

[mariobuoninfante]

no, that's a totally different issue. use the one above ("pd-0.49.0-midi-fix.zip")

@Spacechild1
I cannot find neither the MacOS App nor the autogen to build it. Am I missing something?

[Spacechild1]

oh, it's a Windows build. if you want to build from source for Mac, clone my branch: https://github.com/Spacechild1/pure-data/tree/midi-fix

[mariobuoninfante]

@Spacechild1 it's rock-solid!!!!!
21 bytes per msec, and it doesn't seem to drop a single byte, plus it never gets stuck :)

[mariobuoninfante]

Hey guys,

I'm just curious why this has been reverted 7cbc5e4
is there any particular reason?

[danomatika]

is there any particular reason?

This change is in fact in the sources now. You are seeing my first quick fix which ended up being more involved.

I reverted as the response from the port midi developers was "wait, why would you NOT want rate limiting?" After some discussion and testing, we all decided Pd needed to override this setting and they will incorporate the define override upstream.

[Spacechild1]

Also, the commit in question is actually about sending MIDI. The PR about receiving MIDI is still open: #598

[mariobuoninfante]

ok, it makes sense of course
thanks @danomatika @Spacechild1 :)