Large SysEx transfers skipping bytes on some interfaces

[DisasterAreaDesigns]

Sending transfers larger than 255 bytes causes some interfaces to skip the 255th byte. Bytes after that one get sent just fine, but then the 511th byte is skipped, and so on.

I found some info here: https://forum.juce.com/t/problem-with-sysex-transfer-over-usb-midi-on-mac/9830

The same interface and transfer work fine using SendSX on Windows. Any ideas?

[krevis]

What interface are you using, exactly?

There are several MIDI interfaces that have bugs like this. The blacklist names many of them, but I'm sure there are more. The cheap single-port "USB MIDI cables" are generally no good.

The released version of SysEx Librarian is using the standard OS X MIDI libraries and drivers, so if it triggers a bug in a buggy device, there is unfortunately no workaround and no way to detect that the device didn't send all the data. (I tell CoreMIDI to send a whole buffer of data all at once, and it decides how to split it into chunks. There is no feedback path by which the device can even tell CoreMIDI, let alone my app, that it failed.)

However, I have a potential fix that hasn't been released yet:

For bug #43 I added a way for the app to do its own sysex timing and buffering. In the preferences, in the "Transmit Speed" tab, use the "Transmit buffer size" setting at the bottom, and change it from "Default" to something else (e.g. 128 bytes).

That isn't in an official build yet, but you can get that build here. It's from commit 7601de6.

This might fix your problem, or might not. Please try it and let me know what happens.

[DisasterAreaDesigns]

It's based on the Pluggable USB driver stack for Arduino. I've found another app that uses the JUCE framework for the backend and was able to change the sysex packet size from 256 to 255 which fixes things.

Thanks for sending over the build, but it still doesn't work correctly with my interface. That's not a knock on you at all, just an observation. Setting the transfer sizes to any other value just causes the missing bytes to be in different spots - essentially the byte after the packet is full gets missed due to the issues noted above in the forum thread.

Let's say that we have a large transfer, which could be anything larger than the packet size. 319 bytes, 4901 bytes, a million bytes - doesn't matter.

We have this message that goes something like "0xF0 0xF7." When we start transmitting, the MIDI driver is going to chop that up into messages for the USB, and will prepend a header byte onto each message, to let the receiving device know what to expect. This is not something that you have to worry about much on the Mac side, but on an embedded system it's crucial.

So our message is now broken up into these chunks -
{0x4, 0xF0, byte1, byte2}
{0x4, byte3, byte4, byte5}
{0x4, byte6, byte7, byte8}
... until the last message, when we need to know how many bytes we are sending.
If we have three bytes, the driver prepends 0x7. Two bytes is 0x6, one is 0x5.
So the shortest possible sysex message over USB is {0x5, 0xF7}.

If the packet size is exceeded, the driver will send the packet and then start building the next one. It's the driver's responsibility to package up the messages and send them out in a way that's compliant with the USB MIDI spec, so for our part we can just blast out the data without worrying and trust the driver to handle things. But if the packet size is smaller than the complete message things start to go wrong. The driver expects to see messages that are 1, 2, or 3 bytes long - there are plenty of examples of all of those. But if the packet ends and part of the message won't fit into it, what happens? It's pretty clear that the driver is either discarding that last byte or it's packing it up into another message in the next packet.

If it's being packed into the next packet, then it will be transmitted as a malformed message.
There's no way to send a single sysex byte UNLESS that byte is 0xF7, but the packet will always end with a single byte message since 256 % 3 = 1. The workaround is to make the packet size divisible by three so that the last message in the packet can be a full three bytes.

If the transfer happens to be 256 bytes exactly then of course the next message will be one byte (0xF7) but the driver will automatically use 0x5 for the header ("sysex ends with the following byte.") If the transfer is 255 bytes exactly then the last message can be {0x7, byte1, byte2, byte3} and the transfer succeeds. This problem ONLY happens when the transfer is bigger than the packet size and the packet size isn't divisible by three.

My humble suggestion would be to add some options to the packet size field that are divisible by three. That should fix issues with any oddball interfaces that are giving problems and won't break anything with interfaces that are currently working.

[krevis]

Try setting the preference value directly. The UI has a preset list of choices but you can set it to anything you like. In Terminal:

defaults write com.snoize.SysExLibrarian SSECustomSysexBufferSize -int 255

If that works then I can change the preset values in the UI.

[DisasterAreaDesigns]

Setting the defaults to 255 completely fixes the issue for me. Thanks for the tip.

[krevis]

I'm still trying to understand what's going wrong with these particular bad devices, though.

Does your device have a custom CoreMIDI driver, or is it a class-compliant device and you're using the built-in OS X CoreMIDI USB driver?

After reading the USB-MIDI spec, I understand how the driver might receive a buffer of (say) 256 bytes, and how after slicing those up into USB-MIDI event packets, it would be left with a single byte of sysex data. Call that byte aa.

It looks like there are two options the driver might take:

1. Send a USB-MIDI event packet for a single unparsed MIDI byte: pF aa 00 00.
2. Hang on to that byte, and remember that it's in the middle of sending sysex. Send it along with the next buffer. If it doesn't get a next buffer in a reasonable amount of time, just flush it.

Of these two options, only the first should be visible to your device.

Turns out that we can actually see this in the driver code. Apple used to provide a sample USB-MIDI driver, and I assume the real class-compliant driver is similar. You can see the code in this project. Specifically, around here, there's an #if CAN_USE_USB_UNPARSED_EVENTS which chooses one of the two options.

I suspect the real driver has that set to 1, so it's sending your device a USB-MIDI packet with a single byte unparsed event, and your device isn't handling that case properly (since it's a bit obscure).

But that's only a guess; I don't know how to inspect what's going on at the USB level.

(Apparently Wireshark can show USB on OS X now, so at least there's a possibility of doing that without spending hundreds of dollars on a hardware analyzer, but it's still not exactly my idea of a fun weekend.)

[DisasterAreaDesigns]

It's class-compliant. I think the issue is that the code in the user application (the firmware on the device) figures out what a message is by parsing the prepended header byte. If the driver is sending that unparsed event, there is no type byte added and so the device doesn't know how to deal with it.

I'm not sure that I would even want to have the device deal with an unparsed byte - I have a vested interest in making sure my sysex transfers are received correctly and so I want the device to reject other MIDI data when it's filling the sysex buffer. I can use the type bytes to make sure that only sysex (0x4, 0x5, 0x6, or 0x7) types get added to the data stream. If I receive another message type like clock or something, I can still deal with it but not add it in. My device can't know what kind of data an unparsed event is, so I would prefer it to ignore those.

Let's be fair - the folks that wrote the MIDI device code that runs on this platform weren't all that interested in sysex. It works but not all that well, and I'm using an older version of the library because recent changes have removed the ability to parse longer sysex strings entirely. Most of the time I'm only worried about ~30 bytes at a time for saving user settings. The long transfers are a way for me to update a big chunk of eeprom memory on the device and the end-user will never ever see or need that. I'm okay with the process being a little bit obscure. Going in and changing one value in a setup menu isn't a hardship to me. But I think if you were to make the default something divisible by three you might never have another reported issue about this.

[krevis]

My device can't know what kind of data an unparsed event is,

It's actually pretty simple to handle this case, without implementing a whole MIDI parser.

If you are in the middle of receiving a sysex message, and you get an unparsed event packet pF bb 00 00, look at the single byte bb inside.

< 80: must be more sysex data, so append it to your sysex message.

== F7: the sysex message is ending, and, for some reason, the driver chose to send it as unparsed. Seems unlikely, but it's easy to handle, just do the same thing as you would have with p5 F7 00 00.

>= F8: a single-byte System Realtime message (like clock) was sent as unparsed, for no apparent reason. Again, treat it the same way as p5 bb 00 00. (In your case, just ignore it.)

Anything else must be some other MIDI event, and in that case you can abort the sysex message right there.

But I agree, on my side, changing the default buffer size to work around this problem in existing devices might be a good idea. I'll keep this bug open until I finish it.

[krevis]

Just released version 1.4 which has the buffer size control.