How do SysEx events actually work?

[MaurizioB]

I'm trying to understand how to write actual SysEx events, but, so far, I wasn't able to.
I tried appending midi.SysexEvent(data=[list of values]) but it doesn't seem to create a valid MIDI file (at least, not for aplaymidi).
I also tried adding/stripping the leading 240 and trailing 247 SysEx values in different combinations, still with no result.
I was thinking about using struct, but, according to the code, that's not the case.

[MaurizioB]

Ok, after analyzing the output of a generated mid file with sysex data in it, I finally found out how this works.
The data is composed by the list of sysex data without the leading 240 and trailing 247 values, and preceded by the length of bytes plus the terminal 247 value.

So, for example, if the complete original message is this:
0xf0, 0x00, 0x20, 0x29, 0xf7
the actual content is this:
0x00, 0x20, 0x29
the data for the event will be:
[0x04, 0x00, 0x20, 0x29]
or, in integer values:
[4, 0, 32, 41]

I hope this helps anybody struggling with the same problem.

UPDATE:
I realized that this doesn't obviously work if the sysex length is greater than 127 bytes (always including the terminal value).
This can be achieved by converting the number to 7 bits (most significant), where the least significant has byte 7 set to 0, while the preceding have it set to 1. I suppose that this is how any SysEx value greater than 127 is usually represented. Since the title of this issue is about SysEx in general, I thought it could be nice to get some insight on how all this works.

For example, with a data length of 390 bytes, the length byte should be 391, counting the terminal 0xF7, and is represented as 131, 7.

The first value is obtained by shifting the length by seven bits to the right, then adding the leading 1 (bitwise OR for 128):

391 >> 7 | 128 = 131

391 is "110000111", shifting its bits to the right 7 times makes it "11" (3); since 3 is less than 127 we don't need to make further conversions; the "|" operator OR adds "10000000" (128), which makes it "10000011" (131). If the result of the shifting were greater than 127 (for example, for an original value greater than 16383) we'd need to take the result of the shifting and do the same operation again.
For the last value, we mask the original value with 127 to get only its last 7 bytes:

391 & 127 = 7

The "&" operator returns only the bits that are equal to "1" in both values, since 391 is "110000111" and 127 is "11111111" (or "001111111"), only the last three bytes are considered ("111" is 7):

110000111
      ||| & # return only the common "1"'s
  1111111
---------
000000111

This simple function converts any 32bit positive integer to 7 bit sysex values, even when greater than 16383:

def get7bit(v):
    values = [v & 127]
    v >>= 7
    while v:
        values.insert(0, v & 127 | 128)
        v >>= 7
    return values

The reverse (obtaining the integer values from a full sysex data list) can be something like this:

def getInt(values):
    intList = []
    shift = 0                 
    for v in reversed(values):
        if v & 128:                                     
            shift += 1                                  
            intList[-1] = intList[-1] | ((v & 127) << (7 * shift))
        else:                
            shift = 0        
            intList.append(v)
    return list(reversed(intList))

I hope this helps...

[vishnubob]

so, in other words, you just had to write your data to an array and everything else just worked? 👍

[MaurizioB]

Well, that seemed to work, until I run into a strange bug yesterday.
If I create a track with a normal event (let's say a note on event), a sysex and another "normal" event, I get a file that seems to be corrupted.
This is an example:

midi.Track(\
  [midi.NoteOnEvent(tick=0, channel=0, data=[22, 64]),
   midi.SysexEvent(tick=0, channel=0, data=[8, 11, 12, 13, 14, 15, 16, 17]),
   midi.NoteOnEvent(tick=0, channel=0, data=[22, 64]),
   midi.EndOfTrackEvent(tick=0, data=[])])

For this track, kMidimon says: "Unexpected byte (16) at 38 at offset 39Unexpected byte ( 0) at 40 at offset 41Unexpected byte ( 0) at 43 at offset 44".
The strange thing is that the same track without the first NoteOn event outputs a valid file.
Can anyone confirm this?