[MS-DOS] Timing-sensitive issues sending SysEx via Music Quest PC-MIDI/Orpheus "Intelligent" MPUs

[dwhinham]

Hi @gmcn42, hope you are well!

@appiah4 has encountered problems using mt32-pi-control to send SysEx from MS-DOS via a Music Quest PC-MIDI "Intelligent" MPU clone card from keropi - there's a (long!) thread here: dwhinham/mt32-pi#127 (reply in thread)

I have an Orpheus card which has the same MIDI hardware/"intelligent" MPU as the PC-MIDI clone, and was able to reproduce this.

The problem is that the MPU emits no data at all when using mt32-pi.exe -g or mt32-pi.exe -m, for example, and so the Pi of course receives nothing. DUMP.EXE, on the other hand, works fine with an equivalent .SYX file; games and other intelligent-mode aware software also works fine.

The issue is timing-sensitive, because:

• For @appiah4, passing -v seems to "fix" it - but not for me. The printf() caused by -v appears to add just enough of a delay between putting the MPU into UART mode and sending the data to allow it to work.
• For me, spamming the command (up-arrow + return) very fast may sometimes send some data to the Pi. My PC is an Am5x86-133MHz "486", so a fairly fast DOS machine.
• For me, using -Y and providing a .SYX file works too; I think the delay caused by reading the file from disk allows it to work too.

As a hack, I added a call to delay_ms(1000); here, just after the mpu401_uart(mpubase) line:

mt32-pi-control/dos_src/midi_dev.c

Line 18 in bb63181

mpu401_uart(mpubase);

mpu401_waitread(mpubase); instead of the delay also appears to work, but I don't think this is correct.
I'm not sure of the true fix, as I'm not experienced with programming the MPU-401, but this knowledge may help narrow things down.

I'm guessing there's a sequencing problem with the checking and waiting of the MPU read/write status bits, but I can't see it.

[gmcn42]

Hi @dwhinham !

That's interesting... and somewhat confusing. :D Actually, mt32-pi.exe does a waitwrite (busy waiting on the MPU's STATUS register) every time before it writes anything to it. It is a bit weird that waitread would work here because a real MPU401 is not supposed to actively ACK a switch to UART mode AFAIK. After switching it will just transparently make any incoming bytes available, i.e. waitread would (should?) block until an actual MIDI message gets sent to the PC.

As a semi-technical reference I had a look at this: http://midi.teragonaudio.com/tech/mpu.htm

The whole MPU-401 interface code is basically 'stolen' from DOSMID (http://dosmid.sourceforge.net/). Can you check if you get similar problems playing a midi file with that program? If not, then I apparently added some really weird bug along the way.

Unfortunately, I am unable to reproduce the bug either in DOSBox or on a real K6-2 450MHz with a dumb ESS1868 and an intelligent HardMPU (based on an AVR). This doesn't mean anything of course as HardMPU is basically the DOSBox MPU-emulation put on a microcontroller. So it's not authoritative on real MPU behavior. Is the PC-MIDI supposed to be a 1:1 HW clone?

In any case, if adding a delay after switch to UART mode fixes it, there's nothing really wrong with having that. This happens only once every program run. So it wouldn't hurt throughput of longer SYX-files or anything else. But I guess I'd go lower than 1s, probably 100-200ms or something like that. I am reluctant to add a waitread because I am pretty sure it will lock up on many cards which send no reply. Would be better to find the root cause though ofc.

[dwhinham]

...i.e. waitread would (should?) block until an actual MIDI message gets sent to the PC.

Yes, this makes no sense to me either :)

The whole MPU-401 interface code is basically 'stolen' from DOSMID (http://dosmid.sourceforge.net/). Can you check if you get similar problems playing a midi file with that program? If not, then I apparently added some really weird bug along the way.

DOSMID is working just fine over here, although looking at its sources, it does the MPU reset/UART switch, then loads the MIDI file, and then begins to output the bytes. The file I/O it does could be having the same effect as a delay (like how using -Y to load a sysex file also works fine with no delay on mt32-pi-control).

Unfortunately, I am unable to reproduce the bug either in DOSBox or on a real K6-2 450MHz with a dumb ESS1868 and an intelligent HardMPU (based on an AVR). This doesn't mean anything of course as HardMPU is basically the DOSBox MPU-emulation put on a microcontroller. So it's not authoritative on real MPU behavior. Is the PC-MIDI supposed to be a 1:1 HW clone?

I don't think it's a 1:1 clone; it has a Z80 CPU on board with some firmware to do the "intelligent" stuff, so differences in the implementation compared to a real Roland MPU could be at play here. In other words, another "non-authoritative" MPU implementation, though a pretty good/compatible one nonetheless according to considerable testing on VOGONS.

In any case, if adding a delay after switch to UART mode fixes it, there's nothing really wrong with having that. This happens only once every program run. So it wouldn't hurt throughput of longer SYX-files or anything else. But I guess I'd go lower than 1s, probably 100-200ms or something like that. I am reluctant to add a waitread because I am pretty sure it will lock up on many cards which send no reply. Would be better to find the root cause though ofc.

I agree, and I don't think the waitread is a good idea at all, was more of a blind "let's see what happens" :)
I lowered my delay to 100ms, and it's still working fine, so I think worst-case we could just add that and call it good.

Keropi (one of the creators of the Orpheus and PC-MIDI clone cards) has offered to do some testing as he has a big collection of MPUs. He has an mt32-pi, but has not tried your tool yet, so we could wait and see if he can reproduce it too if you like.

Cheers!

[gmcn42]

So, after a bit of searching I had a look at the MPU-401 Technical Reference.
My takeaways:

• Trivia: The original MPU-401 uses a HD6801 microcontroller. It's basically the MCU variant of a 6800 processor.
• There is no mandatory waiting period after sending the command to switch to UART mode, at least not on systems from 1985 (which might also be the problem) 🙂
• Documentation on the UART mode itself is scarce because Roland only ever intended it for debugging purposes. They didn't think everyone would use it in production.
• The DRR Bit on the STATUS register should nevertheless always indicate if the MPU is busy with something, even in UART mode. DRR=1 means "NOT ready to receive", DRR=0 means all is well. (Dumb UART cards with a FIFO buffer AFAIK also use this to indicate the buffer is getting full.)

The DRR=0/1 polling is what mpu401_waitwrite does which is called before every DATA port write. From how I understand the technical reference, if the PC-MIDI needs some time for the switch to UART mode to complete, it should set DRR to 1 until it has safely completed and is ready to accept the first UART Byte. If something gets lost here, that seems to indicate DRR is 0 before a DATA port write is safe in this specific condition. Or I am doing something very wrong but I haven't found anything yet. Especially considering sending an arbitrary number of Bytes works fine otherwise.

This might not crop up with other programs/games due to the timing stuff you outline, like loading a file or something else and it would cause issues only for the first few sent notes/commands anyway.

It may well be that the original MPU also has this timing issue and they didnt bother to write about it as PCs in 1985 simply weren't fast enough to cause the problem.

Nevertheless it's all a bit academical. If there are cards in the wild that need a hard 100ms delay after the UART switch, then that's the fix and I'll do a small bugfix release for the MS-DOS version probably later today. 🙂

If you have contact with him, I'd be happy to hear keropi's thoughts on this and how he implemented the intelligent/UART switch in firmware, especially the handling/timing of the DRR Bit in this case.

[appiah4]

Thanks to both of you for looking into this, really appreciate it. I'd be more than happy to assist with testing debug/fix builds.

[dwhinham]

So, after a bit of searching I had a look at the MPU-401 Technical Reference.
My takeaways:

Thanks for this! You've basically just confirmed my thoughts from briefly reading some of the MPU401 technical reference yesterday; the code does seem to be doing the right thing, or if it's not, I can't see it either.

It may well be that the original MPU also has this timing issue and they didnt bother to write about it as PCs in 1985 simply weren't fast enough to cause the problem.

This appears to be true - we now have some evidence to support this from keropi:

just tested control program
on 386DX40 so no speed issues in the way
I can replicate the no-data send with pcmidi and a scc-1
but the hacked version you posted works perfect with both
maybe it is just a matter of a small delay needed to be inserted
dump.exe for example works fine for both (and has been working for ages)

So on a 386, with either a genuine Roland SCC-1 (which has an intelligent MPU) or the PC-MIDI clone, he gets the problem, and a delay fixes it on both. He's tested with both a 1 second delay version and a 100ms delay, both apparently working fine, so I think 100ms should be a safe bet!

@appiah4 If you want to confirm that a 100ms delay works for you too, here's a binary:
mt32-pi-ctl-hack.zip

Nevertheless it's all a bit academical. If there are cards in the wild that need a hard 100ms delay after the UART switch, then that's the fix and I'll do a small bugfix release for the MS-DOS version probably later today. slightly_smiling_face

Great!

If you have contact with him, I'd be happy to hear keropi's thoughts on this and how he implemented the intelligent/UART switch in firmware, especially the handling/timing of the DRR Bit in this case.

My understanding is that keropi has re-used the original Music Quest firmware blob cloned from the official cards' EPROMS released in the 80s, so we won't be able to understand this unless someone fancies doing some Z80 disassembly. 🙂

[appiah4]

@dwhinham I will check the build and report later tonight!

[gmcn42]

Alrighty, added a 100ms delay after mpu401_uart(), probably the same as @dwhinham did. As expected, it doesn't break anything on my end. If one of you can confirm this fixes the issue, I'll release it.
mt32-pi-ctl_1.0.0a_dos.zip

[dwhinham]

Tested working, LGTM!