Auxiliary inputs stop updating

[cstawarz]

I've been experiencing a very strange issue where the acquisition board's analog and digital inputs sometimes stop updating in the GUI.

I'm running the latest code from the master branch (commit ab50889) under OS X 10.10.5. I have two TTL output lines from a DAQ connected to input channels 1 & 2 (digital or analog; I've tried both) via an OE I/O board. There are no headstages connected. (I've been testing clock sync with another application, so I've only need TTL inputs, not neural data.) The DAQ treats the TTL lines as a 2-bit word and increments the value every second, so the signal just cycles through 0,1,2,3 repeatedly.

Sometimes things work as I'd expect: When using the analog inputs, the LFP viewer shows the expected square waves; with digital inputs, it shows a sequence of four colors with a four-second period. However, sometimes the LFP window just stops updating. With digital inputs, I see the LEDs on the board changing as expected, so the TTLs are still being updated, but those updates aren't reaching the GUI. Frustratingly, the failure doesn't happen consistently: Sometimes the board will run fine for over an hour, while other times the problem shows up after only a minute or two.

In an attempt to figure out what's happening, I've tried running the GUI in the Xcode debugger. I've found that when the inputs stop updating, the calls to Rhd2000DataBlock::checkUsbHeader also start failing. Also, the raw TTL input values appear to be garbage, with bits assigned apparently random values. This suggests that either the data is getting corrupted at the Rhythm API level, or the board is just sending garbage.

I initially thought there might be some defect in the OE board or FPGA I was using. However, I've now reproduced the problem with a different board+FPGA pair, so this doesn't appear to be an isolated hardware failure. My only other thought is that maybe running the board without headstages is causing issues, but the sporadic nature of the failures makes that seem unlikely.

Does anyone have any idea what might be going on?

[aacuevas]

While I'm not completely sure, I think that the issue might be that the USB is sending data to the PC faster than it's acquiring it, eventually trying to transfer from an empty buffer and causing data misaligment. When the Rhythm firmware was first designed it was with the idea of always having at least one headstage plugged, so they aren't any checks for that. Since USB bursts are somewhat random (it mainly depends on the machine load) it might explain it.

Can you test if with a headstage connected it works or still fails?

At some point I'd like to backport some of the changes I did for USB3 support (in the works) to the old USB2 boards that take things like these into account, but that might break compatibility with the Intan board, so we need to be careful with that.

[cstawarz]

With a headstage connected, I was able to run for over four hours without problems, so your theory may be correct. It would also explain why this issue hasn't been reported before, as running without any headstages obviously isn't a typical usage scenario. Thanks!