USB Packet loss

[yoctopuce]

Hi,

We have found a serious issue on the raspberry PI USB stack (or at last the HID USB stack). We produce USB device and have found that we often get missing packet.

We have create an specific firmware illustrate the problem. This debug protocol is very simple the Raspberry will send one packet of 64 bytes, then the device will start a burst of USB transaction. Our USB device work in full speed (USB 1.1) and follow the HID protocol. All packets have the same size (64 bytes). the first uint32_t will contain the current packet number and the second uint32_t will contain the number of packet to send. Each packet is numbered and we stop after burst of 4096 packet of on the first missing packet.

The debug program call libusb-1.0, and will look for one of our device with the debug firmware, then send one USB packet to the device to start the burst. Then we will wait for all packet and check that all packet are send. You can download the source here http://www.yoctopuce.com/tmp/debug_libusb.zip

We have test this code and firmware on many linux platform (i386 and x86_64), and on a ARM based NAS (QNAP TS-219P II). We never get any missing packet. but on the Raspberry PI we always get missing packets. We have check with a USB analyser that the missing packet is correctly sent on the wire (the packet has been correctly acked by the Raspberry PI USB host). It seems that the packet is lost by the Raspberry PI firmware.

We have run this test on both the debian and arm-linux image, we have recompile the last version libusb without success. Our customers also have reported this issue.

We are ready to send one of our device to help to reproduce this issue, or to give a remote access to our Raspberry PI (or do any thing that can help you to fix this issue).

[popcornmix]

Can you get a usb trace from kernel:
http://www.kernel.org/doc/Documentation/usb/usbmon.txt

You may need to rebuild kernel with CONFIG_USB_MON, although the kernel_debug.img, and debug/modules may be built with the right options.

[yoctopuce]

I have recompile the kernel with usbmon activated and
run a test (a burst of 4096 packets) and I get two missing packets. I have look at the usbmon trace
and the two missing packet are... missing. So it seems to be into the linux "driver" or into the
hardware. I have also add the dump of what our USB analyser has captured.
You can download this trace here:

http://www.yoctopuce.com/tmp/usb_trace.zip

One thing that I have not mentioned, is that I work on my raspberry using ssh so I have some
network traffic during the run.

[voneiden]

So this could be related to the sticky key problems with certain USB keyboards too?

[ledow]

I believe this may also be showing itself in the dying of 3G PPP sessions that I'm experiencing with a Huawei E160. When the data rate is low (e.g. PPP over GPRS, AT Commands, etc.), there's no problem. But with a 3G PPP session, it can connect and work for ages until you put a lot of traffic through it when PPP just thinks "the modem has hung up". No other USB devices are present, power is more-than adequate, and the stick works fine in other Linux machines with identical configurations.

[maximeh]

Do you think this issue can have something to do with the crackling that can be experienced while using a USB soundcard ?

[mickael9]

I also noticed strange problems that could be related :

• Under Xorg, the mouse pointer will sometimes freeze for a split second
• Mouse clicks are often missed
• My USB camera will emit a bunch of IO errors : [ 1597.922185] gspca: ISOC data error: [1] len=0, status=-4004 leading to Corrupt JPEG data: premature end of data segment
• Probably unrelated, but the default kernel driver for my Wifi adapter will freeze (although it just works on my linux desktop). Using the Realtek driver works (Realtek RTL8188CUS chipset)

[popcornmix]

Can you try latest firmware (e.g. from rpi-update or bleeding-edge firmware repo) and add:
sdhci-bcm2708.missing_status=0 sdhci-bcm2708.sync_after_dma=0

to cmdline.txt. We think there may be USB packet loss caused by sdcard driver disabling interrupts for too long.

[mickael9]

Well, it turns out this didn't change anything.

• I found the mouse problem seems unrelated, because it happens only with my wireless mouse (and only if my wifi dongle is connected). My kernel log gets spammed with those messages :

Aug  2 01:58:46 alarmpi kernel: [  214.463697] usb 1-1.3: reset full speed USB device number 5 using dwc_otg
Aug  2 01:58:46 alarmpi kernel: [  214.703697] usb 1-1.3: reset full speed USB device number 5 using dwc_otg
Aug  2 01:58:46 alarmpi kernel: [  214.893700] usb 1-1.3: reset full speed USB device number 5 using dwc_otg
Aug  2 01:58:48 alarmpi kernel: [  216.413711] usb 1-1.3: reset full speed USB device number 5 using dwc_otg
Aug  2 01:58:48 alarmpi kernel: [  216.603694] usb 1-1.3: reset full speed USB device number 5 using dwc_otg

It's therefore not surprising that my mouse freezes if it gets reset all the time! (no idea why it does that though)

• The camera errors still happen.

[root@alarmpi ~]# /opt/vc/bin/vcgencmd version
Aug  1 2012 19:51:34 
Copyright (c) 2012 Broadcom
version 328870 (release)

[root@alarmpi ~]# cat /proc/cmdline 
dma.dmachans=0x3c bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708.boardrev=0x2 bcm2708.serial=0x3ec56e9c smsc95xx.macaddr=B8:27:EB:C5:6E:9C smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 loglevel=2 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait sdhci-bcm2708.missing_status=0 sdhci-bcm2708.sync_after_dma=0

By the way, I'm using ArchLinux ARM (with associated kernel), it's probably a week old.
Don't know if this matters.

[mickael9]

I'm also experiencing a very strange problem with the latest elf files.
Using anything else than arm240_start.elf, the Pi will freeze soon after loading the kernel, but only when doing a cold boot: if I copy /boot/arm128_start.elf to /boot/start.elf and reboot, it will load, but as soon as I power it down, it won't boot again. The freeze happens just after the Raspberry Pi logo shows up (the cursor blinks for a second and then stops).

Just to be sure, here are the md5 sums of the files :

923c58d7a0d22aad45e01c7970881671  /boot/arm128_start.elf
55f2bc56382b66cad5e78ce2e8690abe  /boot/arm192_start.elf
78eacdd91dee56f1dcf24b1fd9122ada  /boot/arm224_start.elf
7521744c5fc6ac409767732f4beee5e7  /boot/arm240_start.elf

[popcornmix]

does
boot_delay=1
(or 2) in config.txt help?

[mickael9]

Nope, it doesn't help. Even with larger delays (10sec).

[yoctopuce]

Hi, We are quite busy these days (next week will be the same).I will try to redo some testing as soon as I have some time. But if someone want a device to reproduce the the issue you can send a mail to support@yoctopuce.com.

[LX3KR]

Thank you popcornmix!

I've spent days trying to figure out why my SignaLink USB sound card kept dying due to EMI rUSB resets, having installed Raspbian yet again and updated the to the latest firmware with rpi-update the problem was still present, but when I included the lines below to cmdline.txt as advised by you and rebooted no more resets! my USB card is now super stable, thanks for your help.

Can you try latest firmware (e.g. from rpi-update or bleeding-edge firmware repo) and add:
sdhci-bcm2708.missing_status=0 sdhci-bcm2708.sync_after_dma=0

to cmdline.txt. We think there may be USB packet loss caused by sdcard driver disabling interrupts for too long.

Cheers,

Karl - LX3KR

[jvinolas]

I was getting mad with a 3g modem till I realized the raspi doesn't have a wide enough power line (5V) to the USB plug. Even the micro usb is not good enough to supply 5V as stable as it should be. If that 5V doesn't keep stable, the raspi does extrange things.

If you measure the voltage on TP1/TP2, it is higher than the real voltage on USB plug. My 3g modem seemed to work well during configuration but hanged just at the end or on downloading (high power consumption). I measured the voltage on usb case on raspi (on the back of the board) and it went down to 3.6V!!!! So the problem is that the raspi doesn't have a wider enough line between the power supply and the usb plug.

My solution was to plug my power supply to 5V pin on P1 directly (do not use micro-usb because it have the same problem: too slim wires!) and to solder a cable from that pin to the 5V pin on USB. Now I got connected without any problems.

I don't know if it will be useful, but mine now is not hanging anytime! This is my soldering solution: http://static.inky.ws/image/2573/image.jpg

[ninlilizi]

I previously had the issue of the option module freezing up my pi due to dropping usb packets all over the place when attempting to use by 3g dongles.
I previously worked around it using the kludge I detailed on my blog
http://blog.phoenixhaven.net/2012/07/13/rpi-raspbian-usb-dongle-segfault-workaround/

Just tested it after adding 'sdhci-bcm2708.missing_status=0 sdhci-bcm2708.sync_after_dma=0' to my cmdline.txt as advised. And the problem is resolved. I can now stably connect my 2 usb dongles without the previous workaround (ie, via the correct option module rather than forcing usbserial directly).

Tested using latest rpi-update + raspian + wvdial for creating the connections,
Tested for stability my pulling 1080p HD youtube content over both dongles simultaneously. (The pi is serving as a home gateway appliance)

[r10r]

Hi, I think my issue raspberrypi/linux#82 is related with this one.
I also experience URB package loss from a DVB-C USB device.

[ghollingworth]

What bandwidth are you expecting? Is this a low bandwidth figure?

There are no errors because it's isochronous transfers and not
guaranteed transmission.

What else do you have plugged in? Can you do a lsusb -v and post the output?

Gordon

On 04/11/2012, Robert Massa notifications@github.com wrote:

I think I'm also having a problem related to this issue. When connecting a
USB webcam the stream is corrupt. This webcam outputs MJPEG so I've dumped
the raw frames and checked them using jpeginfo. This tells me most (90%)
of the frames end prematurely. Similar to what @mickael9 is saying, although
I'm not getting any errors in dmesg.

Setting the cmdline options doesn't change things. I've tried this with an
rpi-updated system.

Anything else I could try to debug this?

---

Reply to this email directly or view it on GitHub:
#19 (comment)

Sent from my mobile device

[Grepsy]

I think I'm also having a problem related to this issue. When connecting a USB webcam the stream is corrupt. This webcam outputs MJPEG so I've dumped the raw frames and checked them using jpeginfo. This tells me most (90%) of the frames end prematurely. Similar to what @mickael9 is saying, although I'm not getting any errors in dmesg.

Setting the cmdline options doesn't change things. I've tried this with an rpi-updated system.

Anything else I could try to debug this?

[Grepsy]

lsusb output: https://gist.github.com/4012340

I'm not exactly sure about the bandwidth, but a crude approximation would be 100kb (640x480 jpeg) * 30 (fps) = 2.9 MB/s, probably less.

I have a keyboard plugged in, nothing else. I've tried running without the keyboard, that doesn't change things. I also tried to reduce the image size to 320x240, still same result.

[Grepsy]

@ghollingworth Would you have any other ideas about how I could debug this?

[ghollingworth]

Sorry, in general to debug this we're going to need to use the USB analyser and some deep debug information !

Interesting thing here is it's a USB2.0 device and therefore I would not expect to see problems in receiving data... This would be an interesting device to try to debug, we've seen issues with dropping data recently on USB 2.0 devices (like the ethernet controller) and need to debug...

Will spend some time debugging this in the new year when I'll actually have time to do it...

[ethelthefrog]

ghollingworth, I have one of the problematic wireless keyboards. I can bring it in for you to plug into the analyser if you want. (I'm the tall FPGA bloke, in case you were wondering).

[ghollingworth]

Sounds like a plan although I'll need to pop in and pick it up sometime...

Other problem is I don't have the analyser anymore! We are looking to buy
a new one soon though so I can actually get onto it!

Gordon

On Sunday, 6 January 2013, ethelthefrog wrote:

ghollingworth, I have one of the problematic wireless keyboards. I can
bring it in for you to plug into the analyser if you want. (I'm the tall
FPGA bloke, in case you were wondering).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/19#issuecomment-11934946.

[ethelthefrog]

D'oh. I'd forgotten about that particular change in situation (I was in Devon that particular Friday, sorry). You are still welcome to borrow it. Maplin have them half price at the moment (Microsoft 800 for twenty quid), so that may be an option too.

I'm afraid I'm a bit too busy with the day job to grab some traces from our analyser at the moment...

[ghollingworth]

One thing you could do is help test a hypothesis …

Plug in your keyboard, power it up and test the keys ten times

1. how many times did it produce dodgy keys? (Is it every time or does it seem to be sometimes better than at other times)
2. If it starts up very poor, trying unplugging it and plugging back in again, does this effect the likely error rate?

Thanks

Gordon

On 6 Jan 2013, at 22:27, ethelthefrog notifications@github.com wrote:

D'oh. I'd forgotten about that particular change in situation (I was in Devon that particular Friday, sorry). You are still welcome to borrow it. Maplin have them half price at the moment (Microsoft 800 for twenty quid), so that may be an option too.

I'm afraid I'm a bit too busy with the day job to grab some traces from our analyser at the moment...

—
Reply to this email directly or view it on GitHub.

[ethelthefrog]

First time plugged in. 10 'P' presses. 6 actual P's; the last one got stuck and made p all over the place.

Second time plugged in. 10 'P' pressed. 6 actual P's with no stuckness.

Third time plugged in: 10 'P' pressed. 3 of the first 5, then stuck, then stuck again.

I've not noticed any times being more or less reliable. It loses at least 20% of key-down/key-up events. Obviously, I don't know exactly how this translates to USB traffic.

EtF

[ethelthefrog]

I should probably also mention that dmesg is full of "usb 1-1.2: reset full-speed USB device number 7 using dwc_otg" messages. lsusb confirms that device 7 is the wireless keyboard transceiver thingy.

[ghollingworth]

OK,

Can you bring it with you to work and I'll pick it up later in the day?

Thanks

Gordon

On 7 Jan 2013, at 22:41, ethelthefrog notifications@github.com wrote:

First time plugged in. 10 'P' presses. 6 actual P's; the last one got stuck and made p all over the place.

Second time plugged in. 10 'P' pressed. 6 actual P's with no stuckness.

Third time plugged in: 10 'P' pressed. 3 of the first 5, then stuck, then stuck again.

I've not noticed any times being more or less reliable. It loses at least 20% of key-down/key-up events. Obviously, I don't know exactly how this translates to USB traffic.

EtF

—
Reply to this email directly or view it on GitHub.

[ethelthefrog]

It's on my desk. Just get Victoria to let me know when you're here.

EtF

[ethelthefrog]

Any movement?

[augustkst]

With my RPI (rev. 000e) this manifests itself as pops and clicks when I play music with MPD through a USB DAC. Therefore I currently cannot use the RPI as a music server as planned.

[dos1]

👍 +1 from me.
This issue is pretty annoying, as neither majority of webcams nor audio DACs can be used with Raspberry Pi Rev. B at the moment. It's limiting and disappointing, I hope you get this sorted soon.

[arcturus80]

I was trying to build a high end audio player with a raspi model B and a fancy USB soundcard but ran into problems with stuttering sound. I'm sure it's because of this... Does anyone know if they solved this somehow in the latest raspbian release? Does the model A (without the internal usb hub) have the same problem with packed loss over USB?

[popcornmix]

@arcturus80
There's certainly been improvements in latest raspbian release.
You can get these with
sudo apt-get update && sudo apt-get upgrade

If that doesn't fix it you should try the fiq_split branch:
http://www.raspberrypi.org/phpBB3/viewtopic.php?p=345905#p345905

[arcturus80]

Thanks! I'll try that.

[arcturus80]

Unfortunately this did nothing to my problem. Audio is still skipping and crackling :( Problem increases with high samplerate 24bit flac files but are also present when playing ordinary mp3's. Maybe my problem isn't packet loss after all, but something else...

[P33M]

@arcturus80

It is likely that your USB audio device(s) are suffering from the same problem as seen elsewhere.

Please post full dmesg output && sudo lsusb -v -d [deviceid of your audio device].

fiq_split_enable is now merged into the default branch on rpi-update. Can you compare results with the /boot/cmdline.txt parameter dwc_otg.fiq_split_enable=1 / 0 ?

For certain types of device you can replug them and get "better" sounding audio. This is due to the clash or otherwise of where the microframe scheduler puts the split transactions.

[P33M]

This issue is no longer present in the fiq_fsm rewrite.