mmcblk0 error -110 errors (regression?) #280

Closed
oerik opened this Issue Apr 22, 2013 · 149 comments

Projects

None yet
@oerik
oerik commented Apr 22, 2013

On 3.8.8+ kernel with an class 10 4Gb card (unknown brand) gives me mmcblk0 errors, the 3.6.11 version didn't give me these errors (was running 2 weeks non-stop with 3.6.11 kernel). A second rpi with an Transcend SDTRHC8GB is working just fine. I'm going the replace the SD card anyway.
Note, the 4Gb card doesn't support fstrim, the 8Gb does.

[13288.492420] mmc0: final write to SD card still running
[13298.479242] mmc0: Timeout waiting for hardware interrupt - cmd12.
[13298.480375] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[13298.930974] mmc0: final write to SD card still running
[13308.919721] mmc0: Timeout waiting for hardware interrupt - cmd12.
[13308.920845] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[15090.374963] mmc0: final write to SD card still running
[15100.403772] mmc0: Timeout waiting for hardware interrupt - cmd12.
[15100.404897] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[15100.856441] mmc0: final write to SD card still running
[15110.884252] mmc0: Timeout waiting for hardware interrupt - cmd12.
[15110.885382] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900

@popcornmix
Contributor

I've not seen this so far. Can you revert back to 3.6.11 firmware on same sdcard and confirm the messages are gone?

Anyone else seeing sdcard errors on 3.8.8 that were not present on 3.6.11?

@oerik
oerik commented Apr 22, 2013

Ok, I've reverted back to 3.6.11 and keep it running for a night. There is at least every hour some write access to the disk. I'll report tomorrow . The 3.8.8 kernel was reporting errors within a few hours.

@oerik
oerik commented Apr 23, 2013

3.6.11 was running all night without any mmc errors logged. Activated 3.8.8 and during boot time:
[ 26.797844] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 26.807896] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900

These are the contents of /sys/block/mmcblk0/device :

root@raspberrypi:/sys/block/mmcblk0/device# cat cid && cat csd && cat date && cat fwrev && cat hwrev && cat manfid && cat oemid && cat scr && cat serial && cat erase_size && cat preferred_erase_size
4134325344344742300065558e00ca00
400e00325b5900001da77f800a400000
10/2012
0x0
0x3
0x000041
0x3432
0235800001000000
0x0065558e
512
4194304

I'll have to admit it is the cheapest micro SD class 10 card I could find. but so far it was working reliable.

@popcornmix
Contributor

I'll have a look at the changes and see if there's anything I can try reverting.

@hvenzke
hvenzke commented Apr 23, 2013

Have the same issue with Transcend SD HC class 10 16GB card
Not happend on 3.8.7 .

Initializing Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.8.8-4-raspberrypi (root@lb1) (gcc version 4.6.3 (Debian 4.6.3-14+rpi1) ) #2 PREEMPT Tue Apr 23 16:29:45 CEST 2013
[    0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[    0.000000] Machine: BCM2708
[    0.000000] cma: CMA: reserved 16 MiB at 1d000000
[    0.000000] Memory policy: ECC disabled, Data cache writeback
[    0.000000] On node 0 totalpages: 122880
[    0.000000] free_area_init_node: node 0, pgdat 406765f0, node_mem_map 40725000
[    0.000000]   Normal zone: 960 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 121920 pages, LIFO batch:31
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 121920
[    0.000000] Kernel command line: dma.dmachans=0x7f35 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708.boardrev=0xf bcm2708.serial=0xf30d7eb6 smsc95xx.macaddr=B8:27:EB:0D:7E:B6 sdhci-bcm2708.emmc_clock_freq=100000000 vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000  dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 rootflags=commit=120,data=writeback elevator=deadline rootwait
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] __ex_table already sorted, skipping sort
[    0.000000] allocated 983040 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.000000] Memory: 480MB = 480MB total
[    0.000000] Memory: 462536k/462536k available, 28984k reserved, 0K highmem
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
[    0.000000]     vmalloc : 0x5e800000 - 0xff000000   (2568 MB)
[    0.000000]     lowmem  : 0x40000000 - 0x5e000000   ( 480 MB)
[    0.000000]     modules : 0x3f000000 - 0x40000000   (  16 MB)
[    0.000000]       .text : 0x40008000 - 0x4060a688   (6154 kB)
[    0.000000]       .init : 0x4060b000 - 0x406351c4   ( 169 kB)
[    0.000000]       .data : 0x40636000 - 0x40676e60   ( 260 kB)
[    0.000000]        .bss : 0x40676e60 - 0x40724b54   ( 696 kB)
[    0.000000] NR_IRQS:330
[    0.000000] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 4294967ms
[    0.000000] Switching to timer-based delay loop
[    0.000000] Console: colour dummy device 80x30
[    0.000000] console [tty1] enabled
[    0.001192] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=10000)
[    0.001253] pid_max: default: 32768 minimum: 301
[    0.001403] Security Framework initialized
[    0.001462] AppArmor: AppArmor disabled by boot time parameter
[    0.001745] Mount-cache hash table entries: 512
[    0.002500] Initializing cgroup subsys cpuacct
[    0.002555] Initializing cgroup subsys memory
[    0.002634] Initializing cgroup subsys devices
[    0.002669] Initializing cgroup subsys freezer
[    0.002700] Initializing cgroup subsys net_cls
[    0.002728] Initializing cgroup subsys blkio
[    0.002755] Initializing cgroup subsys perf_event
[    0.002801] Initializing cgroup subsys net_prio
[    0.002922] CPU: Testing write buffer coherency: ok
[    0.003416] Setting up static identity map for 0x466838 - 0x466894
[    0.005078] devtmpfs: initialized
[    0.014607] regulator-dummy: no parameters
[    0.016927] NET: Registered protocol family 16
[    0.023188] DMA: preallocated 4096 KiB pool for atomic coherent allocations
[    0.024550] bcm2708.uart_clock = 0
[    0.026543] hw-breakpoint: found 6 breakpoint and 1 watchpoint registers.
[    0.026595] hw-breakpoint: maximum watchpoint size is 4 bytes.
[    0.026631] mailbox: Broadcom VideoCore Mailbox driver
[    0.026734] bcm2708_vcio: mailbox at f200b880
[    0.026838] bcm_power: Broadcom power driver
[    0.026875] bcm_power_open() -> 0
[    0.026900] bcm_power_request(0, 8)
[    0.527590] bcm_mailbox_read -> 00000080, 0
[    0.527628] bcm_power_request -> 0
[    0.527863] Serial: AMBA PL011 UART driver
[    0.528034] dev:f1: ttyAMA0 at MMIO 0x20201000 (irq = 83) is a PL011 rev3
[    0.901800] console [ttyAMA0] enabled
[    0.929010] bio: create slab  at 0
[    0.934393] SCSI subsystem initialized
[    0.938387] usbcore: registered new interface driver usbfs
[    0.944106] usbcore: registered new interface driver hub
[    0.949675] usbcore: registered new device driver usb
[    0.956034] NetLabel: Initializing
[    0.959475] NetLabel:  domain hash size = 128
[    0.963839] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.968979] NetLabel:  unlabeled traffic allowed by default
[    0.975238] Switching to clocksource stc
[    0.979452] FS-Cache: Loaded
[    0.982671] CacheFiles: Loaded
[    1.002058] NET: Registered protocol family 2
[    1.007306] TCP established hash table entries: 4096 (order: 3, 32768 bytes)
[    1.014558] TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
[    1.021113] TCP: Hash tables configured (established 4096 bind 4096)
[    1.027554] TCP: reno registered
[    1.030847] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    1.036715] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    1.043391] NET: Registered protocol family 1
[    1.048306] RPC: Registered named UNIX socket transport module.
[    1.054359] RPC: Registered udp transport module.
[    1.059080] RPC: Registered tcp transport module.
[    1.063828] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.071276] NetWinder Floating Point Emulator V0.97 (extended precision)
[    1.078203] bcm2708_dma: DMA manager at f2007000
[    1.083008] bcm2708_gpio: bcm2708_gpio_probe 406445f0
[    1.088539] vc-mem: phys_addr:0x00000000 mem_base=0x1ec00000 mem_size:0x20000000(512 MiB)
[    1.097197] major device number 252
[    1.097218] vma list size 12, page list size 2008, page size 4096
[    1.097237] allocated dma channel 2 (f2007200), initial state 00000008
[    1.097247] post-reset 00000008
[    1.098051] audit: initializing netlink socket (disabled)
[    1.103601] type=2000 audit(0.950:1): initialized
[    1.287257] VFS: Disk quotas dquot_6.5.2
[    1.291410] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    1.298523] FS-Cache: Netfs 'nfs' registered for caching
[    1.304330] NFS: Registering the id_resolver key type
[    1.309617] Key type id_resolver registered
[    1.313821] Key type id_legacy registered
[    1.318223] msgmni has been set to 935
[    1.324505] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    1.332360] io scheduler noop registered
[    1.336314] io scheduler deadline registered (default)
[    1.341613] io scheduler cfq registered
[    1.351837] crc32: CRC_LE_BITS = 64, CRC_BE BITS = 64
[    1.356914] crc32: self tests passed, processed 225944 bytes in 3099000 nsec
[    1.366554] crc32c: CRC_LE_BITS = 64
[    1.370183] crc32c: self tests passed, processed 225944 bytes in 1246000 nsec
[    1.479986] uart-pl011 dev:f1: no DMA platform data
[    1.484946] kgdb: Registered I/O driver kgdboc.
[    1.490290] vc-cma: Videocore CMA driver
[    1.494242] vc-cma: vc_cma_base      = 0x00000000
[    1.498954] vc-cma: vc_cma_size      = 0x00000000 (0 MiB)
[    1.504419] vc-cma: vc_cma_initial   = 0x00000000 (0 MiB)
[    1.520567] brd: module loaded
[    1.529630] loop: module loaded
[    1.533195] vchiq: vchiq_init_state: slot_zero = 0x5d000000, is_master = 0
[    1.541307] Loading iSCSI transport class v2.0-870.
[    1.547613] Rounding down aligned max_sectors from 4294967295 to 4294967288
[    1.555266] usbcore: registered new interface driver smsc95xx
[    1.561206] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[    1.767059] Core Release: 2.80a
[    1.770312] Setting default values for core params
[    1.775160] Finished setting default values for core params
[    1.980855] Using Buffer DMA mode
[    1.984196] Periodic Transfer Interrupt Enhancement - disabled
[    1.990070] Multiprocessor Interrupt Enhancement - disabled
[    1.995658] OTG VER PARAM: 0, OTG VER FLAG: 0
[    2.000055] Dedicated Tx FIFOs mode
[    2.005105] dwc_otg: Microframe scheduler enabled
[    2.005322] dwc_otg bcm2708_usb: DWC OTG Controller
[    2.010347] dwc_otg bcm2708_usb: new USB bus registered, assigned bus number 1
[    2.017693] dwc_otg bcm2708_usb: irq 75, io mem 0x00000000
[    2.023263] Init: Port Power? op_state=1
[    2.027198] Init: Power Port (0)
[    2.030582] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    2.037397] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.044682] usb usb1: Product: DWC OTG Controller
[    2.049443] usb usb1: Manufacturer: Linux 3.8.8-4-raspberrypi dwc_otg_hcd
[    2.056251] usb usb1: SerialNumber: bcm2708_usb
[    2.061676] hub 1-0:1.0: USB hub found
[    2.065484] hub 1-0:1.0: 1 port detected
[    2.069964] dwc_otg: FIQ disabled
[    2.069981] dwc_otg: NAK holdoff enabled
[    2.070002] Module dwc_common_port init
[    2.070244] Initializing USB Mass Storage driver...
[    2.075338] usbcore: registered new interface driver usb-storage
[    2.081460] USB Mass Storage support registered.
[    2.086652] mousedev: PS/2 mouse device common for all mice
[    2.093054] bcm2835-cpufreq: min=700000 max=950000 cur=700000
[    2.098915] bcm2835-cpufreq: switching to governor performance
[    2.104814] bcm2835-cpufreq: switching to governor performance
[    2.111519] cpuidle: using governor ladder
[    2.115631] cpuidle: using governor menu
[    2.119661] sdhci: Secure Digital Host Controller Interface driver
[    2.125851] sdhci: Copyright(c) Pierre Ossman
[    2.130340] sdhci: Enable low-latency mode
[    2.134764] platform supply vqmmc not found, using dummy regulator
[    2.141140] platform supply vmmc not found, using dummy regulator
[    2.147525]  platform: could not set regulator OCR (-22)
[    2.152921] mmc0: Switching to 3.3V signalling voltage  failed
[    2.189424] mmc0: SDHCI controller on BCM2708_Arasan [platform] using platform's DMA
[    2.197406] mmc0: BCM2708 SDHC host at 0x20300000 DMA 0 IRQ 16
[    2.203484] sdhci-pltfm: SDHCI platform and OF driver helper
[    2.211257] ledtrig-cpu: registered to indicate activity on CPUs
[    2.222750] usbcore: registered new interface driver usbhid
[    2.228426] usbhid: USB HID core driver
[    2.234379] zram: num_devices not specified. Using default: 1
[    2.242241] zram: Creating 1 devices ...
[    2.255134] TCP: cubic registered
[    2.258487] TCP: highspeed registered
[    2.262261] TCP: lp registered
[    2.265404] Initializing XFRM netlink socket
[    2.269912] Indeed it is in host mode hprt0 = 00021501
[    2.279370] NET: Registered protocol family 10
[    2.284721] NET: Registered protocol family 17
[    2.299389] Key type dns_resolver registered
[    2.304178] VFP support v0.3: implementor 41 architecture 1 part 20 variant b rev 5
[    2.314061] mmc0: read SD Status register (SSR) after 2 attempts
[    2.330106] registered taskstats version 1
[    2.336327] mmc0: new high speed SDHC card at address b368
[    2.349254] IMA: No TPM chip found, activating TPM-bypass!
[    2.355702] mmcblk0: mmc0:b368 SDC   15.0 GiB
[    2.370486] ### of_selftest(): No testcase data in device tree; not running tests
[    2.378418]  mmcblk0: p1 p2
[    2.409507] EXT4-fs (mmcblk0p2): mounted filesystem with writeback data mode. Opts: commit=120,data=writeback
[    2.419633] VFS: Mounted root (ext4 filesystem) on device 179:2.
[    2.435689] devtmpfs: mounted
[    2.439190] Freeing init memory: 168K
[    2.519351] usb 1-1: new high-speed USB device number 2 using dwc_otg
[    2.527271] Indeed it is in host mode hprt0 = 00001101
[    2.729784] usb 1-1: New USB device found, idVendor=0424, idProduct=9512
[    2.736531] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    2.745302] hub 1-1:1.0: USB hub found
[    2.750107] hub 1-1:1.0: 3 ports detected
[    3.029550] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[    3.139942] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[    3.146880] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    3.170024] smsc95xx v1.0.4
[    3.235045] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:0d:7e:b6
[    3.339593] usb 1-1.2: new full-speed USB device number 4 using dwc_otg
[    3.474004] usb 1-1.2: New USB device found, idVendor=0bda, idProduct=8150
[    3.482413] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    3.489852] usb 1-1.2: Product: USB 10/100 LAN
[    3.494336] usb 1-1.2: Manufacturer: REALTEK
[    3.498635] usb 1-1.2: SerialNumber: 1609
[    4.174850] udevd[161]: starting version 175
[    5.798122] rtl8150 1-1.2:1.0: eth1: rtl8150 is detected
[    5.836073] usbcore: registered new interface driver rtl8150
[   12.519210] mmc0: final write to SD card still running
[   22.469261] mmc0: Timeout waiting for hardware interrupt - cmd12.
[   22.476555] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[   24.630698] mmc0: final write to SD card still running
[   34.589270] mmc0: Timeout waiting for hardware interrupt - cmd12.
[   34.596567] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[   34.618797] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   35.187991] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   35.983110] nf_conntrack version 0.5.0 (7485 buckets, 29940 max)
[   36.010562] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
[   36.016701] IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
[   36.034023] IPVS: Each connection entry needs 212 bytes at least
[   36.034380] IPVS: Creating netns size=1200 id=0
[   36.039010] IPVS: ipvs loaded.
[   36.052568] IPVS: [dh] scheduler registered.
[   36.323262] IPVS: [lblc] scheduler registered.
[   36.413012] IPVS: [lblcr] scheduler registered.
[   36.492519] IPVS: [lc] scheduler registered.
[   36.590294] IPVS: [nq] scheduler registered.
[   36.671104] IPVS: [rr] scheduler registered.
[   36.750347] IPVS: [sed] scheduler registered.
[   36.829947] IPVS: [sh] scheduler registered.
[   36.918399] IPVS: [wlc] scheduler registered.
[   36.999563] IPVS: [wrr] scheduler registered.
[   37.163309] device-mapper: ioctl: 4.23.1-ioctl (2012-12-18) initialised: dm-devel@redhat.com
[   43.152590] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.160210] mmc0: DMA IRQ 6 ignored - results were reset
[   43.166739] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.174294] mmc0: DMA IRQ 6 ignored - results were reset
[   43.180716] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.188272] mmc0: DMA IRQ 6 ignored - results were reset
[   43.194672] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.202222] mmc0: DMA IRQ 6 ignored - results were reset
[   43.208591] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.216147] mmc0: DMA IRQ 6 ignored - results were reset
[   43.222475] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.230031] mmc0: DMA IRQ 6 ignored - results were reset
[   43.236351] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.243927] mmc0: DMA IRQ 6 ignored - results were reset
[   43.260173] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.267750] mmc0: DMA IRQ 6 ignored - results were reset
[   43.274182] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.281834] mmc0: DMA IRQ 6 ignored - results were reset
[   43.292858] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.300493] mmc0: DMA IRQ 6 ignored - results were reset
[   43.306855] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.314484] mmc0: DMA IRQ 6 ignored - results were reset
[   43.320781] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.328354] mmc0: DMA IRQ 6 ignored - results were reset
[   43.334706] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.342340] mmc0: DMA IRQ 6 ignored - results were reset
[   43.348584] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.356217] mmc0: DMA IRQ 6 ignored - results were reset
[   43.362491] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.370118] mmc0: DMA IRQ 6 ignored - results were reset
[   43.376332] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.383955] mmc0: DMA IRQ 6 ignored - results were reset
[   43.390184] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.397793] mmc0: DMA IRQ 6 ignored - results were reset
[   43.410295] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.417870] mmc0: DMA IRQ 6 ignored - results were reset
[   43.424434] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.432071] mmc0: DMA IRQ 6 ignored - results were reset
[   43.438516] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.446146] mmc0: DMA IRQ 6 ignored - results were reset
[   43.452631] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.460254] mmc0: DMA IRQ 6 ignored - results were reset
[   43.466637] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.474262] mmc0: DMA IRQ 6 ignored - results were reset
[   43.480723] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.488309] mmc0: DMA IRQ 6 ignored - results were reset
[   43.494742] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.502379] mmc0: DMA IRQ 6 ignored - results were reset
[   43.508754] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.516393] mmc0: DMA IRQ 6 ignored - results were reset
[   43.522872] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.530495] mmc0: DMA IRQ 6 ignored - results were reset
[   43.536855] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.544490] mmc0: DMA IRQ 6 ignored - results were reset
[   43.550858] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.558422] mmc0: DMA IRQ 6 ignored - results were reset
[   43.564755] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.572385] mmc0: DMA IRQ 6 ignored - results were reset
[   43.578636] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.586274] mmc0: DMA IRQ 6 ignored - results were reset
[   43.592567] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.600196] mmc0: DMA IRQ 6 ignored - results were reset
[   43.606447] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.614078] mmc0: DMA IRQ 6 ignored - results were reset
[   43.620413] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   43.627976] mmc0: DMA IRQ 6 ignored - results were reset
[   46.851581] net eth1: eth1: allmulti set
[   46.851803] net eth1: eth1: allmulti set
[   46.852033] net eth1: eth1: allmulti set
[   46.852319] net eth1: eth1: allmulti set
[   47.093767] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   47.277767] net eth1: eth1: allmulti set
[   48.019432] net eth1: eth1: allmulti set
[   48.317305] sctp: Hash tables configured (established 16384 bind 32768)
[   48.348036] DLM installed
[   48.556089] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   48.556930] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
[   52.119291] [sched_delayed] sched: RT throttling activated
[   53.664479] fenced (2328): /proc/2328/oom_adj is deprecated, please use /proc/2328/oom_score_adj instead.
[   59.457392] dlm: Using TCP for communications
[   79.509094] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   79.509152] mmc0: DMA IRQ 6 ignored - results were reset
[   79.510299] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   79.510358] mmc0: DMA IRQ 6 ignored - results were reset
[   79.511814] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   79.511873] mmc0: DMA IRQ 6 ignored - results were reset
[   79.513033] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   79.513091] mmc0: DMA IRQ 6 ignored - results were reset
[   79.903897] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   79.903954] mmc0: DMA IRQ 6 ignored - results were reset
[   79.905083] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   79.905143] mmc0: DMA IRQ 6 ignored - results were reset
[   79.906603] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   79.906660] mmc0: DMA IRQ 6 ignored - results were reset
[   79.907709] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   79.907766] mmc0: DMA IRQ 6 ignored - results were reset
[   85.733000] Adding 102396k swap on /var/swap.  Priority:-1 extents:129 across:1754856k SS
[   99.970093] mmc0: final write to SD card still running
[  109.989881] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  109.991062] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  110.894941] mmc0: final write to SD card still running
[  120.850409] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  120.851588] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  122.210909] mmc0: final write to SD card still running
[  132.230938] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  132.231139] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  135.950697] mmc0: final write to SD card still running
[  145.951591] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  145.951771] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  146.627959] mmc0: final write to SD card still running
[  156.592104] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  156.592282] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  160.637568] mmc0: final write to SD card still running
[  170.592766] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  170.593944] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  173.444624] mmc0: final write to SD card still running
[  183.473371] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  183.474546] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  192.866128] mmc0: final write to SD card still running
[  202.874301] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  202.875486] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  204.578845] mmc0: final write to SD card still running
[  214.594866] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  214.596044] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  214.689822] mmc0: final write to SD card still running
[  224.715338] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  224.715590] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  224.793116] mmc0: final write to SD card still running
[  234.795837] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  234.797008] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  234.893710] mmc0: final write to SD card still running
[  244.916311] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  244.917503] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  257.502511] mmc0: final write to SD card still running
[  267.517383] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  267.517604] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  269.601266] mmc0: final write to SD card still running
[  279.557951] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  279.559153] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  301.676887] mmc0: final write to SD card still running
[  311.679476] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  311.680788] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  336.640707] mmc0: Timeout waiting for hardware interrupt - cmd25.
[  336.640744] mmc0: resetting ongoing cmd 25DMA before 4096/4096 [12]/[19] complete
[  336.646472] mmcblk0: error -110 transferring data, sector 22771336, nr 152, cmd response 0x900, card status 0xc00
[  336.646663] mmc0: DMA IRQ 6 ignored - results were reset
[  336.646749] end_request: I/O error, dev mmcblk0, sector 22771425
[  336.646783] end_request: I/O error, dev mmcblk0, sector 22771432
[  336.646797] end_request: I/O error, dev mmcblk0, sector 22771440
[  336.646809] end_request: I/O error, dev mmcblk0, sector 22771448
[  336.646819] end_request: I/O error, dev mmcblk0, sector 22771456
[  336.646829] end_request: I/O error, dev mmcblk0, sector 22771464
[  336.646840] end_request: I/O error, dev mmcblk0, sector 22771472
[  336.646850] end_request: I/O error, dev mmcblk0, sector 22771480
[  336.646898] Buffer I/O error on device mmcblk0p2, logical block 2831057
[  336.646967] Buffer I/O error on device mmcblk0p2, logical block 2831058
[  336.646988] Buffer I/O error on device mmcblk0p2, logical block 2831059
[  336.647014] Buffer I/O error on device mmcblk0p2, logical block 2831060
[  336.647030] Buffer I/O error on device mmcblk0p2, logical block 2831061
[  336.647044] Buffer I/O error on device mmcblk0p2, logical block 2831062
[  336.647057] Buffer I/O error on device mmcblk0p2, logical block 2831063
[  336.647069] Buffer I/O error on device mmcblk0p2, logical block 2831064
[  336.647081] Buffer I/O error on device mmcblk0p2, logical block 2831065
[  336.647093] Buffer I/O error on device mmcblk0p2, logical block 2831066
[  336.647105] Buffer I/O error on device mmcblk0p2, logical block 2831067
[  336.647128] Buffer I/O error on device mmcblk0p2, logical block 2831068
[  336.647144] Buffer I/O error on device mmcblk0p2, logical block 2831069
[  336.647157] Buffer I/O error on device mmcblk0p2, logical block 2831070
[  336.647170] Buffer I/O error on device mmcblk0p2, logical block 2831071
[  336.647183] Buffer I/O error on device mmcblk0p2, logical block 2831072
[  336.647195] Buffer I/O error on device mmcblk0p2, logical block 2831073
[  336.647207] Buffer I/O error on device mmcblk0p2, logical block 2831074
[  336.647220] Buffer I/O error on device mmcblk0p2, logical block 2831075
[  336.647269] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:317: I/O error writing to inode 4400 (offset 44896256 size 77824 starting block 2846436)
[  344.988318] IPVS: __ip_vs_del_service: enter
[  345.064394] IPVS: __ip_vs_del_service: enter
[  346.701146] mmc0: Timeout waiting for hardware interrupt - cmd25.
[  346.701183] mmc0: resetting ongoing cmd 25DMA before 4096/4096 [8]/[17] complete
[  346.705247] mmcblk0: error -110 transferring data, sector 22740832, nr 136, cmd response 0x900, card status 0xc00
[  346.705477] mmc0: DMA IRQ 6 ignored - results were reset
[  346.705569] end_request: I/O error, dev mmcblk0, sector 22740889
[  346.705586] end_request: I/O error, dev mmcblk0, sector 22740896
[  346.705599] end_request: I/O error, dev mmcblk0, sector 22740904
[  346.705626] end_request: I/O error, dev mmcblk0, sector 22740912
[  346.705640] end_request: I/O error, dev mmcblk0, sector 22740920
[  346.705651] end_request: I/O error, dev mmcblk0, sector 22740928
[  346.705662] end_request: I/O error, dev mmcblk0, sector 22740936
[  346.705672] end_request: I/O error, dev mmcblk0, sector 22740944
[  346.705682] end_request: I/O error, dev mmcblk0, sector 22740952
[  346.705692] end_request: I/O error, dev mmcblk0, sector 22740960
[  346.705713] Buffer I/O error on device mmcblk0p2, logical block 2827244
[  346.705750] Buffer I/O error on device mmcblk0p2, logical block 2827245
[  346.705767] Buffer I/O error on device mmcblk0p2, logical block 2827246
[  346.705781] Buffer I/O error on device mmcblk0p2, logical block 2827247
[  346.705794] Buffer I/O error on device mmcblk0p2, logical block 2827248
[  346.705807] Buffer I/O error on device mmcblk0p2, logical block 2827249
[  346.705819] Buffer I/O error on device mmcblk0p2, logical block 2827250
[  346.705831] Buffer I/O error on device mmcblk0p2, logical block 2827251
[  346.705843] Buffer I/O error on device mmcblk0p2, logical block 2827252
[  346.705854] Buffer I/O error on device mmcblk0p2, logical block 2827253
[  346.705881] Buffer I/O error on device mmcblk0p2, logical block 2827254
[  346.705896] Buffer I/O error on device mmcblk0p2, logical block 2827255
[  346.705908] Buffer I/O error on device mmcblk0p2, logical block 2827256
[  346.705920] Buffer I/O error on device mmcblk0p2, logical block 2827257
[  346.705932] Buffer I/O error on device mmcblk0p2, logical block 2827258
[  346.705944] Buffer I/O error on device mmcblk0p2, logical block 2827259
[  346.705956] Buffer I/O error on device mmcblk0p2, logical block 2827260
[  346.705994] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:317: I/O error writing to inode 6807 (offset 247382016 size 69632 starting block 2842621)
[  348.439316] EXT4-fs error (device mmcblk0p2): htree_dirblock_to_tree:892: inode #200: block 7340: comm bash: bad entry in directory: directory entry across range - offset=0(0), inode=956301312, rec_len=11888, name_len=111
[  349.976650] mmc0: final write to SD card still running
[  360.001793] mmc0: Timeout waiting for hardware interrupt - cmd12.
[  360.002025] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[  360.221943] EXT4-fs error (device mmcblk0p2): htree_dirblock_to_tree:892: inode #200: block 7340: comm ls: bad entry in directory: directory entry across range - offset=0(0), inode=956301312, rec_len=11888, name_len=111
[  380.022721] mmc0: Timeout waiting for hardware interrupt - cmd25.
[  380.022759] mmc0: resetting ongoing cmd 25DMA before 4096/4096 [14]/[19] complete
[  380.026330] mmcblk0: error -110 transferring data, sector 22740960, nr 152, cmd response 0x900, card status 0xc00
[  380.026505] mmc0: DMA IRQ 6 ignored - results were reset
[  380.026591] end_request: I/O error, dev mmcblk0, sector 22741063
[  380.026608] end_request: I/O error, dev mmcblk0, sector 22741064
[  380.026619] end_request: I/O error, dev mmcblk0, sector 22741072
[  380.026630] end_request: I/O error, dev mmcblk0, sector 22741080
[  380.026658] end_request: I/O error, dev mmcblk0, sector 22741088
[  380.026670] end_request: I/O error, dev mmcblk0, sector 22741096
[  380.026681] end_request: I/O error, dev mmcblk0, sector 22741104
[  380.026702] Buffer I/O error on device mmcblk0p2, logical block 2827260
[  380.026724] Buffer I/O error on device mmcblk0p2, logical block 2827261
[  380.026739] Buffer I/O error on device mmcblk0p2, logical block 2827262
[  380.026751] Buffer I/O error on device mmcblk0p2, logical block 2827263
[  380.026791] Buffer I/O error on device mmcblk0p2, logical block 2827264
[  380.026847] Buffer I/O error on device mmcblk0p2, logical block 2827265
[  380.026867] Buffer I/O error on device mmcblk0p2, logical block 2827266
[  380.026882] Buffer I/O error on device mmcblk0p2, logical block 2827267
[  380.026908] Buffer I/O error on device mmcblk0p2, logical block 2827268
[  380.026924] Buffer I/O error on device mmcblk0p2, logical block 2827269
[  380.026938] Buffer I/O error on device mmcblk0p2, logical block 2827270
[  380.026950] Buffer I/O error on device mmcblk0p2, logical block 2827271
[  380.026963] Buffer I/O error on device mmcblk0p2, logical block 2827272
[  380.026975] Buffer I/O error on device mmcblk0p2, logical block 2827273
[  380.026988] Buffer I/O error on device mmcblk0p2, logical block 2827274
[  380.027000] Buffer I/O error on device mmcblk0p2, logical block 2827275
[  380.027022] Buffer I/O error on device mmcblk0p2, logical block 2827276
[  380.027038] Buffer I/O error on device mmcblk0p2, logical block 2827277
[  380.027052] Buffer I/O error on device mmcblk0p2, logical block 2827278
[  380.027090] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:317: I/O error writing to inode 6807 (offset 247447552 size 77824 starting block 2842639)
root@lb1:/usr/src# cd /sys/block/mmcblk0/device
root@lb1:/sys/block/mmcblk0/device# ls
block  csd   driver      fwrev  manfid  oemid  preferred_erase_size  serial     type
cid    date  erase_size  hwrev  name    power  scr                   subsystem  uevent
root@lb1:/sys/block/mmcblk0/device# for i in `ls *`
> do cat $i
> done
744a455344432020100805e67d00cb00
400e00325b590000782d7f800a400000
11/2012
512
0x0
0x1
0x000074
SDC
0x4a45
4194304
0235800000000000
0x0805e67d
SD
DRIVER=mmcblk
MMC_TYPE=SD
MMC_NAME=SDC
MODALIAS=mmc:block
cat: block:: Datei oder Verzeichnis nicht gefunden
cat: mmcblk0: Datei oder Verzeichnis nicht gefunden
cat: driver:: Datei oder Verzeichnis nicht gefunden
cat: bind: Datei oder Verzeichnis nicht gefunden
cat: mmc0:b368: Datei oder Verzeichnis nicht gefunden
DRIVER=mmcblk
MMC_TYPE=SD
MMC_NAME=SDC
MODALIAS=mmc:block
cat: unbind: Datei oder Verzeichnis nicht gefunden
cat: power:: Datei oder Verzeichnis nicht gefunden
cat: autosuspend_delay_ms: Datei oder Verzeichnis nicht gefunden
cat: control: Datei oder Verzeichnis nicht gefunden
cat: runtime_active_time: Datei oder Verzeichnis nicht gefunden
cat: runtime_status: Datei oder Verzeichnis nicht gefunden
cat: runtime_suspended_time: Datei oder Verzeichnis nicht gefunden
cat: subsystem:: Datei oder Verzeichnis nicht gefunden
cat: devices: Datei oder Verzeichnis nicht gefunden
cat: drivers: Datei oder Verzeichnis nicht gefunden
cat: drivers_autoprobe: Datei oder Verzeichnis nicht gefunden
cat: drivers_probe: Datei oder Verzeichnis nicht gefunden
DRIVER=mmcblk
MMC_TYPE=SD
MMC_NAME=SDC
MODALIAS=mmc:block
[  872.457680] EXT4-fs error (device mmcblk0p2): htree_dirblock_to_tree:892: inode #200: block 7340: comm ls: bad entry in directory: directory entry across range - offset=0(0), inode=956301312, rec_len=11888, name_len=111
[ 1077.819335] request_module: runaway loop modprobe binfmt-0000
[ 1080.424542] request_module: runaway loop modprobe binfmt-0000
...
[ 1087.992397] request_module: runaway loop modprobe binfmt-0000
[ 1104.143688] mmc0: final write to SD card still running
[ 1114.157600] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 1114.157790] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[ 1116.131697] mmc0: final write to SD card still running
[ 1126.158135] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 1126.158360] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[ 1126.527009] mmc0: final write to SD card still running
fsck offline the sdcard .. 
 fsck /dev/sdb2
fsck from util-linux 2.21.2
e2fsck 1.42.6 (21-Sep-2012)
/dev/sdb2: recovering journal
fsck.ext4: Bad magic number in super-block while trying to re-open /dev/sdb2
Signal (11) SIGSEGV si_code=SEGV_MAPERR fault addr=0x61
fsck.ext4[0x8071b26]
linux-gate.so.1(__kernel_rt_sigreturn+0x0)[0xb773d410]
/lib/libext2fs.so.2(ext2fs_mmp_stop+0x27)[0xb7712877]
fsck.ext4(fatal_error+0x4e)[0x80682de]
fsck.ext4(e2fsck_run_ext3_journal+0x316)[0x80677b6]
fsck.ext4(main+0x5ec)[0x805038c]
/lib/libc.so.6(__libc_start_main+0xf5)[0xb754c525]
fsck.ext4[0x8052e75]
rpool2:/mnt # dmesg | grep -i EXT4 | grep sdb
[  379.879229] EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: (null)
[  441.177255] EXT4-fs error (device sdb2): ext4_lookup:1384: inode #2: comm ls: deleted inode referenced: 8194
[  441.399216] EXT4-fs error (device sdb2): ext4_lookup:1384: inode #2: comm ls: deleted inode referenced: 8193
cd /usr
 ls -la
ls: cannot access lib: Input/output error
ls: cannot access sbin: Input/output error
ls: cannot access bin: Input/output error
ls: cannot access games: Input/output error
ls: cannot access share: Input/output error
ls: cannot access include: Input/output error
total 24
drwxr-xr-x 10 root root 4096 Dec 15 18:31 .
drwxr-xr-x 24 root root 4096 Apr 22 16:39 ..
d?????????  ? ?    ?       ?            ? bin
d?????????  ? ?    ?       ?            ? games
d?????????  ? ?    ?       ?            ? include
d?????????  ? ?    ?       ?            ? lib
drwxrwsr-x 11 root   50 4096 Apr 19 03:25 local
d?????????  ? ?    ?       ?            ? sbin
d?????????  ? ?    ?       ?            ? share
-rw-r--r--  1 root root 9427 Jan 14 20:33 src

so what , the OS been corupped using EXT4 on an SD class 10 card .. somewhat.

Past .config kernel compile changes since my last 3.8.7
(my working base is http://www.raspberrypi.org/phpBB3/download/file.php?id=3174 with 3.8.7 )

 diff a/linux/.config linux-rpi-3.8.7-temp/.config
109c109
< # CONFIG_CGROUP_PERF is not set
---
> CONFIG_CGROUP_PERF=y
673a674
> # CONFIG_NETFILTER_XT_TARGET_LED is not set
976c977
< CONFIG_NET_CLS_CGROUP=m
---
> CONFIG_NET_CLS_CGROUP=y
1006c1007
< CONFIG_NETPRIO_CGROUP=m
---
> CONFIG_NETPRIO_CGROUP=y
1166a1168
> # CONFIG_MAC80211_LEDS is not set
1172a1175
> CONFIG_RFKILL_LEDS=y
1316c1319
< # CONFIG_OF_SELFTEST is not set
---
> CONFIG_OF_SELFTEST=y
1529c1532,1537
< # CONFIG_TARGET_CORE is not set
---
> CONFIG_TARGET_CORE=y
> CONFIG_TCM_IBLOCK=m
> CONFIG_TCM_FILEIO=m
> CONFIG_TCM_PSCSI=m
> CONFIG_LOOPBACK_TARGET=m
> CONFIG_ISCSI_TARGET=m
1755a1764
> CONFIG_RT2X00_LIB_LEDS=y
1821a1831
> # CONFIG_KEYBOARD_LM8323 is not set
1867a1878
> # CONFIG_INPUT_PWM_BEEPER is not set
1868a1880
> # CONFIG_INPUT_PCAP is not set
1944a1957
> CONFIG_HVC_DRIVER=y
1946c1959
< # CONFIG_VIRTIO_CONSOLE is not set
---
> CONFIG_VIRTIO_CONSOLE=m
2188c2201
< # CONFIG_MFD_CORE is not set
---
> CONFIG_MFD_CORE=m
2195,2197c2208,2210
< # CONFIG_HTC_EGPIO is not set
< # CONFIG_HTC_PASIC3 is not set
< # CONFIG_HTC_I2CPLD is not set
---
> CONFIG_HTC_EGPIO=y
> CONFIG_HTC_PASIC3=m
> CONFIG_HTC_I2CPLD=y
2241c2254
< # CONFIG_EZX_PCAP is not set
---
> CONFIG_EZX_PCAP=y
2265a2279
> # CONFIG_REGULATOR_PCAP is not set
2483c2497
< # CONFIG_FB_SYS_FOPS is not set
---
> CONFIG_FB_SYS_FOPS=m
2498a2513
> # CONFIG_FB_TMIO is not set
2520a2536
> # CONFIG_BACKLIGHT_PWM is not set
2538c2554
< # CONFIG_LOGO_LINUX_MONO is not set
---
> CONFIG_LOGO_LINUX_MONO=y
2540,2541c2556,2557
< CONFIG_LOGO_LINUX_CLUT224=y
< # CONFIG_FB_SSD1307 is not set
---
> # CONFIG_LOGO_LINUX_CLUT224 is not set
> CONFIG_FB_SSD1307=m
2653a2670
> # CONFIG_HID_PICOLCD_LEDS is not set
2670a2688,2689
> # CONFIG_HID_WACOM is not set
> # CONFIG_HID_WIIMOTE is not set
2904c2923,2964
< # CONFIG_NEW_LEDS is not set
---
> CONFIG_NEW_LEDS=y
> CONFIG_LEDS_CLASS=m
>
> #
> # LED drivers
> #
> CONFIG_LEDS_ATMEL_PWM=m
> CONFIG_LEDS_LM3530=m
> CONFIG_LEDS_LM3642=m
> CONFIG_LEDS_PCA9532=m
> CONFIG_LEDS_PCA9532_GPIO=y
> CONFIG_LEDS_GPIO=m
> CONFIG_LEDS_LP3944=m
> CONFIG_LEDS_LP5521=m
> CONFIG_LEDS_LP5523=m
> CONFIG_LEDS_PCA955X=m
> CONFIG_LEDS_PCA9633=m
> CONFIG_LEDS_DAC124S085=m
> CONFIG_LEDS_REGULATOR=m
> CONFIG_LEDS_BD2802=m
> CONFIG_LEDS_LT3593=m
> CONFIG_LEDS_TCA6507=m
> CONFIG_LEDS_LM355x=m
> CONFIG_LEDS_OT200=m
> CONFIG_LEDS_BLINKM=m
> CONFIG_LEDS_TRIGGERS=y
>
> #
> # LED Triggers
> #
> CONFIG_LEDS_TRIGGER_TIMER=m
> CONFIG_LEDS_TRIGGER_ONESHOT=m
> CONFIG_LEDS_TRIGGER_HEARTBEAT=m
> CONFIG_LEDS_TRIGGER_BACKLIGHT=m
> CONFIG_LEDS_TRIGGER_CPU=y
> CONFIG_LEDS_TRIGGER_GPIO=m
> CONFIG_LEDS_TRIGGER_DEFAULT_ON=m
>
> #
> # iptables trigger is under Netfilter config (LED target)
> #
> CONFIG_LEDS_TRIGGER_TRANSIENT=m
2906c2966,2969
< # CONFIG_EDAC is not set
---
> CONFIG_EDAC=y
> CONFIG_EDAC_LEGACY_SYSFS=y
> # CONFIG_EDAC_DEBUG is not set
> CONFIG_EDAC_MM_EDAC=m
2981a3045
> # CONFIG_RTC_DRV_PCAP is not set
3091c3155
< # CONFIG_DGRP is not set
---
> CONFIG_DGRP=m
3131c3195
< # CONFIG_PWM is not set
---
> CONFIG_PWM=y
@popcornmix
Contributor

Can you update (sudo BRANCH=next rpi-update) and then replace the kernel file with this one:
https://dl.dropboxusercontent.com/u/3669512/temp/kernel_mmc_revert.img

This backs out the changes between 3.6 and 3.8 kernels for sdhci.c. Let me know if it fixes anything.

@hvenzke
hvenzke commented Apr 23, 2013

will give it an try as far as i have OS back working.
at moment im force an fsck with extra options .. filesystem editing path guide by Tso .-)

/dev/sdb2 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes

Running additional passes to resolve blocks claimed by more than one inode...

Pass 1B: Rescanning for multiply-claimed blocks

well an here the reason why the ext4 died

1B:
Illegal block number passed to ext2fs_test_block_bitmap #268435455 for multiply claimed block map
Illegal block number passed to ext2fs_test_block_bitmap #1065259007 for multiply claimed block map

...
1D:
clone_file_block: internal error: can't find dup_blk for 33026

@oerik
oerik commented Apr 23, 2013

Negative at boot time:

[ 2.139912] mmc0: read SD Status register (SSR) after 3 attempts
[ 2.162078] mmc0: new high speed SDHC card at address 0007
[ 2.170416] mmcblk0: mmc0:0007 SD4GB 3.70 GiB
[ 2.179081] mmcblk0: p1 p2

[ 15.286984] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 15.934531] mmc0: final write to SD card still running
[ 25.957972] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 25.967136] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[ 1019.457103] mmc0: final write to SD card still running
[ 1029.464236] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 1029.465397] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900

Rebooted twice, same error. I'll keep it running for a night. I've seen these kind of errors before on other platforms, it had something to do with the initial/calculated SD clock frequency.

uname -a
Linux raspberrypi 3.8.8+ #421 PREEMPT Tue Apr 23 16:57:08 BST 2013 armv6l GNU/Linux

@hvenzke
hvenzke commented Apr 23, 2013

Due the ext4 died , had to reinstall rasbian and restore my configs .
I droped the overlocking to None .
Willl stress the kernel_mmc_revert.img kernel over night with kernel make 3.8.8 and revert tomorrow

@hvenzke
hvenzke commented Apr 24, 2013

So my test results +12h later ( still compiling 3.8.8 code from git 3.8.y ) : NO issues YET.

I assume this error
" mmc0: DMA IRQ 6 ignored - results were reset "
relalated to the not per default enabled dmaer_master module.

I assume this error :
"mc0: Timeout waiting for hardware interrupt - cmd12.
[ 183.474546] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900"

Realated to the over-clocked and over-voltaged SD card controler - that Going OUT OF spec
Class 6 and 10 use lover volages the class 2 and 4 as i read some RPI Forums and specs past half day.

therefore ... for CLASS 6 and 10 Cards : Overclock is FORBIDDEN

resolution of the "BUG" :
add Kernel waring for users that Overcloking class 6 and 10 WILL CAUSE Filesystem corruption - ALLWAYS !!!
Ill update my Own Hack page and add an warning asap.

@popcornmix
Contributor

@remsnet are you talking about default 3.8.y kernel or kernel_mmc_revert.img?

"mmc0: DMA IRQ 6 ignored - results were reset" is harmless, and should probably be removed.
(as is "mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it")

"mmc0: Timeout waiting for hardware interrupt - cmd12."/"mmcblk0: error -110 sending stop command" are dangerous.

Overclocking can cause sdcard errors. It tends to be board specific (i.e. some Pi's are more tolerant to overclock and sdcard corruption). I believe it is generally core_freq that causes the sdcard corruption (rather than arm_freq or sdram_freq).

I don't think this is related to class6/class10. Overclock doesn't affect the speed we communicate with sdcards.
We should never use the sdcard speeds that require a lower sdcard IO voltage. We certainly didn't for 3.6.y kernel, but the code has changed in 3.8.y kernel, so it is possible that is no longer behaving as expected.

kernel_mmc_revert.img reverts the mmc driver to 3.6.y version.

@hvenzke
hvenzke commented Apr 24, 2013

yes !

kernel_mmc_revert.img, test done past 14H with that.
I did that to be shure not caused by code changes.

Will give the 3.8.y same try as far as it been compiled .. still it runs yet..

yes the RPI is somewhat tolerant faults .. Thanks to driver coders as well.
BUT the Standard & Today ( kingston ,Transceed, others ) class 6/10 cards always use lower voltages then 2 & 4 .
This are seems SPEC realated if i not readed wrongly . Google for it .. you will find some usefull Infos.

root@lb1:/usr/src/linux# dmesg
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.8.8+ (dc4@dc4-arm-01) (gcc version 4.7.2 20120731 (prerelease) (crosstool-NG linaro-1.13.1+bzr2458 - Linaro GCC 2012.08) ) #421 PREEMPT Tue Apr 23 16:57:08 BST 2013
[ 0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[ 0.000000] Machine: BCM2708
[ 0.000000] cma: CMA: reserved 16 MiB at 1d000000
[ 0.000000] Memory policy: ECC disabled, Data cache writeback
[ 0.000000] On node 0 totalpages: 122880
[ 0.000000] free_area_init_node: node 0, pgdat c0580244, node_mem_map c062a000
[ 0.000000] Normal zone: 960 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 121920 pages, LIFO batch:31
[ 0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[ 0.000000] pcpu-alloc: [0] 0
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 121920
[ 0.000000] Kernel command line: dma.dmachans=0x7f35 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708.boardrev=0xf bcm2708.serial=0xf30d7eb6 smsc95xx.macaddr=B8:27:EB:0D:7E:B6 sdhci-bcm2708.emmc_clock_freq=100000000 vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000 dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
[ 0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[ 0.000000] __ex_table already sorted, skipping sort
[ 0.000000] Memory: 480MB = 480MB total
[ 0.000000] Memory: 464500k/464500k available, 27020k reserved, 0K highmem
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
[ 0.000000] fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
[ 0.000000] vmalloc : 0xde800000 - 0xff000000 ( 520 MB)
[ 0.000000] lowmem : 0xc0000000 - 0xde000000 ( 480 MB)
[ 0.000000] modules : 0xbf000000 - 0xc0000000 ( 16 MB)
[ 0.000000] .text : 0xc0008000 - 0xc05287e4 (5250 kB)
[ 0.000000] .init : 0xc0529000 - 0xc054a52c ( 134 kB)
[ 0.000000] .data : 0xc054c000 - 0xc0580ac0 ( 211 kB)
[ 0.000000] .bss : 0xc0580ac0 - 0xc0629180 ( 674 kB)
[ 0.000000] NR_IRQS:330
[ 0.000000] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 4294967ms
[ 0.000000] Switching to timer-based delay loop
[ 0.000000] Console: colour dummy device 80x30
[ 0.000000] console [tty1] enabled
[ 0.001176] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=10000)
[ 0.001241] pid_max: default: 32768 minimum: 301
[ 0.001582] Mount-cache hash table entries: 512
[ 0.002287] Initializing cgroup subsys cpuacct
[ 0.002343] Initializing cgroup subsys devices
[ 0.002377] Initializing cgroup subsys freezer
[ 0.002408] Initializing cgroup subsys blkio
[ 0.002502] CPU: Testing write buffer coherency: ok
[ 0.002936] Setting up static identity map for 0x3cf7b8 - 0x3cf814
[ 0.004497] devtmpfs: initialized
[ 0.015083] NET: Registered protocol family 16
[ 0.021070] DMA: preallocated 4096 KiB pool for atomic coherent allocations
[ 0.022141] bcm2708.uart_clock = 0
[ 0.023428] hw-breakpoint: found 6 breakpoint and 1 watchpoint registers.
[ 0.023481] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.023518] mailbox: Broadcom VideoCore Mailbox driver
[ 0.023628] bcm2708_vcio: mailbox at f200b880
[ 0.023728] bcm_power: Broadcom power driver
[ 0.023766] bcm_power_open() -> 0
[ 0.023792] bcm_power_request(0, 8)
[ 0.524485] bcm_mailbox_read -> 00000080, 0
[ 0.524525] bcm_power_request -> 0
[ 0.524552] Serial: AMBA PL011 UART driver
[ 0.524699] dev:f1: ttyAMA0 at MMIO 0x20201000 (irq = 83) is a PL011 rev3
[ 0.855162] console [ttyAMA0] enabled
[ 0.877564] bio: create slab at 0
[ 0.882475] SCSI subsystem initialized
[ 0.886562] usbcore: registered new interface driver usbfs
[ 0.892145] usbcore: registered new interface driver hub
[ 0.897743] usbcore: registered new device driver usb
[ 0.904081] Switching to clocksource stc
[ 0.908325] FS-Cache: Loaded
[ 0.911458] CacheFiles: Loaded
[ 0.926614] NET: Registered protocol family 2
[ 0.931904] TCP established hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.939154] TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.945676] TCP: Hash tables configured (established 4096 bind 4096)
[ 0.952150] TCP: reno registered
[ 0.955406] UDP hash table entries: 256 (order: 0, 4096 bytes)
[ 0.961310] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[ 0.967873] NET: Registered protocol family 1
[ 0.972782] RPC: Registered named UNIX socket transport module.
[ 0.978826] RPC: Registered udp transport module.
[ 0.983548] RPC: Registered tcp transport module.
[ 0.988295] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.995444] bcm2708_dma: DMA manager at f2007000
[ 1.000229] bcm2708_gpio: bcm2708_gpio_probe c0559cd0
[ 1.005720] vc-mem: phys_addr:0x00000000 mem_base=0x1ec00000 mem_size:0x20000000(512 MiB)
[ 1.014898] audit: initializing netlink socket (disabled)
[ 1.020499] type=2000 audit(0.890:1): initialized
[ 1.146917] VFS: Disk quotas dquot_6.5.2
[ 1.151057] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[ 1.158047] FS-Cache: Netfs 'nfs' registered for caching
[ 1.163808] NFS: Registering the id_resolver key type
[ 1.169094] Key type id_resolver registered
[ 1.173299] Key type id_legacy registered
[ 1.177737] msgmni has been set to 939
[ 1.183525] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[ 1.191341] io scheduler noop registered
[ 1.195297] io scheduler deadline registered (default)
[ 1.200582] io scheduler cfq registered
[ 1.212225] Console: switching to colour frame buffer device 82x26
[ 1.223652] kgdb: Registered I/O driver kgdboc.
[ 1.230795] vc-cma: Videocore CMA driver
[ 1.236603] vc-cma: vc_cma_base = 0x00000000
[ 1.243216] vc-cma: vc_cma_size = 0x00000000 (0 MiB)
[ 1.250418] vc-cma: vc_cma_initial = 0x00000000 (0 MiB)
[ 1.266419] brd: module loaded
[ 1.276169] loop: module loaded
[ 1.281401] vchiq: vchiq_init_state: slot_zero = 0xdd000000, is_master = 0
[ 1.290823] Loading iSCSI transport class v2.0-870.
[ 1.298424] usbcore: registered new interface driver smsc95xx
[ 1.306152] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[ 1.513807] Core Release: 2.80a
[ 1.518721] Setting default values for core params
[ 1.525232] Finished setting default values for core params
[ 1.732621] Using Buffer DMA mode
[ 1.737627] Periodic Transfer Interrupt Enhancement - disabled
[ 1.745149] Multiprocessor Interrupt Enhancement - disabled
[ 1.752434] OTG VER PARAM: 0, OTG VER FLAG: 0
[ 1.758512] Dedicated Tx FIFOs mode
[ 1.764173] dwc_otg: Microframe scheduler enabled
[ 1.764371] dwc_otg bcm2708_usb: DWC OTG Controller
[ 1.771073] dwc_otg bcm2708_usb: new USB bus registered, assigned bus number 1
[ 1.780101] dwc_otg bcm2708_usb: irq 32, io mem 0x00000000
[ 1.787314] Init: Port Power? op_state=1
[ 1.792950] Init: Power Port (0)
[ 1.797953] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 1.806682] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.815790] usb usb1: Product: DWC OTG Controller
[ 1.822378] usb usb1: Manufacturer: Linux 3.8.8+ dwc_otg_hcd
[ 1.829892] usb usb1: SerialNumber: bcm2708_usb
[ 1.837020] hub 1-0:1.0: USB hub found
[ 1.842663] hub 1-0:1.0: 1 port detected
[ 1.848681] dwc_otg: FIQ enabled
[ 1.848700] dwc_otg: NAK holdoff enabled
[ 1.848738] Module dwc_common_port init
[ 1.848956] Initializing USB Mass Storage driver...
[ 1.855768] usbcore: registered new interface driver usb-storage
[ 1.863623] USB Mass Storage support registered.
[ 1.870315] mousedev: PS/2 mouse device common for all mice
[ 1.878321] bcm2835-cpufreq: min=700000 max=700000 cur=700000
[ 1.885931] bcm2835-cpufreq: switching to governor powersave
[ 1.893337] bcm2835-cpufreq: switching to governor powersave
[ 1.900681] cpuidle: using governor ladder
[ 1.906409] cpuidle: using governor menu
[ 1.911998] sdhci: Secure Digital Host Controller Interface driver
[ 1.919843] sdhci: Copyright(c) Pierre Ossman
[ 1.925895] sdhci: Enable low-latency mode
[ 1.968209] mmc0: SDHCI controller on BCM2708_Arasan [platform] using platform's DMA
[ 1.979542] mmc0: BCM2708 SDHC host at 0x20300000 DMA 2 IRQ 77
[ 1.987191] sdhci-pltfm: SDHCI platform and OF driver helper
[ 1.997169] usbcore: registered new interface driver usbhid
[ 2.007690] usbhid: USB HID core driver
[ 2.015931] TCP: cubic registered
[ 2.023099] Initializing XFRM netlink socket
[ 2.031245] NET: Registered protocol family 17
[ 2.039680] Key type dns_resolver registered
[ 2.048310] Indeed it is in host mode hprt0 = 00021501
[ 2.058552] VFP support v0.3: implementor 41 architecture 1 part 20 variant b rev 5
[ 2.078931] registered taskstats version 1
[ 2.094414] Waiting for root device /dev/mmcblk0p2...
[ 2.103555] mmc0: new high speed SDHC card at address b368
[ 2.118854] mmcblk0: mmc0:b368 SDC 15.0 GiB
[ 2.139948] mmcblk0: p1 p2
[ 2.278262] usb 1-1: new high-speed USB device number 2 using dwc_otg
[ 2.286838] Indeed it is in host mode hprt0 = 00001101
[ 2.459588] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 2.471846] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 2.488530] usb 1-1: New USB device found, idVendor=0424, idProduct=9512
[ 2.497562] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 2.507158] devtmpfs: mounted
[ 2.512764] hub 1-1:1.0: USB hub found
[ 2.519025] Freeing init memory: 132K
[ 2.525122] hub 1-1:1.0: 3 ports detected
[ 2.808535] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[ 2.919094] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[ 2.928396] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 2.941035] smsc95xx v1.0.4
[ 3.003224] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:0d:7e:b6
[ 3.098557] usb 1-1.2: new full-speed USB device number 4 using dwc_otg
[ 3.242685] usb 1-1.2: New USB device found, idVendor=0bda, idProduct=8150
[ 3.259592] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3.278888] usb 1-1.2: Product: USB 10/100 LAN
[ 3.285593] usb 1-1.2: Manufacturer: REALTEK
[ 3.292025] usb 1-1.2: SerialNumber: 1609
[ 3.398606] usb 1-1.3: new low-speed USB device number 5 using dwc_otg
[ 3.538302] usb 1-1.3: New USB device found, idVendor=1c4f, idProduct=0002
[ 3.547329] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 3.560467] usb 1-1.3: Product: USB Keykoard
[ 3.566865] usb 1-1.3: Manufacturer: USB
[ 3.586903] input: USB USB Keykoard as /devices/platform/bcm2708_usb/usb1/1-1/1-1.3/1-1.3:1.0/input/input0
[ 3.603528] hid-generic 0003:1C4F:0002.0001: input,hidraw0: USB HID v1.10 Keyboard [USB USB Keykoard] on usb-bcm2708_usb-1.3/input0
[ 3.633548] input: USB USB Keykoard as /devices/platform/bcm2708_usb/usb1/1-1/1-1.3/1-1.3:1.1/input/input1
[ 3.664429] hid-generic 0003:1C4F:0002.0002: input,hidraw1: USB HID v1.10 Device [USB USB Keykoard] on usb-bcm2708_usb-1.3/input1
[ 4.231671] udevd[155]: starting version 175
[ 5.535883] rtl8150 1-1.2:1.0: eth1: rtl8150 is detected
[ 5.633633] usbcore: registered new interface driver rtl8150
[ 10.665452] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 11.159572] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 11.972767] bcm2835 ALSA card created!
[ 11.986398] bcm2835 ALSA chip created!
[ 11.997720] bcm2835 ALSA chip created!
[ 12.005726] bcm2835 ALSA chip created!
[ 12.016127] bcm2835 ALSA chip created!
[ 12.027131] bcm2835 ALSA chip created!
[ 12.035041] bcm2835 ALSA chip created!
[ 12.046275] bcm2835 ALSA chip created!
[ 20.685266] net eth1: eth1: allmulti set
[ 21.118462] net eth1: eth1: allmulti set
[ 22.393466] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
[ 24.016884] bcm2835-cpufreq: switching to governor ondemand
[ 24.016931] bcm2835-cpufreq: switching to governor ondemand
[ 25.188346] nf_conntrack version 0.5.0 (7515 buckets, 30060 max)
[ 25.288554] NET: Registered protocol family 10
[ 25.290952] net eth1: eth1: allmulti set
[ 25.291275] net eth1: eth1: allmulti set
[ 25.328352] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
[ 25.328473] IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
[ 25.328865] IPVS: Creating netns size=1200 id=0
[ 25.328948] IPVS: ipvs loaded.
[ 26.045631] IPVS: sync thread started: state = MASTER, mcast_ifn = eth1, syncid = 0, id = 0
[ 28.354748] net eth1: eth1: allmulti set
[ 29.571133] Adding 102396k swap on /var/swap. Priority:-1 extents:129 across:1754856k SS
[ 45.111214] IPVS: [wlc] scheduler registered.
[ 568.924629] device-mapper: ioctl: 4.23.1-ioctl (2012-12-18) initialised: dm-devel@redhat.com
[ 884.732541] IPVS: __ip_vs_del_service: enter
[ 5595.335291] net eth1: eth1: allmulti set
[35927.007688] major device number 248
[35927.007713] vma list size 12, page list size 2008, page size 4096
[35927.007749] allocated dma channel 0 (f2007000), initial state 00000008
[35927.007761] post-reset 00000008
[38509.667439] smsc95xx 1-1.1:1.0 eth0: kevent 2 may have been dropped
root@lb1:/usr/src/linux# update
-su: update: Kommando nicht gefunden.
root@lb1:/usr/src/linux# uptime
12:49:02 up 12:31, 1 user, load average: 1,31, 1,24, 1,28

ls -la /boot
insgesamt 25004
drwxr-xr-x 2 root root 16384 Jan 1 1970 .
drwxr-xr-x 24 root root 4096 Apr 23 21:57 ..
-rwxr-xr-x 1 root root 17764 Apr 23 22:00 bootcode.bin
-rwxr-xr-x 1 root root 142 Dez 15 18:43 cmdline.txt
-rwxr-xr-x 1 root root 1234 Apr 23 20:02 config.txt
-rwxr-xr-x 1 root root 41 Apr 23 22:00 .firmware_revision
-rwxr-xr-x 1 root root 2264 Apr 23 22:00 fixup_cd.dat
-rwxr-xr-x 1 root root 5902 Apr 23 22:00 fixup.dat
-rwxr-xr-x 1 root root 8826 Apr 23 22:00 fixup_x.dat
-rwxr-xr-x 1 root root 137 Dez 16 20:51 issue.txt
-rwxr-xr-x 1 root root 2946872 Apr 23 22:00 kernel.3.8.8.img
-rwxr-xr-x 1 root root 9703816 Apr 23 22:00 kernel_emergency.img
-rwxr-xr-x 1 root root 2946864 Apr 23 22:03 kernel.img
-rwxr-xr-x 1 root root 2946864 Apr 23 22:03 kernel_mmc_revert.img
-rwxr-xr-x 1 root root 569976 Apr 23 22:00 start_cd.elf
-rwxr-xr-x 1 root root 2697556 Apr 23 22:00 start.elf
-rwxr-xr-x 1 root root 3661156 Apr 23 22:00 start_x.elf

.

@oerik
oerik commented Apr 24, 2013

@popcornmix I've swapped the 4G SD card (It is an Kingston 4G microsd card) with an 8GB Transcend TS8GSDHC10. The Transcend does not return any errors.

If you want I can send the 4G card to you.

@popcornmix
Contributor

@oerik
Can you confirm the card's behaviour. Is it:
Works without problem on 3.6.11
Produces errors with 3.8.8
Produces errors with kernel_mmc_revert.img
?

@oerik
oerik commented Apr 24, 2013

@popcornmix Yes confirmed, I've tripple checked it.

@oerik
oerik commented Apr 25, 2013

@popcornmix Another rpi is having the same issues, same brand of (micro) SD card. Defaults settings (no overclocking):
[ 888.348054] mmc0: final write to SD card still running
[ 898.305008] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 898.306222] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[ 898.797336] mmc0: final write to SD card still running
[ 908.825241] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 908.826431] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900

uname -a
Linux raspberrypi 3.8.8+ #420 PREEMPT Tue Apr 23 16:17:46 BST 2013 armv6l GNU/Linux

@hvenzke
hvenzke commented Apr 26, 2013

@popcornmix , stresstested with 3.8.y with kernel compile : NO issues seen yet - Overclock disabled.
May you tell me what code differences may made between 3.6.y 3.8.y in some few words as you said ?

So it shuold not - from my point - an code issue with named branded sd cards , if i am not wrong.

Possible issues with No name cards that are not listed as supported at elinux.org.
as there listed some "Micro*" branded cards with RED flag.

@oerik may you able to update the list at http://elinux.org/RPi_SD_cards ?
add your blaming SD to not (yet ) support once...

If not send me the details i´ll do that for you. pictures i.e

@popcornmix , I added an Overclock Waring for class6 and 10 cards at http://elinux.org to drop down
Blaming Kernel DEV with Overclock SD Cards issues.
137 threads at raspberry.org ´s Forum are more then required about this...

@oerik
oerik commented Apr 26, 2013

@remsnet, My el cheapo SD cards are working flawless under 3.6.11, even when overclocking 900. So there is a regression. I'm afraid that I am not the only one with this problem once 3.8 is going in various distro's. I'll update the rpi_sd_cards list.

@hvenzke
hvenzke commented Apr 26, 2013

@oerik well, yeah ... maybe there may an regression..
.. but hard to find without an fully debug flaged kernel( and IO, USB & Storange parts i.e )
and an 230K serial speed attached kdb debugger +log server ..
.. that whuold the next step on you site i.e , provide mass output on kernel debug level trace ..

Thats why i think´d to add an better an warning, either due the too many Forum threads, bevor we overload Gurus here. I think Stabily may more important then get the SD Stuff to work & stable on Overclocked mode. Call me party stuid or whatesoever.. . In the 199x i was doing equial tests things either with Alan and others for serval kernel parts due interests in HamRadio ( i am dg2sbv ) On Linux. .. my First kernel was 0.4.2

I don´t thing there will be an short fix YET abalible soon - correct me some one if you on other mood ..

@oerik
oerik commented Apr 29, 2013

Small update, no improvement in Linux raspberrypi 3.8.10+ #429 PREEMPT Sun Apr 28 20:55:17 BST 2013 armv6l GNU/Linux.

@hvenzke
hvenzke commented May 8, 2013

Small update as well ,

Due to EXT4 filesystem cashes , 2 of my Pis where go offline and the wuole partion table was GONE !!
All files & dirs went to lost+found

( i allways have somewhere an fdisk -l stored .. so no data lost after fsck -)

Aother issue with EXT4 outage happen while fresh installed rabian december release at FIRST repoot with resizefs2.

I have my data on XFS due that now .

@popcornmix
Contributor

I spotted a possible spinlock issue in sdcard driver (on 3.8 tree), and have committed a fix.
I've pushed it to next firmware tree.
Can you test it?

@Vincent-C

I think the fix might have worked...previously, with 3.8.11 and older from the firmware -next branch, I would see the same errors as above, but they don't show up in dmesg anymore after a reboot and waiting for about half a hour.

$ dmesg | grep mmc
[ 0.000000] Kernel command line: dma.dmachans=0x7f35 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708.boardrev=0xe bcm2708.serial=0x68e0c684 smsc95xx.macaddr=B8:27:EB:E0:C6:84 sdhci-bcm2708.emmc_clock_freq=100000000 vc_mem.mem_base=0x1fa00000 vc_mem.mem_size=0x20000000 dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
[ 1.988793] mmc0: SDHCI controller on BCM2708_Arasan [platform] using platform's DMA
[ 2.000188] mmc0: BCM2708 SDHC host at 0x20300000 DMA 2 IRQ 77
[ 2.119692] Waiting for root device /dev/mmcblk0p2...
[ 2.150906] mmc0: read SD Status register (SSR) after 2 attempts
[ 2.172736] mmc0: new high speed SDHC card at address b368
[ 2.186914] mmcblk0: mmc0:b368 NCard 7.47 GiB
[ 2.195053] mmcblk0: p1 p2
[ 2.265249] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 28.998259] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 29.432833] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
$ uname -a
Linux vcheng.org 3.8.12+ #440 PREEMPT Wed May 8 23:14:03 BST 2013 armv6l GNU/Linux

@oerik
oerik commented May 9, 2013

@popcornmix Negative, same errors are reported on 3.8.12+.

@hvenzke
hvenzke commented May 9, 2013

@oerik did you overclock ?

@Vincent-C , try stress the sd card & FS i.e with an kernel or samba4 compile,
this create doszens of tons of mini files .

During the past lost-found digging near 90% all dirs contained just that files . some where realy DUPLICATED ID´s
Some where EXT4 COMPRESSED flagged ( how can that be , if i not explicit enable that ..)

@popcornmix updated , starting blaming the FS with kernel compile 3.9.y ..
no errors yet.

Linux nagios1 3.8.12+ #440 PREEMPT Wed May 8 23:14:03 BST 2013 armv6l GNU/Linux

@oerik
oerik commented May 9, 2013

@remsnet Tested with default settings (no overclock). Other powersupply, overclocking (to the extreme), other RPI boards, doesn't matter. I have 4 of these SD cards and the behave all the same. At rootfs mount time the errors occur immediately.

@hvenzke
hvenzke commented May 10, 2013
[2443.807495] mmc0: Timeout waiting for hardware interrupt - cmd12.
[82443.808657] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[82447.703908] mmc0: final write to SD card still running
[82457.728280] mmc0: Timeout waiting for hardware interrupt - cmd12.
[82457.729432] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[82487.865070] mmc0: final write to SD card still running
[82497.890562] mmc0: Timeout waiting for hardware interrupt - cmd12.
[82497.891728] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
nagios1 ~ $ uptime
 11:14:58 up 22:56,  2 users,  load average: 0,60, 0,49, 0,26

Corruped within 1 day . last job on system - compiled 3.9.y kern

@hvenzke
hvenzke commented May 10, 2013

started doing an experiment : move /dev/mmcblk0p2 ( root ) to ext2 with 4 k blocks

@Ferroin
Ferroin commented May 10, 2013

I have seen this transiently myself with a PNY brand class 4 16GB microSDHC card, but only on one of my two systems (interestingly the one with the problem is the newer of the two boards).

Only seems to happen for me on the first few boots after an unclean shutdown (first time I saw it was after having to reboot the system using SysRq after it had hung).

@ottuzzi
ottuzzi commented May 13, 2013

Hi,

I can see same errors as reported above (mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900) on a stock 3.6.11+ kernel simply overclocking to 1000MHz. Removing overclock fixes the problem.
I'm on a Samsung 8GB Class 10 SDHC - Model code: MB-SP8GA/EU.
Fortunately the ext4 FS did not get corrupted.

@hvenzke
hvenzke commented May 13, 2013

@oerik well , ext4 on class10 SHDC/SHDX cards are stable untill you don´t deploy / create mulible 1000 files.
Thats the same for micro cards

Thats what i verified with 32g and 64G sd cards.

as CUERRENT , i assume that ANY SD cards ( vendor don´t matter ) class 6 &10 SHDC/SHDX bigger then 8GB
are UNSTABLE for STORANGE USE purposes.

My Both corosync app servers use 16G , and they ONLY stable WITHOUT IO / storange load, at current.

@oerik
oerik commented May 13, 2013

@remsnet I disagree with you. My case is the exact oposite: 4Gb class 10, raspberry pi that writes every 5 minutes to an rrd database (single file), nothing more. Logging files are on ramlog, so very little write access.

@hvenzke
hvenzke commented May 13, 2013

well 1 file all 5min are not an issue.
the issues happens when you write massive files 10.000x of them with 24h.
i.e compile an 3.9.y kernel .

while im done that past weeks , EVERY card ´s OS are blamed , distro don´t matter.

@Ferroin
Ferroin commented May 13, 2013

Did some more testing, it appears that higher overclocking increases the chances of this happening.
Overclocking to 950Mhz produces errors much more frequently than 800Mhz, and underclocking at 600Mhz seems to work with no problems whatsoever.

@popcornmix
Contributor

@Ferroin (and others)
Can you make it clear whether sdcard issues are a regression in 3.8 kernel, that weren't present in 3.6 kernel?

Also, we have a belief that overclock related sdcard corruption may be due to noise in the power supply (which tends to increase with load). Can you try a different power supply?

@ottuzzi
ottuzzi commented May 13, 2013

I'm sure I saw sdcard issues on a 3.6.11+ stock kernel with stock bootcode.bin only when I overclocked the raspi.
Please note that I overclocked for the sake of overclocking... not because I needed some more MHz: in my own case the load was very low.
I can try a second power supply tonight and let you know tomorrow or the the day after...

@Ferroin
Ferroin commented May 13, 2013

@popcornmix
Can't really be sure whether it's a regression or not, I have seen the errors with both 3.8 and 3.6, but they seem more common in 3.8.

I've tested with a couple different power supplies, the only power source that I haven't seen these errors with is a computer USB port.

Also, just finished doing even more testing, decreasing init_emmc_clock seems to improve things.

@popcornmix
Contributor

@Ferroin
It would be very interesting if you could confirm corruption behaviour is improved with power from USB port.

@ottuzzi
ottuzzi commented May 13, 2013

Hi,

just tested overclock with my rpi running 'Linux raspberrypi 3.6.11+ #446 PREEMPT' with a new power supply: after few seconds I saw ' mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900' in syslog. This is clearly not a regression of 3.8.

@hvenzke
hvenzke commented May 13, 2013

@ottuzzi well thats my expirence here near as well unter 2 rules

1 - no overclock
2 - not write massive files to the SD card ( high IO load )

2 seems an BUG definity somewhere Arround the USB IO Stack .
i.e the sdcard controler, USB software stack / drivers , ext4.

@Ferroin
Ferroin commented May 16, 2013

@popcornmix
I believe the improvement from the USB port is because the computer I tested it with has a very good PSU (Active PFC, line noise cancellation, high stability independent 5, 12, and 3,3 volt rails, etc), and is rated at 150% of the wattage the computer is actually using. I just did some testing powered from my laptop (a cheap Lenovo ThinkPad), and it dosen't appear to do any better than the wall-wart psu's I've tried.

Interestingly, the USB wall plugs I usually use to power the Pi seem to do better when plugged into a high quality power strip than they do when plugged directly into the wall.

@Ferroin
Ferroin commented May 16, 2013

Looking back into really old logfiles from the 3.2 branch, I see some of the same errors, although less frequently. Based on this, i would say that this is probably not a regression, but just a problem that has been made worse by some other change in the kernel.

I'm going to try some heavy I/O testing on my desktop, which has a USB SDHCI card reader and is running 3.9.2-gentoo, if that shows the same errors, then we know that the problem is not specific to the SDHCI driver for the raspberry pi.

@popcornmix
Contributor

@Ferroin
If you can be sure a "perfect" power supply improves the relibabilty of sdcard, then that is very imprortant information.
There is no point trying to solve the corruption problem in the kernel driver, if it is caused by a glitch from the power supply.
If it can be proved that the power supply is responsible for sdcard corruption then we can try to find known good supplies to recommend for people with problems and perhaps look at adding extra filtering on future boards.

@Ferroin
Ferroin commented May 16, 2013

While I agree that it does appear to be caused by power problems, i think that we should still look into trying to find a way to harden the kernel driver against problems like this so that it doesn’t cause data corruption (i.e. if we get such an error, on a write, we could try to verify the write against the buffer in memory).

I would love to help prove that the problem is in the supply, i really don't have the right tools to verify this properly.

@oerik
oerik commented May 16, 2013

@Ferroin You could be right that the mmc errors are just the symptoms. I do not believe it is directly related to the power supply since I can reproduce it very easy on my rPI boards. 3.6 does not return any errors, 3.8 does...

@hvenzke
hvenzke commented May 16, 2013

@popcornmix , @oerik @Ferroin

please re-read #298

from my Point the reported looks like an im-propper and or / incomplete written code .
Maybe i am wrong .
RT-Linux Code are very very strickt in i.e. io sheduling

some coder of you i.e shuold handel the SD(HC/HX) Code very strickt
( i dont have the skill for that myself )
complete and / or correct it to the SD(HC/HX/DX/HX) SPEC´s.
(https://www.sdcard.org/developers/overview/sdio/sdio_spec/Simplified_SDIO_Card_Spec.pdf)
(http://www.siwawi.arubi.uni-kl.de/avr_projects/arm_projects/arm_memcards/)

a and well with cards +4G SPEC BUG to take care
https://www.google.de/url?sa=t&rct=j&q=&esrc=s&source=web&cd=4&cad=rja&ved=0CEoQFjAD&url=https%3A%2F%2Fmy.st.com%2Fpublic%2FSTe2ecommunities%2Fmcu%2FLists%2Fcortex_mx_stm32%2FBug%2520for%2520large%2520SD%2520cards%2520in%2520SDIO%2520library%2520code&ei=JBOVUdvaNIKCtAbf94CIDA&usg=AFQjCNFzFgItiJ08-S_FZTqedM8b9nDxsA&bvm=bv.46471029,d.Yms

IF HELP by mainstream IO/USB Freaks required , then we shuold asap call them.

+1 Millions RPI affected world wide when use class 6/10 cards, next Miillion reached en 2013 i.e .
I whuold call this an serios Issue.

re Power:
I have taken 2.5Amp at prescise regulated 5V , but either THAT DON´T FIX the issue with my class 6/10 cards.
Thus my nagios and smb servers often down and broken frequentry.

The experiment with ext2 fs as ROOT result - > FAILD

@Ferroin
Ferroin commented May 16, 2013

@remsnet
As of yet, there is nothing other than the fact that they involve the same driver that this issue and #298 are related.

I am pretty certain that it is an issue resulting from the combination of the driver having poor error recovery and noise sensitivity in the hardware. I can relatively reliably induce these errors in my default config (static 800 MHz CPU, 400 MHz GPU, 400 MHz RAM, 100MHz SD clock, 16G class 4 PNY microSDHC card) by doing a sysbench IO benchmark while moving a neodymium magnet back and forth within ~10cm of the board.

@Ferroin
Ferroin commented May 17, 2013

Problem seems to be slightly more likely with the bcm_dmaer module loaded.

@P33M
Contributor
P33M commented May 21, 2013

I'm going to chime in on this thread because I believe I have a minimally complex failure case that produces this exact result.

Because my MS Sidewinder keyboard was pretty much unusable with the Pi until the FIQ got split transaction support, I bought the cheapest keyboard on ebuyer - it's a:

Bus 001 Device 006: ID 046e:6000 Behavior Tech. Computer Corp.
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0         8
  idVendor           0x046e Behavior Tech. Computer Corp.
  idProduct          0x6000
  bcdDevice            1.80
  iManufacturer           1 BTC
  iProduct                2 USB Keyboard
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           59
    bNumInterfaces          2
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xa0
      (Bus Powered)
      Remote Wakeup
    MaxPower               98mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      1 Boot Interface Subclass
      bInterfaceProtocol      1 Keyboard
      iInterface              0
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.10
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      69
         Report Descriptors:
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval              10
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      0 No Subclass
      bInterfaceProtocol      0 None
      iInterface              0
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.10
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      50
         Report Descriptors:
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0003  1x 3 bytes
        bInterval              10
Device Status:     0x0000
  (Bus Powered)

The great thing (or annoying depending on perspective) about this keyboard is that it will reliably knacker my SD card with the same -110 "timeout waiting for hardware interrupt" messages. It does this when hid-generic is probing the device and seems associated with a failed usb_submit_urb call. This is fatal if I leave the device plugged in while booting - invariably my filesystem ends up corrupt.

Typical log of broken behaviour:

[  182.922767] usb 1-1.2.5.4: new low-speed USB device number 6 using dwc_otg
[  183.029459] usb 1-1.2.5.4: New USB device found, idVendor=046e, idProduct=6000
[  183.029491] usb 1-1.2.5.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[  183.029507] usb 1-1.2.5.4: Product: USB Keyboard
[  183.029522] usb 1-1.2.5.4: Manufacturer: BTC
[  183.044149] input: BTC  USB Keyboard as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2.5/1-1.2.5.4/1-1.2.5.4:1.0/input/input0
[  183.045032] hid-generic 0003:046E:6000.0001: input,hidraw0: USB HID v1.10 Keyboard [BTC  USB Keyboard] on usb-bcm2708_usb-1.2.5.4/input0
[  193.042827] hid-generic 0003:046E:6000.0002: usb_submit_urb(ctrl) failed: -1
[  193.042890] hid-generic 0003:046E:6000.0002: timeout initializing reports
[  193.045838] input: BTC  USB Keyboard as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2.5/1-1.2.5.4/1-1.2.5.4:1.1/input/input1
[  193.047830] hid-generic 0003:046E:6000.0002: input,hidraw1: USB HID v1.10 Device [BTC  USB Keyboard] on usb-bcm2708_usb-1.2.5.4/input1

Typical log if I break my filesystem:

[    2.642966] usb 1-1: New USB device found, idVendor=0424, idProduct=9512
[    2.657571] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    2.672700] hub 1-1:1.0: USB hub found
[    2.684576] hub 1-1:1.0: 3 ports detected
[    2.972843] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[    3.103354] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[    3.130742] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    3.165726] smsc95xx v1.0.4
[    3.247827] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:97:5a:1f
[    3.362897] usb 1-1.2: new high-speed USB device number 4 using dwc_otg
[    3.493519] usb 1-1.2: New USB device found, idVendor=1a40, idProduct=0201
[    3.520923] usb 1-1.2: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[    3.550104] usb 1-1.2: Product: USB 2.0 Hub [MTT]
[    3.583667] hub 1-1.2:1.0: USB hub found
[    3.610593] hub 1-1.2:1.0: 7 ports detected
[    3.912836] usb 1-1.2.5: new high-speed USB device number 5 using dwc_otg
[    4.043619] usb 1-1.2.5: New USB device found, idVendor=1a40, idProduct=0201
[    4.072730] usb 1-1.2.5: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[    4.106510] usb 1-1.2.5: Product: USB 2.0 Hub [MTT]
[    4.128830] hub 1-1.2.5:1.0: USB hub found
[    4.140887] hub 1-1.2.5:1.0: 7 ports detected
[    4.179084] udevd[154]: starting version 175
[    4.432928] usb 1-1.2.5.2: new low-speed USB device number 6 using dwc_otg
[    4.578315] usb 1-1.2.5.2: New USB device found, idVendor=045e, idProduct=0039
[    4.598046] usb 1-1.2.5.2: New USB device strings: Mfr=1, Product=3, SerialNumber=0
[    4.613582] usb 1-1.2.5.2: Product: Microsoft 5-Button Mouse with IntelliEye(TM)
[    4.628633] usb 1-1.2.5.2: Manufacturer: Microsoft
[    4.669080] input: Microsoft Microsoft 5-Button Mouse with IntelliEye(TM) as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2.5/1-1.2.5.2/1-1.2.5.2:1.0/input/input0
[    4.743270] hid-generic 0003:045E:0039.0001: input,hidraw0: USB HID v1.10 Mouse [Microsoft Microsoft 5-Button Mouse with IntelliEye(TM)] on usb-bcm2708_usb-1.2.5.2/input0
[    4.902870] usb 1-1.2.5.4: new low-speed USB device number 7 using dwc_otg
[    5.048773] usb 1-1.2.5.4: New USB device found, idVendor=046e, idProduct=6000
[    5.092670] usb 1-1.2.5.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    5.151879] usb 1-1.2.5.4: Product: USB Keyboard
[    5.182719] usb 1-1.2.5.4: Manufacturer: BTC
[    5.234606] input: BTC  USB Keyboard as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2.5/1-1.2.5.4/1-1.2.5.4:1.0/input/input1
[    5.300862] hid-generic 0003:046E:6000.0002: input,hidraw1: USB HID v1.10 Keyboard [BTC  USB Keyboard] on usb-bcm2708_usb-1.2.5.4/input0
[   15.376203] hid-generic 0003:046E:6000.0003: usb_submit_urb(ctrl) failed: -1
[   15.417770] hid-generic 0003:046E:6000.0003: timeout initializing reports
[   15.453369] input: BTC  USB Keyboard as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2.5/1-1.2.5.4/1-1.2.5.4:1.1/input/input2
[   15.493088] hid-generic 0003:046E:6000.0003: input,hidraw2: USB HID v1.10 Device [BTC  USB Keyboard] on usb-bcm2708_usb-1.2.5.4/input1
[   24.442569] mmc0: Timeout waiting for hardware interrupt - cmd25.
[   24.462872] mmcblk0: error -110 transferring data, sector 1696160, nr 24, cmd response 0x900, card status 0xc00
[   24.483431] end_request: I/O error, dev mmcblk0, sector 1696168
[   24.500060] end_request: I/O error, dev mmcblk0, sector 1696176
[   24.515670] Aborting journal on device mmcblk0p2-8.
[   24.545903] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[   24.564696] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[   24.583165] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[   24.601522] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[   24.619843] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[   24.638103] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[   24.656346] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[   28.472876] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: Detected aborted journal
[   28.504677] EXT4-fs (mmcblk0p2): Remounting filesystem read-only
[   28.521591] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: Detected aborted journal
[   28.550737] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: Detected aborted journal
[   28.563594] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349:
[   28.585499] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349:
[   28.607068] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: Detected aborted journal
[   28.661198] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349:
[   28.668143] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349:
[   28.785181] Detected aborted journal
[   28.797944] Detected aborted journal
[   28.817712] Detected aborted journal
[   28.904880] Detected aborted journal
[   28.920482] Detected aborted journal
[   28.928148] Detected aborted journalDetected aborted journal
[   29.013488]
[   29.077584] Registered led device: led0
[   29.124671]
[   32.207674] EXT4-fs error (device mmcblk0p2): ext4_remount:4584: Abort forced by user
[   32.224515] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   45.527322] EXT4-fs error (device mmcblk0p2): ext4_remount:4584: Abort forced by user
[   45.563663] EXT4-fs error (device mmcblk0p2): ext4_remount:4584: Abort forced by user
[   55.175706] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1
@popcornmix
Contributor

@P33M

Interesting. So the USB is causing a timout in sdcard driver.
Does the usb_submit_urb call failure leave interrupts disabled or held off for a long period?

You would hope that if the sdcard times out doing something, it would retry. I guess it doesn't.
If the behaviour following a timeout is sdcard corruption, wouldn't makeing the timeout infinite be preferable?
It might complete successfully later. It might hang forever (and hopefully not get a chance to corrupt further).

Is your test on 3.8 kernel? Also on 3.6 kernel?

@hvenzke
hvenzke commented May 21, 2013

@popcornmix can do us an favor an made an DEBUG flagged kernel-firmware avalible that i.e has mutch more sd-card/OTG/USB2/.../
Hardware relalated DEBUG enabled.
we have to find the holy shit BUG(s)

@ottuzzi
ottuzzi commented May 21, 2013

Actually, on my system, I have not usb devices attached but I have a remote drive mounted via davfs that probably is causing quite a bit of network traffic: if I'm not mistaken even the ethernet port is attached to usb... isn't it?

@P33M
Contributor
P33M commented May 21, 2013

Interestingly with the fiq_split branch I only get the more benign error:

[    5.089267] input: BTC  USB Keyboard as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2.5/1-1.2.5.5/1-1.2.5.5:1.0/input/input1
[    5.157636] hid-generic 0003:046E:6000.0002: input,hidraw1: USB HID v1.10 Keyboard [BTC  USB Keyboard] on usb-bcm2708_usb-1.2.5.5/input0
[   15.219809] hid-generic 0003:046E:6000.0003: usb_submit_urb(ctrl) failed: -1
[   15.238818] hid-generic 0003:046E:6000.0003: timeout initializing reports
[   15.256451] input: BTC  USB Keyboard as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2.5/1-1.2.5.5/1-1.2.5.5:1.1/input/input2
[   15.287568] hid-generic 0003:046E:6000.0003: input,hidraw2: USB HID v1.10 Device [BTC  USB Keyboard] on usb-bcm2708_usb-1.2.5.5/input1
[   23.046281] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   23.600613] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   30.162990] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   30.180470] mmc0: DMA IRQ 6 ignored - results were reset
[   34.529606] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1
[   38.824104] Adding 102396k swap on /var/swap.  Priority:-1 extents:1 across:102396k SS

hexxeh doesn't have the vanilla 3.8 branch available on rpi-update and my build machine is ... poorly. I will report back with a test on 3.8.

My gut feeling is that the hid probing is taking place with preemption disabled - though I have not yet traced the states of the two concurrent systems (sd card read and usbhid probe)

@ghollingworth
Contributor

I believe this is the same problem as the dequeue problem, basically when you get some kind of error the system tries to dequeue a transaction. This then turns off a whole bunch of interrupts whilst it waits for the channel halted interrupt (which should happen when you disable an active channel)

But I never saw this work properly and instead the channel never halts… (we basically lose a channel) it might explain the same problem.

Are you sure we're not seeing a channel halted interrupt that's being ignored because of the changes a while back to get rid of the channel halted messages? This is easy to reproduce with a stty –F /dev/ttyUSB0

Gordon

From: P33M <notifications@github.commailto:notifications@github.com>
Reply-To: raspberrypi/linux <reply@reply.github.commailto:reply@reply.github.com>
Date: Tuesday, 21 May 2013 15:46
To: raspberrypi/linux <linux@noreply.github.commailto:linux@noreply.github.com>
Subject: Re: [linux] mmcblk0 error -110 errors (regression?) (#280)

I'm going to chime in on this thread because I believe I have a minimal failure case that produces this exact result.

Because my MS Sidewinder keyboard was pretty much unusable with the Pi until the FIQ got split transaction support, I bought the cheapest keyboard on ebuyer - it's a:

Bus 001 Device 006: ID 046e:6000 Behavior Tech. Computer Corp.
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 1.10
bDeviceClass 0 (Defined at Interface level)
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 8
idVendor 0x046e Behavior Tech. Computer Corp.
idProduct 0x6000
bcdDevice 1.80
iManufacturer 1 BTC
iProduct 2 USB Keyboard
iSerial 0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 59
bNumInterfaces 2
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xa0
(Bus Powered)
Remote Wakeup
MaxPower 98mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 3 Human Interface Device
bInterfaceSubClass 1 Boot Interface Subclass
bInterfaceProtocol 1 Keyboard
iInterface 0
HID Device Descriptor:
bLength 9
bDescriptorType 33
bcdHID 1.10
bCountryCode 0 Not supported
bNumDescriptors 1
bDescriptorType 34 Report
wDescriptorLength 69
Report Descriptors:
** UNAVAILABLE **
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0008 1x 8 bytes
bInterval 10
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 1
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 3 Human Interface Device
bInterfaceSubClass 0 No Subclass
bInterfaceProtocol 0 None
iInterface 0
HID Device Descriptor:
bLength 9
bDescriptorType 33
bcdHID 1.10
bCountryCode 0 Not supported
bNumDescriptors 1
bDescriptorType 34 Report
wDescriptorLength 50
Report Descriptors:
** UNAVAILABLE **
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x82 EP 2 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0003 1x 3 bytes
bInterval 10
Device Status: 0x0000
(Bus Powered)

The great thing (or annoying depending on perspective) about this keyboard is that it will reliably knacker my SD card with the same -110 "timeout waiting for hardware interrupt" messages. It does this when hid-generic is probing the device and seems associated with a failed usb_submit_urb call. This is fatal if I leave the device plugged in while booting - invariably my filesystem ends up corrupt.

Typical log of broken behaviour:

[ 183.029459] usb 1-1.2.5.4: New USB device found, idVendor=046e, idProduct=6000
[ 183.029491] usb 1-1.2.5.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 183.029507] usb 1-1.2.5.4: Product: USB Keyboard
[ 183.029522] usb 1-1.2.5.4: Manufacturer: BTC
[ 183.044149] input: BTC USB Keyboard as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2.5/1-1.2.5.4/1-1.2.5.4:1.0/input/input0
[ 183.045032] hid-generic 0003:046E:6000.0001: input,hidraw0: USB HID v1.10 Keyboard [BTC USB Keyboard] on usb-bcm2708_usb-1.2.5.4/input0
[ 193.042827] hid-generic 0003:046E:6000.0002: usb_submit_urb(ctrl) failed: -1
[ 193.042890] hid-generic 0003:046E:6000.0002: timeout initializing reports
[ 193.045838] input: BTC USB Keyboard as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2.5/1-1.2.5.4/1-1.2.5.4:1.1/input/input1
[ 193.047830] hid-generic 0003:046E:6000.0002: input,hidraw1: USB HID v1.10 Device [BTC USB Keyboard] on usb-bcm2708_usb-1.2.5.4/input1

Typical log if I break my filesystem:

[ 2.642966] usb 1-1: New USB device found, idVendor=0424, idProduct=9512
[ 2.657571] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 2.672700] hub 1-1:1.0: USB hub found
[ 2.684576] hub 1-1:1.0: 3 ports detected
[ 2.972843] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[ 3.103354] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[ 3.130742] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.165726] smsc95xx v1.0.4
[ 3.247827] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:97:5a:1f
[ 3.362897] usb 1-1.2: new high-speed USB device number 4 using dwc_otg
[ 3.493519] usb 1-1.2: New USB device found, idVendor=1a40, idProduct=0201
[ 3.520923] usb 1-1.2: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 3.550104] usb 1-1.2: Product: USB 2.0 Hub [MTT]
[ 3.583667] hub 1-1.2:1.0: USB hub found
[ 3.610593] hub 1-1.2:1.0: 7 ports detected
[ 3.912836] usb 1-1.2.5: new high-speed USB device number 5 using dwc_otg
[ 4.043619] usb 1-1.2.5: New USB device found, idVendor=1a40, idProduct=0201
[ 4.072730] usb 1-1.2.5: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 4.106510] usb 1-1.2.5: Product: USB 2.0 Hub [MTT]
[ 4.128830] hub 1-1.2.5:1.0: USB hub found
[ 4.140887] hub 1-1.2.5:1.0: 7 ports detected
[ 4.179084] udevd[154]: starting version 175
[ 4.432928] usb 1-1.2.5.2: new low-speed USB device number 6 using dwc_otg
[ 4.578315] usb 1-1.2.5.2: New USB device found, idVendor=045e, idProduct=0039
[ 4.598046] usb 1-1.2.5.2: New USB device strings: Mfr=1, Product=3, SerialNumber=0
[ 4.613582] usb 1-1.2.5.2: Product: Microsoft 5-Button Mouse with IntelliEye(TM)
[ 4.628633] usb 1-1.2.5.2: Manufacturer: Microsoft
[ 4.669080] input: Microsoft Microsoft 5-Button Mouse with IntelliEye(TM) as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2.5/1-1.2.5.2/1-1.2.5.2:1.0/input/input0
[ 4.743270] hid-generic 0003:045E:0039.0001: input,hidraw0: USB HID v1.10 Mouse [Microsoft Microsoft 5-Button Mouse with IntelliEye(TM)] on usb-bcm2708_usb-1.2.5.2/input0
[ 4.902870] usb 1-1.2.5.4: new low-speed USB device number 7 using dwc_otg
[ 5.048773] usb 1-1.2.5.4: New USB device found, idVendor=046e, idProduct=6000
[ 5.092670] usb 1-1.2.5.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 5.151879] usb 1-1.2.5.4: Product: USB Keyboard
[ 5.182719] usb 1-1.2.5.4: Manufacturer: BTC
[ 5.234606] input: BTC USB Keyboard as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2.5/1-1.2.5.4/1-1.2.5.4:1.0/input/input1
[ 5.300862] hid-generic 0003:046E:6000.0002: input,hidraw1: USB HID v1.10 Keyboard [BTC USB Keyboard] on usb-bcm2708_usb-1.2.5.4/input0
[ 15.376203] hid-generic 0003:046E:6000.0003: usb_submit_urb(ctrl) failed: -1
[ 15.417770] hid-generic 0003:046E:6000.0003: timeout initializing reports
[ 15.453369] input: BTC USB Keyboard as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2.5/1-1.2.5.4/1-1.2.5.4:1.1/input/input2
[ 15.493088] hid-generic 0003:046E:6000.0003: input,hidraw2: USB HID v1.10 Device [BTC USB Keyboard] on usb-bcm2708_usb-1.2.5.4/input1
[ 24.442569] mmc0: Timeout waiting for hardware interrupt - cmd25.
[ 24.462872] mmcblk0: error -110 transferring data, sector 1696160, nr 24, cmd response 0x900, card status 0xc00
[ 24.483431] end_request: I/O error, dev mmcblk0, sector 1696168
[ 24.500060] end_request: I/O error, dev mmcblk0, sector 1696176
[ 24.515670] Aborting journal on device mmcblk0p2-8.
[ 24.545903] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[ 24.564696] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[ 24.583165] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[ 24.601522] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[ 24.619843] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[ 24.638103] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[ 24.656346] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:4550: Journal has aborted
[ 28.472876] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: Detected aborted journal
[ 28.504677] EXT4-fs (mmcblk0p2): Remounting filesystem read-only
[ 28.521591] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: Detected aborted journal
[ 28.550737] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: Detected aborted journal
[ 28.563594] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349:
[ 28.585499] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349:
[ 28.607068] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349: Detected aborted journal
[ 28.661198] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349:
[ 28.668143] EXT4-fs error (device mmcblk0p2): ext4_journal_start_sb:349:
[ 28.785181] Detected aborted journal
[ 28.797944] Detected aborted journal
[ 28.817712] Detected aborted journal
[ 28.904880] Detected aborted journal
[ 28.920482] Detected aborted journal
[ 28.928148] Detected aborted journalDetected aborted journal
[ 29.013488]
[ 29.077584] Registered led device: led0
[ 29.124671]
[ 32.207674] EXT4-fs error (device mmcblk0p2): ext4_remount:4584: Abort forced by user
[ 32.224515] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 45.527322] EXT4-fs error (device mmcblk0p2): ext4_remount:4584: Abort forced by user
[ 45.563663] EXT4-fs error (device mmcblk0p2): ext4_remount:4584: Abort forced by user
[ 55.175706] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1

—
Reply to this email directly or view it on GitHubhttps://github.com/raspberrypi/linux/issues/280#issuecomment-18213443.
@popcornmix
Contributor

https://github.com/Hexxeh/rpi-firmware/commits/next is vanilla 3.8, so
sudo BRANCH=next rpi-update
will get it.

@hvenzke
hvenzke commented May 21, 2013

@ottuzzi all devices , include the sd card , are connected via the USB(2) Path - as far as i understood the dmesg kernel output. someone is allowed to correct me..

@ghollingworth
Contributor

No not the SDCard that is connected through SDIO

Gordon

From: "", <notifications@github.commailto:notifications@github.com>
Reply-To: raspberrypi/linux <reply@reply.github.commailto:reply@reply.github.com>
Date: Tuesday, 21 May 2013 16:28
To: raspberrypi/linux <linux@noreply.github.commailto:linux@noreply.github.com>
Cc: Gordon Hollingworth <gordon@raspberrypi.orgmailto:gordon@raspberrypi.org>
Subject: Re: [linux] mmcblk0 error -110 errors (regression?) (#280)

@ottuzzihttps://github.com/ottuzzi all devices , include the sd card , are connected via the USB(2) Path - as far as i understood the dmesg kernel output. someone is allowed to correct me..

Reply to this email directly or view it on GitHubhttps://github.com/raspberrypi/linux/issues/280#issuecomment-18216538.

@Ferroin
Ferroin commented May 21, 2013

@remsnet
The SD card is connected through an independent SDHCI interface on the CPU itself.

@P33M
Contributor
P33M commented May 22, 2013

Edit: My somewhat artificial issue is diverging a bit but bear with me - I think this one is still lurking.

I can reproduce this on 2 different types of SD card (Sandisk 4GB class 4, Kingston 8GB class 10) and with both 3.6 and 3.8 branches. fiq_split doesn't give me the same behaviour, only the benign "missed completion" messages.

If I boot with keyboard attached but eliminate the probing with hid quirks, the timeout errors disappear (presumably because I just told Linux not to probe the boot subclass interface for the keyboard)

I note for the previously recorded errors that other people have seen, the preceding line:

[ 1104.143688] mmc0: final write to SD card still running
[ 1114.157600] mmc0: Timeout waiting for hardware interrupt - cmd12.

is quite telling. The sdhci-bcm2708 driver has already waited 150ms for the stop command to go through as per the default sync_after_dma=1 so this timeout may be genuine - bug is noted in the code comments.

By default the timeout that sdhci specifies is 10 seconds. In my case I can reliably break the same timeout because - guess what - the HID probing timeout is also 10 seconds.

This behaviour in itself seems broken in the sdhci layer - basically the sdhci irq top layer will schedule host->finish_tasklet on completion of a SD card command (whatever it may be, DMA or otherwise). If however the timeout timer expires before this tasklet is scheduled AND preemption or interrupts are disabled then when re-enabled the timer softirq will run before the finish tasklet (softirq pecking order). The sdhci timeout forces the finish_tasklet to discard all data and return -ETIMEDOUT, even though the command may have completed successfully. This is fatal if we have a high latency on either the command completing or preemption (10 seconds is a bit ridiculous, but what about a slow SD card having to do a lot of read-erase-write for multiple sectors coupled with a badly timed preempt_disable()?).

I will poke around a bit more - see what happens if I bump finish_tasklet to the high priority schedule.

@hvenzke
hvenzke commented May 23, 2013

well maybe thats still the same as #58 issue

sdhci-bcm2708.missing_status=0 sdhci-bcm2708.sync_after_dma=0
at cmdline.txt required per default for class 6 and 10 cards ?!?...

EDIT1:
popcornmix ´s patch realated to that was made at 3.1 kernel
patches-3.1/0940-add_sync_after_dma_module_parameter.patch

EDIT2: while googling for "sync_after_dma="
near 99,99% of the results recoment set explicit it to 0 as it caused usb timeouts when set to 1
Thus - sorry - i thing thats an bad coded patch. seems Not fit to class 6 and 10 cards.

@licaon-kter

I get some of these too on my model B w/ 256Mb, I'm up to BRANCH=next, only ethernet connection, accessing via ssh, usually updating some git ( like dwb browser or i3wm ), Sandisk Ultra 8Gb Class10 30mb/s (SDSDU-008G-U46):

May 18 23:55:42 raspberrypi vmunix: [ 270.869777] mmc0: final write to SD card still running
May 18 23:55:52 raspberrypi vmunix: [ 280.819080] mmc0: Timeout waiting for hardware interrupt - cmd12.
May 18 23:55:53 raspberrypi vmunix: [ 280.820254] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
May 19 00:03:35 raspberrypi vmunix: [ 744.194226] mmc0: final write to SD card still running
May 19 00:03:48 raspberrypi vmunix: [ 754.216707] mmc0: Timeout waiting for hardware interrupt - cmd12.
May 19 00:03:48 raspberrypi vmunix: [ 754.217878] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
May 19 02:13:54 raspberrypi vmunix: [ 6903.350661] mmc0: final write to SD card still running
May 19 02:14:05 raspberrypi vmunix: [ 6913.317326] mmc0: Timeout waiting for hardware interrupt - cmd12.
May 19 02:14:05 raspberrypi vmunix: [ 6913.317500] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
May 22 12:37:52 raspberrypi vmunix: [295812.804925] mmc0: final write to SD card still running
May 22 12:38:02 raspberrypi vmunix: [295822.771242] mmc0: Timeout waiting for hardware interrupt - cmd12.
May 22 12:38:02 raspberrypi vmunix: [295822.772422] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900

@hvenzke
hvenzke commented May 23, 2013

@licaon-kter, with that 0x900 msgs indicates an unfinished & corrupped ext4 jornal data.
the final jornal write blocked due to sdcard timeout , thus the ext4 goes in bad mode.with fs flag recover_required

save asap somewhere your data ...!!

@popcornmix
Contributor

Anyone's problem's fixed with "sdhci-bcm2708.sync_after_dma=0" added to command line?

@hvenzke
hvenzke commented May 23, 2013

rasberian default changed the cmdline.txt at my own base Image

from

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

to

EDIT1 ( thanks to @licaon-kter )

 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200  elevator=noop sdhci-bcm2708.missing_status=0 sdhci-bcm2708.sync_after_dma=0 profile=2 loglevel=7  console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 rootflags=commit=120,data=writeback rootwait ro

will report soon what happen wiile stress the SD card with compile 3.9.y kernel

@licaon-kter

@remsnet: you can clean up cmdline.txt a bit since they're AFAIK default anyway:

dwc_otg.microframe_schedule=1
sdhci-bcm2708.enable_llm=1
dwc_otg.fiq_fix_enable=1
dwc_otg.lpm_enable=0

and listed twice each:

sdhci-bcm2708.missing_status=0
sdhci-bcm2708.sync_after_dma=0

@Ferroin
Ferroin commented May 23, 2013

I'm kind of curious, has anyone had this problem with the rpi-3.9.y branch?

@hvenzke
hvenzke commented May 23, 2013

@Ferroin since the bmc2708 sdhc code not changed mutch , the ERRORS shuold be the same.
and thats why i thinking the reported been an serious issue for ALL woldwide deployed RPI

@licaon-kter

my logs are from 3.9.y

@Ferroin
Ferroin commented May 23, 2013

@remsnet
This isn't just caused by the SDHCI code, but by the interaction of the SDHCI driver with the other drivers, thus changes outside of the driver might affect it as well. Also, as for the ALL RPI part, that is patently false as not all RPi's are running Linux (Although almost every other OS for it has it's own unique problems...).

@hvenzke
hvenzke commented May 24, 2013

@Ferroin ,
ok thanks to make that more clear to me as the BMC2708 design PIC i seen mentioned it all via USB somehow.
And well its mabee only 98% running linux the rest other , witch is still +1million RPI run linux...
Sorry for my ignorance somehow . But the reported looks like an serios issue .

I want run my PIs for years without any interactions ,
same i did with my old i386/40 till the cpu died 2003 after run 7/24 since 1991..
Stabilty are here my focus and absolut required

While the stress tests - Fresh installed 2013-02 rasberian running plain 3.6.11 rasberian kernel - with
stressing past 14h at RPI model B with 64GB class 10 shxc card with kernel 3.9.y compile and samba4.06 compile at same time ..
with the folowing set of RPI RUN options at fstab , config.txt , cmdline.txt

  • not used the raspi-config for resize.
  • Modifys the fstab, config.txt , cmdline.txt was done BEFOR the first reboot ( login as pi , sudo bash ... vi .. )
  • fsck was forced befor modify the partions with fdisk ,
    then delete primary 2 , add 2 with same start block but only 4G ,
    then fdisk write , reboot , resize , add 3 & 4 reboot
~ $ cat /etc/fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime,nodiratime  0       1
/dev/mmcblk0p3  swap           swap    defaults          0       0
/dev/mapper/System0-LOG_lv  /LOG               ext4    defaults,noatime,nodiratime  0       1
/dev/mapper/System0-usr_src_lv  /usr/src               ext4    defaults,noatime,nodiratime  0       1
/dev/mapper/System0-backup_lv  /backup               ext4    defaults,noatime,nodiratime  0       1
~ $ cat /boot/cmdline.txt
console=ttyAMA0,115200 kgdboc=ttyAMA0,115200  elevator=noop sdhci-bcm2708.missing_status=0 sdhci-bcm2708.sync_after_dma=0 profile=2 loglevel=7  console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 rootflags=commit=120,data=writeback rootwait ro
~ $ cat /boot/config.txt
# uncomment this if your display has a black border of unused pixels visible
# and your display can output without overscan
disable_overscan=1
#uncomment to overclock the arm. 700 MHz is the default.
arm_freq=650
# for more options see http://elinux.org/RPi_config.txt
core_freq=250
sdram_freq=400
over_voltage=0
gpu_mem=32
 ~ $ cat /etc/sysctl.conf
# rpi tweaks
vm.swappiness=1
vm.min_free_kbytes = 8192
# Increase defaults for IPC (bnc#146656)
kernel.msgmax = 65536
kernel.msgmnb = 65536
kernel.sem = 250 256000 32 1024
# By default this script does nothing.
# Increase TCP Buffers to 16 MB
net.core.rmem_default=16777216
net.core.wmem_default=16777216
net.core.rmem_max=16777216
net.core.wmem_max=16777216
- my SD cards seems significant faster , boots in less then 12 seconds - befor that took up to 90sek - no corruption ( YET ) . - no more false IO optimszing , due the chosen settings . in fact i not get any errors seen bevor(YET ).

will report later with 3.8.13 and 3.9.y kernels this weekend

@hvenzke
hvenzke commented May 24, 2013

@popcornmix due samba4 requirements , see https://wiki.samba.org/index.php/Samba_4/OS_Requirements ,

This kconfig´s are nessary to be Included at default rasberian kconfig please.

  CONFIG_EXT3_FS_XATTR=y
  CONFIG_EXT3_FS_SECURITY=y
  CONFIG_EXT3_FS_POSIX_ACL=y
  CONFIG_EXT4_FS_XATTR=y
  CONFIG_EXT4_FS_SECURITY=y
  CONFIG_EXT4_FS_POSIX_ACL=y
@popcornmix
Contributor

We use

CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT23=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y

CONFIG_EXT4_FS_XATTR does not exist. Have you tried samba4?

@hvenzke
hvenzke commented May 25, 2013

Updated today from samba 4.0.5 to 4.0.7 .

~ :/usr/src/GIT/samba4/samba-v4-0-test# ./bin/smbd --version
Version 4.0.7-DEVELOPERBUILD

I RUN samba 4 AD enabled pdc/bdc with RPI model B here .
This been why i.e need an stable kernel with no issues on stablity.

The rasberian RPI samba4.0.0 betta2 package been fully outdated .
latest samba4 contains and Buildin sldapd and dns .

@Ferroin
Ferroin commented May 26, 2013

CONFIG_EXT4_FS_XATTR is always true in recent kernels, and is also certain to be true if CONFIG_EXT4_FS_POSIX_ACL is true, because xattrs are how ext4 stores acls.

@P33M
Contributor
P33M commented May 26, 2013

I can reproduce this on fiq_split and 3.8.

May 25 16:27:02 raspberrypi kernel: [  248.220767] mmc0: final write to SD card still running
May 25 16:27:12 raspberrypi kernel: [  258.231703] mmc0: Timeout waiting for hardware interrupt - cmd12.
May 25 16:27:12 raspberrypi kernel: [  258.232864] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
May 25 16:27:47 raspberrypi kernel: [  293.491357] mmc0: final write to SD card still running
May 25 16:27:57 raspberrypi kernel: [  303.491706] mmc0: Timeout waiting for hardware interrupt - cmd12.
May 25 16:27:57 raspberrypi kernel: [  303.492884] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
May 25 16:30:50 raspberrypi kernel: [  476.844081] mmc0: final write to SD card still running
May 25 16:31:00 raspberrypi kernel: [  486.791691] mmc0: Timeout waiting for hardware interrupt - cmd12.
May 25 16:31:00 raspberrypi kernel: [  486.792885] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
May 25 16:31:01 raspberrypi kernel: [  487.738701] mmc0: final write to SD card still running
May 25 16:31:11 raspberrypi kernel: [  497.691691] mmc0: Timeout waiting for hardware interrupt - cmd12.
May 25 16:31:11 raspberrypi kernel: [  497.692855] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
May 25 16:31:12 raspberrypi kernel: [  498.622312] mmc0: final write to SD card still running
May 25 16:31:22 raspberrypi kernel: [  508.571694] mmc0: Timeout waiting for hardware interrupt - cmd12.
May 25 16:31:22 raspberrypi kernel: [  508.572879] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
May 25 16:31:23 raspberrypi kernel: [  509.301761] mmc0: final write to SD card still running
May 25 16:31:33 raspberrypi kernel: [  519.251692] mmc0: Timeout waiting for hardware interrupt - cmd12.
May 25 16:31:33 raspberrypi kernel: [  519.252860] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
May 25 16:32:12 raspberrypi kernel: [  558.843552] mmc0: final write to SD card still running
May 25 16:32:22 raspberrypi kernel: [  568.811691] mmc0: Timeout waiting for hardware interrupt - cmd12.

This is with a class 4 Sandisk 4GB SDHC card which has never given me trouble (at least until now). Heavy read/write of the card at the time - was compiling wayland. It appears to have caused data corruption but not filesystem corruption - several compile errors were for bogus characters in generated files which I subsequently found to be truncated with garbage at the end.

With sdhci_bcm2708.sync_after_dma=0, there's no such timeouts occurring and additionally the compile has produced a valid result (i.e. working weston).

@oerik
oerik commented May 26, 2013

I can confirm that with Linux 'raspberrypi 3.8.13+ #454 PREEMPT Fri May 17 15:58:19 BST 2013 armv6l GNU/Linux' and sdhci_bcm2708.sync_after_dma=0 the timeouts are gone! Everything seems ok after boot, I leave the system running for a night and report the status tomorrow.

@popcornmix
Contributor

Interesting. sdhci_bcm2708.sync_after_dma=0 is desirable as it avoids a non-standard sleepy/busy waiting in the driver. However if you search around for that, I think you'll find a number of people reporting corruption with sdhci_bcm2708.sync_after_dma=0.

I'm not too familar with the code, but there is a slightly suspicious commented out spinlock nearby.

I wonder if a workaround is required, but this one is buggy.

@hvenzke
hvenzke commented May 26, 2013

Feeedback on with rpi-update 3.8.13 and compile the 3.9.3 kernel ( 3.9.y ) : No timeouts seen, no corruption.
at moment im install the 3.9.3 and will try hurt it ...

So the issue - forgive me - been the sdhci_bcm2708 timings & IO truning behavior.

@popcornmix find an Linux freak that knows the SD Card standards ... and fix it.

Anybody here that can help him with review the code ?! .. ( my own coding skills not fit ) .

@Ferroin
Ferroin commented May 27, 2013

The claims of corruption caused by sdhic-bcm2708.sync-after-dma=0 seem interesting to me, I haven't had such problems except when the system crashes (and corruption is expected then anyway). I'm planning on switching my rpi's to use nilfs2 instead of ext4 for the root fs on the SD card, it will be interesting to see if the different I/O load created by nilfs2 affects anything.

@hvenzke
hvenzke commented May 27, 2013

nilfs2 don´t matter mutch.
While passt weeks ive tested xfs , ext2/3/4 well with jffs(2) and its NOT significat faster.
The while tested , only XFS beats ext4.
Cant ´t yet test ZFS as root fs , due the nativ port not compile on RPI ( yet ) .

@licaon-kter

What about F2FS ?

@hvenzke
hvenzke commented May 27, 2013

hav´nt tested F2FS . if you have time enoth test it yourself.
The most important performance settings are archived as follows:

mount options : noatime,nodiratime
kernel options : elevator=noop
stable sdcard class 6/10 disks : sdhci-bcm2708.missing_status=0 sdhci-bcm2708.sync_after_dma=0

just as I allready said : someone shuold assist popcornmix with the sdcard code .
The current not 100% fit for class 6/10 with size +16G without kernel the sdhci-bcm2708 options .

@Ferroin
Ferroin commented May 27, 2013

@remsnet
I don't mean in terms of efficiency, I mean in terms of whether it is more corruption resistant.

@oerik
oerik commented May 27, 2013

My problem system has been up for 21 hours hours without any problems, my problem is fixed. However, I'm curious what can be learned from this.

@hvenzke
hvenzke commented May 27, 2013

@Ferroin , ah got you .. In terms of FS corruption resistant .. XFS beats all .
Untill the "HARDWARE" realated timeout things wee seen with i.e 0x900 msgs.

Why XFS ?
When an today ext2/3/4/jffs/other goes corrupt , the only way to restore the data been via lost+found with fsck -y -f
With XFS you have the XFS tools , and with that you REBUILD the Jornal without data lost
( untill hardware ok , dead blocks marked as sutch , xfs ask you what to do with if one dead found.)

@oerk , well - looks like we learned not trust 100% the given code , and try find an workarround.
if that way not work , an spec specialist are required...
IO must be stable , thats the path it MUST go , without guessing that it does.

There may be possibly BUGS arround with SPINLOCK inside the sdhci-bcm2708
and the IO Optimiszing for class 6/10 cards.

@Ferroin
Ferroin commented May 27, 2013

@remsnet
While that may be true recovery wise, the log structure in nilfs combined with the way recovery happens on mounting the fs means that it is almost as good as a transactional fs regarding corruption (ie, there is only a very small window in which a crash can occur that will cause a file operation to be only partially complete; most of the time, file operations are almost atomic.). I care less about recovering data than I do about minimizing downtime after a crash.

@popcornmix
Contributor

I don't think we can switch to sdhci-bcm2708.sync_after_dma=0 due to:
http://www.raspberrypi.org/phpBB3/viewtopic.php?f=28&t=12097&start=425#p161101

where it caused:

[    7.328991] mmcblk0: r/w command failed, status = 0x80000900
[    7.354533] end_request: I/O error, dev mmcblk0, sector 15564536
[    7.368850] Buffer I/O error on device mmcblk0, logical block 1945567
[   10.846507] mmcblk0: r/w command failed, status = 0x80000900
[   10.870303] end_request: I/O error, dev mmcblk0, sector 122864
[   10.884504] Buffer I/O error on device mmcblk0p1, logical block 14334
@hvenzke
hvenzke commented May 27, 2013

@Ferroin snapshootig don´t help against a fully lost fdisk tab.
What i mean ?

While stressting past Weeks , the whole sdcard Partion Table(fdisk tab ) where completely GONE.
RPI has regulary NO second root disk, so the journal/ZIL gone , the FS goes either with nilfs/zfs corrupt.

You suffer on all FS types when Partion table been lost untill you save the journal(s) / ZIL
on seperat disk and mirror the root disk.

You suffer as well when random sector writes due false IO optimiszing happen to the boot sector parts of an
partion.

-> you still need to keep an the partion table details in an seperat secured place
->> i.e backup dvd-rw
->> i.e seperat OCS server that gather the system inverntory

When fstab table gone you can reenter them , but some don´t allow to partion access fully restored that way.
As i don´t have tested nil2fs with that , i can not know if it will work again with an former deleted/vanished fdisk tab

Well nilfs maybe more transactional like zfs ..
Im an Solaris/ZFS freak - i know how destaster recover the ZIL ... ( wrote my own Wiki for that)
and thats hard when code goes wrong like the sdhci-bcm2708 does .

FYI : XFS has changed at kernel 3.3.0, its an transactional fs since then.
And i know that it works with deleted/vanished fstab
XFS exist since +20years and its OpenSource

@hvenzke
hvenzke commented May 27, 2013

@popcornmix he been using an outdated kernel with USB/audio.

The most users i spoken past 30 weeks had allmost weak powersupplies with allmost less then 700mA with also unstable / unfitting voltages... .. while with external devices on RPI connected...

min. ~ 700 mA Model B standlone with +16G card while BOOT( my 64G SD use 970 mA peak at boot )
plus 150mA with keyboard and mouse
plus 250-600 mA for USB Audio depends on what device..

So, befor say thats not possible the POWER stablity must be checked.
Most Users require an powered HUB for theire external extra devices, but use an unpowerd and suffer then.
See the msg from i.e Jim JKla in that thread.

Take an Rigth funcional powersource to an PI been the Most important to RUN the PI in _stable_ Conditions But that End users work - not at task for RPI linux kernel .

First of my own used powersources was the cheapest .. what happen ?
they blow UP ... EXPLODE within 3 weeks,
pi was malfunction during boot, usb keyb i.e not worked.

@ewheelerinc

Hello all,

I'm having the same mmcblk0 with errors like these and stumbled across thread:
mmc0: Timeout waiting for hardware interrupt - cmd25.
mmc0: Timeout waiting for hardware interrupt - cmd12.
mmc0: Timeout waiting for hardware interrupt - cmd13.
mmcblk0: error -110 sending status command, retrying
mmc0: Timeout waiting for hardware interrupt - cmd13.
mmcblk0: error -110 sending status command, retrying
mmcblk0: error -110 sending status command, retrying
mmc0: Timeout waiting for hardware interrupt - cmd13.
mmcblk0: error -110 sending status command, retrying
mmc0: Timeout waiting for hardware interrupt - cmd13.
mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900

I can reliably reproduce the error in either of 3.6.y and 3.8.y by copying a staged root filesystem from ext4 onto a ZFS filesystem. The RPi uses stock clocking and I have tried multiple power supplies (none of which I have ever had trouble with before).

Reading through the notes above, I tried the rpi-3.6.y branch, copied my kernel .config, rebuilt, rebuilt ZFS, and I am still having trouble. It takes much longer before I start getting the mmc errors.

Incidentally, I tried the sdhci-bcm2708.sync_after_dma=0 fix in 3.8.y, but then then the kernel wouldn't even mount the root filesystem, it would go strait to the mmc0...cmdXX errors---so that is certainly not an option for me.

I'm using a 16GB PNY class-10 (which boasts 20mb/sec on the label and achieves 19MB/s in 4MB write blocks.) The card is labeled SD16G10133MMC2HB in barely readable print on the back. I have two of these cards, both give the same problem (both tested in 3.8.13+, only one was test in 3.6.y).

While the specific configuration may not be relevant since others are having problems with different configurations, here we go:
ZFS is configured in ZRAID-1 on 4 SD-card partitions so we can reliably scrub bitflips on cheap media.
mmcblk0: p1 p2 < p5 p6 p7 p8 > p4
^p5-8: ZFS ^ ^^ p4: ext4
root@rpi:/sys# fdisk -lu /dev/mmcblk0

Disk /dev/mmcblk0: 16.0 GB, 16012804096 bytes
4 heads, 16 sectors/track, 488672 cylinders, total 31275008 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006a906

    Device Boot      Start         End      Blocks   Id  System

/dev/mmcblk0p1 16 129040 64512+ c W95 FAT32 (LBA)
/dev/mmcblk0p2 129041 3906248 1888604 5 Extended
/dev/mmcblk0p4 3906249 31275007 13684379+ 83 Linux
/dev/mmcblk0p5 131072 1064959 466944 fd Linux raid autodetect
/dev/mmcblk0p6 1073152 2007039 466944 fd Linux raid autodetect
/dev/mmcblk0p7 2015232 2949119 466944 fd Linux raid autodetect
/dev/mmcblk0p8 2957312 3891199 466944 fd Linux raid autodetect

@hvenzke
hvenzke commented May 27, 2013

@ewheelerinc
You use native zfs right ? fuse-zfs are not realy ready for rootfs up to my knowlege.
if native , what compile options please.
I don´t get yet the native compile with 3.8.13 / 3.9.3 on rpi yet. filed an BUG report some days ago..

and what powersuply you use ..?

@ewheelerinc

@remsnet : I'm cross-compiling zfs-0.6.1 (native) for whatever kernel I build (3.6, 3.8, 3.9 is broken). It works great, just hangs under load with the mmc errors which I believe to be related more to whatever causes bug#280 than to ZFS itself. I suppose I could format a USB stick with ZFS to confirm that it is not a ZFS issue.
((Building ZFS on RPi is offtopic to this thread, so point me at your bug thread if you want more detail. I plan to blog an article on the subject at some point as well.))

I've used both my PC's USB port and a wall plug rated at 1A. No apparent difference.

Does anyone else have an idea what is causing the mmc0 error's we are getting?

What's the next step in troubleshooting?

I have a reliable way to test, takes about 20 minutes for a result. Send me patches or suggestions and I'll be glad to test them.

-Eric

@ewheelerinc

Also of note, I'm using the "next" firmware branch, no change.

@blueridge2

All

For the Rasperry Pi, for the root file system I use ext3fs, because it
rock solid stable. We should use ext2fs in my humble opinion or a find
a flash card that has micro processor on it that guanenties writes occur
or do not occur.

I do all of my development on an nfs mount file system, using cross
compilers, which is really neat.

Chip

On 5/27/2013 6:55 PM, Remsnet LTD, wrote:

@ewheelerinc https://github.com/ewheelerinc

You use native zfs right ? fuse-zfs are not realy ready for rootfs up to
my knowlege.
if native , what compile options please.
I don´t get yet the native compile with 3.8.13 / 3.9.3 on rpi yet. filed
an BUG report some days ago..


Reply to this email directly or view it on GitHub
#280 (comment).

@ewheelerinc

@blueridge2 : ext3/4 will burn out the journal blocks over time. Unless you go with an industrial-class SD card with really great wear-leveling the NAND's lifetime will be shortened by using ext3/4. F2FS is the most promising for block-based NAND cards like SD cards, but it doesn't include a block-by-block checksum to detect bit-flips and the like. ZFS can actually scrub bad blocks if its configured right.

@ewheelerinc

I believe I've found the fix!

set sdhci-bcm2708.enable_llm=0 on your kernel command line.

It looks like the bug was introduced here:

#72

For the first time ever, after trying various different things for over a week, I have successfully copied a root filesystem onto my ZFS tree. Woot!

Looking at drivers/mmc/host/sdhci.c it appears that enable_llm is only in force if CONFIG_PREEMPT is enabled. Keeping CONFIG_PREEMPT enabled and setting enable_llm=0 fixed it for me. I'm going to try again using the 3.8.13+ kernel (this test worked on the 3.6.11+ kernel) and see if it still works.

-Eric

@hvenzke
hvenzke commented May 29, 2013

@ewheelerinc great work.

Looks like the sdhci-bcm2708 are NOT realy LLM capable, right ?

Espescaly NOT when use SD cards +8G with class 6 / 10

@popcornmix / @ewheelerinc is it possible to add an USER waring into the sdhci-bcm2708 for that ?
We shuold let the PI users know abount possibly way of solution(s).

add an reverence document ... i.e under the rpi-3-8.y/Documentation/arm/bmc2708/sdhci-bcm2708.txt
I wuold voluteer sutch if interests ..

@ewheelerinc

On Tue, 2013-05-28 at 21:07 -0700, Remsnet LTD, wrote:

@ewheelerinc great work.

Looks like the sdhci-bcm2708 are NOT realy LLM capable, right ?

It could be a deadlock or race condition (less likely I think on one
CPU). Someone who knows that code really well could scrutinize it and
possibly provide further troubleshooting. There may be a way to have
low latency and safe locking.

Who maintains that part of the code?

@ewheelerinc

oops.

and now, with formatting:

On Tue, 2013-05-28 at 21:07 -0700, Remsnet LTD, wrote:

@ewheelerinc great work. Looks like the sdhci-bcm2708 are NOT realy LLM capable, right ?

It could be a deadlock or race condition (less likely I think on one CPU). Someone who knows that code really well could scrutinize it and possibly provide further troubleshooting. There may be a way to have low latency and safe locking. Who maintains that part of the code?

@ChristophHaag

sdhci-bcm2708.enable_llm=0 doesn't do it for me.

[    0.000000] Kernel command line: dma.dmachans=0x7f35 bcm2708_fb.fbwidth=1824 bcm2708_fb.fbheight=984 bcm2708.boardrev=0xf bcm2708.serial=0x5a095a2a smsc95xx.macaddr=B8:27:EB:09:5A:2A sdhci-bcm2708.emmc_clock_freq=100000000 vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000  coherent_pool=6M smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=noop rootwait sdhci-bcm2708.enable_llm=0

But

[ 1907.393440] mmc0: final write to SD card still running
[ 1917.349567] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 1917.369654] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[ 9536.513380] mmc0: final write to SD card still running
[ 9546.479239] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 9546.497995] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[    2.338557] mmc0: new high speed SDHC card at address e624
[    2.339250] mmcblk0: mmc0:e624 SU16G 14.8 GiB

During phoronix-test-suite benchmark 1302242-BY-1205272AR49 running 3.9.4-1-ARCH.

But I think it got much more rare. But I haven't really measured.

@popcornmix
Contributor

There's some discussion of enable_llm here:
#72
Basically without that patch the USB gets more unreliable (as sdcard disables interrupts for multiple USB frames).
Possibly it's it less critical with the fiq_split patch.

@hvenzke
hvenzke commented May 29, 2013

@ewheelerinc

mnter of sdhci-bcm2708 are the github members here almost
"Master of desaster" aka of this /raspberrypi/linux github is popcornmix .

I started to reported my issues a while ago on the raspberry.org forum , found that the SOURCE used was this .
Thus i started here my contributions.
Due this BugLOG , we found for some items an workarround.

@ChristophHaag: smsc95xx.turbo_mode=N may not 100% work with LLM=1 thats my result past weeks.

@macerg
macerg commented May 29, 2013

Hello, newbie here, can someone guide me through applying this patch please?
I am on Kernel 3.8.13+ #454 PREEMPT , Getting:-
" mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900"
On a 32 gig Sandisc micro SD. Had repeated f/s corruptions.

@hvenzke
hvenzke commented May 29, 2013

@macerg please post your /boot/cmdline.txt , /boot/config.txt , /etc/sysctl.conf, /etc/fstab HERE.

@macerg
macerg commented May 29, 2013

/boot/cmdline.txt :

-dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

/boot/config.txt:-

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment this if your display has a black border of unused pixels visible
# and your display can output without overscan
disable_overscan=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
arm_freq=900

# for more options see http://elinux.org/RPi_config.txt
gpu_mem=64
core freq=250
sdram_freq=450
over_voltage=2
start_file=start_x.elf
fixup_file=fixup_x.dat

/etc/sysctl.conf:-

#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additonal system variables
# See sysctl.conf (5) for information.
#

#kernel.domainname = example.com

# Uncomment the following to stop low-level messages on console
kernel.printk = 3 4 1 3

##############################################################3
# Functions previously found in netbase
#

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
#net.ipv4.conf.default.rp_filter=1
#net.ipv4.conf.all.rp_filter=1

# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/
# Note: This may impact IPv6 TCP sessions too
#net.ipv4.tcp_syncookies=1

# Uncomment the next line to enable packet forwarding for IPv4
#net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
#  Enabling this option disables Stateless Address Autoconfiguration
#  based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1


###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Do not accept ICMP redirects (prevent MITM attacks)
#net.ipv4.conf.all.accept_redirects = 0
#net.ipv6.conf.all.accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
#net.ipv4.conf.all.log_martians = 1
#

# rpi tweaks
vm.swappiness=1
vm.min_free_kbytes = 8192

/etc/fstab:-

proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
/dev/mmcblk0p3  /home/richard   ext4    defaults,noatime  0       2
/dev/sda1  /media/samsung_ext4  ext4    defaults,nofail,user   0       2

# a swapfile is not a swap partition, so no using swapon|off from here on, use  dphys-swapfile swap[on|off]  for that

Thanks....................

@hvenzke
hvenzke commented May 29, 2013

@macerg some wrong settings for class10 32GB card !

Please kindly re-read my #280 (comment)

cmdline.txt :
elevator=noop sdhci-bcm2708.missing_status=0 sdhci-bcm2708.sync_after_dma=0

and at /etc/fstab :
defaults,noatime,nodiratime

SD cards are NON-SPINNING storange.

You need to force an fsck manualy due the 0x900 errors ( ext4 jounal not written due io timeouts ).

@macerg
macerg commented May 29, 2013

I have backed ou my overclocking, still getting:----

96k SS
[ 148.894982] mmc0: final write to SD card still running
[ 158.914625] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 158.915785] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[ 159.443130] mmc0: final write to SD card still running
[ 169.455162] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 169.456393] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900

@hvenzke
hvenzke commented May 29, 2013

@macerg your filesystem been corrupted due Overclocked.
There been an warning written at i.e elinux.org that cards with +16G class 6 / 10 forbidd ALLWAYS overclocking

We had that discussed allready here weeks ago.

  • halt your PI,
  • unplug the sdcard at PI , Plug it at your sdcard reader that been connected to your PC
    -install virtualbox 3.x on you pc
  • install an debian 7 weezy with 3.8.x kernel inside with i.e 512Mb ram
  • allow USB devices at virtualbox
  • dmesg | grep sdb
  • fsck -y /dev/sda1

if the ext4 died after the fsck, find the lost data at lost+found ( time consuming ..)

after copyied relevant data ( backup i.e ): reinstall your sdcard.

@macerg
macerg commented May 29, 2013

BUT works fine under 3.6, no error messages whatever. F/S is fine with fsck (on Ubuntu, so don't need to
run with virtualbox)

ONLY get the error messages when running Kernel 3.8....

35.965463] Adding 102396k swap on /var/swap. Priority:-1 extents:1 across:102396k SS
[ 148.894982] mmc0: final write to SD card still running
[ 158.914625] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 158.915785] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[ 159.443130] mmc0: final write to SD card still running
[ 169.455162] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 169.456393] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[ 347.229063] mmc0: final write to SD card still running
[ 357.243808] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 357.245055] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[ 675.202938] mmc0: final write to SD card still running
[ 685.218947] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 685.220178] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900

Am NOT NOW OVERCLOCKING. FSCK SHOWS NO PROBLEMS. BUT STILL GETTING ERROR
MESSAGES UNDER 3.8.13

System is fine with 3.6, can I make it any clearer?

@hvenzke
hvenzke commented May 29, 2013

@macerg, calm down ... re-read the current open RPI linux issues .

wait a new days more and your mood melt the ext4 finaly due not listing :-)

@MilhouseVH

My 512MB Pi with 3.6.11+ #452 kernel and ext4 has no corruption problems - not even a single mmc0 error/warning - with a 32GB SanDisk Ultra/Class 10 SD card, 1GHz ARM, 500MHz Core, 600MHz SDRAM so anyone claiming that overclocking will always cause a corruption problem with 16GB+ cards is clearly spreading FUD. Sure, overclocking may increase the chance of corruption but overclocking is certainly not guaranteed to cause it, nor is the size of the SD card an issue as far as I can tell.

Haven't tried any of these more recent kernels though, as this 32GB system is not available for testing.

@macerg
macerg commented May 29, 2013

@remsnet Will take #280 on board, thanks.

@macerg
macerg commented May 29, 2013

@remsnet I have implemented your cmdline and fstab changes. Thank you.
The good new is that the error messages seemed to have stopped.
I will implement your other changes as listed in #280.
I am still on 3.8.13+ #454 (non-overclocked)
F/S appears to be ok.
Thank you, no idea why the problem became apparent since 3.8.
Thanks again & apologies for my impatience .

@hvenzke
hvenzke commented May 29, 2013

@macerg fine to hear that i was rigth...

someone can call me stupid and ignorant - i don´t care mutch
I found an way to solve my own issues , and contributed & published the results here .
WITH TRY and ERROR ( worst way )

Some missunderstandings included as well....
.. as i did´t known mutch about the bmc2708 chipset befor started debugging..

problem seems as popcornmix and other said : issues with spinlock sheduling & llm & usb & RT issues
This thread here describe it muchs that there must be an code bug and / or an hardware bug into sd card controler.

While the past 30 weeks i BURNED / broken multible 32/64 sd cards from different vendors so they are end up in my waste bin.

Did many selfcompiles ( > 100x ) ...
since its one of the best ways to stress an system, if got near stable ,
compile multible BIG packages as X11 , kernel , samba-dev at the same time and see what happen ...

I am an linux adm since kernel release 0.4.2 .. my arpa ip is 44.130.176.130 since 1989
Cheers.

@ewheelerinc

@macerg: add this to your cmdline:
sdhci-bcm2708.enable_llm=0
and see if it fixes the problem. Thats what finally fixed my MMC issues.

@macerg
macerg commented May 29, 2013

@remsnet .........

And I am a mainframe sysprog since IBM CICS 1.4 (1984) -:)
By the way, I didn't mean to insult you, just a little impatient with everything that was going on.
Apologies again if it came over that way.

Now getting:-

mmc0: Controller never released inhibit bit(s).
[ 427.326972] mmcblk0: unknown error -5 sending read/write command, card status 0x900
[ 427.327013] end_request: I/O error, dev mmcblk0, sector 4321392
[ 427.327038] Buffer I/O error on device mmcblk0p2, logical block 524814
[ 427.327051] lost page write due to I/O error on mmcblk0p2
[ 427.327086] end_request: I/O error, dev mmcblk0, sector 4321400
[ 427.327102] Buffer I/O error on device mmcblk0p2, logical block 524815
[ 427.327113] lost page write due to I/O error on mmcblk0p2

Getting i/o errors on mmcblk0, looks like the card is stuffed to me?

@hvenzke
hvenzke commented May 29, 2013

i know CICS as well.. as i an so called enterprise level Unix admin ..working for Big Irons

@macerg
macerg commented May 29, 2013

@ewheelerinc Thanks, I have done as you suggested.....

@hvenzke
hvenzke commented May 29, 2013

@macerg the way i got rid the 0x900 was as descibed .. backup data after force fsck , then rebuild the PI

and DO the changes BEVOR the FIRST REBOOT.

the ext4 jounal get broken ( flag -> need_recover ) without , near 100% when you overclock.

sutch a behavior been uncomon on Storange devices , never seen again since the 90´s
isa card controler at the 90´s , had some equal issues like we have here..

@macerg
macerg commented May 29, 2013

Well, the 0x900 have indeed gone, but I am now getting:-

424.036075] mmc0: Controller never released inhibit bit(s).
[ 424.098284] mmcblk0: unknown error -5 sending read/write command, card status 0x900
[ 424.098325] end_request: I/O error, dev mmcblk0, sector 4321384
[ 424.098349] Buffer I/O error on device mmcblk0p2, logical block 524813
[ 424.098362] lost page write due to I/O error on mmcblk0p2
[ 424.098397] end_request: I/O error, dev mmcblk0, sector 4321392
[ 424.098413] Buffer I/O error on device mmcblk0p2, logical block 524814
[ 424.098423] lost page write due to I/O error on mmcblk0p2
[ 424.098442] end_request: I/O error, dev mmcblk0, sector 4321400
[ 424.098456] Buffer I/O error on device mmcblk0p2, logical block 524815
[ 424.098467] lost page write due to I/O error on mmcblk0p2

Do you think the SD card is busted?

@hvenzke
hvenzke commented May 29, 2013

not yet .
push an new image on it after you deleted the fdisk tab and erase card once simple way
delete parts , create new part , mkfs.xfs , mkfs.ext2 i.e then write new image with DD

if you get an IO error with dd , the card end up in waste bin.

@hvenzke
hvenzke commented May 29, 2013

Someone here with c/c++ skills interested in assit me to get ZFS native compiled to work against the 3.8.y series ?

thats my current major project to get the 0.6.1 included the 3.8.y and up -
found an patch for 3.9.y series as well (ubuntu did that at march 2013)

@macerg
macerg commented May 29, 2013

Does this mean anything to anyone here?
Seem to have a trace dumped to the log:-

[Wed May 29 21:55:44 2013] Adding 102396k swap on /var/swap.  Priority:-1 extents:1 across:102396k SS
[Wed May 29 22:02:01 2013] mmc0: Controller never released inhibit bit(s).
[Wed May 29 22:02:01 2013] mmcblk0: unknown error -5 sending read/write command, card status 0x900
[Wed May 29 22:02:01 2013] end_request: I/O error, dev mmcblk0, sector 4321392
[Wed May 29 22:02:01 2013] Buffer I/O error on device mmcblk0p2, logical block 524814
[Wed May 29 22:02:01 2013] lost page write due to I/O error on mmcblk0p2
[Wed May 29 22:02:01 2013] end_request: I/O error, dev mmcblk0, sector 4321400
[Wed May 29 22:02:01 2013] Buffer I/O error on device mmcblk0p2, logical block 524815
[Wed May 29 22:02:01 2013] lost page write due to I/O error on mmcblk0p2
[Wed May 29 22:02:01 2013] mmc0: Controller never released inhibit bit(s).
[Wed May 29 22:02:01 2013] mmcblk0: unknown error -5 sending read/write command, card status 0x900
[Wed May 29 22:02:01 2013] end_request: I/O error, dev mmcblk0, sector 122880
[Wed May 29 22:02:01 2013] Buffer I/O error on device mmcblk0p2, logical block 0
[Wed May 29 22:02:01 2013] lost page write due to I/O error on mmcblk0p2
[Wed May 29 22:03:36 2013] ------------[ cut here ]------------
[Wed May 29 22:03:36 2013] WARNING: at fs/buffer.c:1114 mark_buffer_dirty+0xb8/0xdc()
[Wed May 29 22:03:36 2013] Modules linked in: snd_bcm2835 snd_pcm snd_page_alloc snd_seq snd_seq_device snd_timer snd 8192cu leds_gpio led_class joydev evdev
[Wed May 29 22:03:36 2013] [<c00139e4>] (unwind_backtrace+0x0/0xf0) from [<c001e8c4>] (warn_slowpath_common+0x4c/0x64)
[Wed May 29 22:03:36 2013] [<c001e8c4>] (warn_slowpath_common+0x4c/0x64) from [<c001e8f8>] (warn_slowpath_null+0x1c/0x24)
[Wed May 29 22:03:36 2013] [<c001e8f8>] (warn_slowpath_null+0x1c/0x24) from [<c00f2dd0>] (mark_buffer_dirty+0xb8/0xdc)
[Wed May 29 22:03:36 2013] [<c00f2dd0>] (mark_buffer_dirty+0xb8/0xdc) from [<c016fdfc>] (__jbd2_journal_unfile_buffer+0xc/0x20)
[Wed May 29 22:03:36 2013] [<c016fdfc>] (__jbd2_journal_unfile_buffer+0xc/0x20) from [<c01735f8>] (jbd2_journal_commit_transaction+0xd8c/0x174c)
[Wed May 29 22:03:36 2013] [<c01735f8>] (jbd2_journal_commit_transaction+0xd8c/0x174c) from [<c017703c>] (kjournald2+0xc0/0x29c)
[Wed May 29 22:03:36 2013] [<c017703c>] (kjournald2+0xc0/0x29c) from [<c003aab8>] (kthread+0xa4/0xb0)
[Wed May 29 22:03:36 2013] [<c003aab8>] (kthread+0xa4/0xb0) from [<c000db18>] (ret_from_fork+0x14/0x3c)
[Wed May 29 22:03:36 2013] ---[ end trace c2e8872b3fa310e4 ]---
@ewheelerinc

looks like a bad disk or incompatible card.

@hvenzke
hvenzke commented Jun 3, 2013

Well Folks ,

Last days i got some "games " with few 16G class 4 cards from transceed and kingston.

the cmdline.txt differs from class 6 and 10 as follows

 cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 sdhci-bcm2708.sync_after_dma=0 profile=2 loglevel=7 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=noop rootflags=commit=120,data=writeback rootwait ro

test results was on that cards

  • sdhci-bcm2708.enable_llm=0 caused 0x900
  • sdhci-bcm2708.missing_status=0 if set then cause 0x900
  • sdhci-bcm2708.sync_after_dma=0 was required
@Neverous
Neverous commented Jun 4, 2013

I don't really know if it has any relevance to the topic, mostly because i dont really follow that closely whats happening in the raspberrypi kernel, but how much have changed between 3.2.27 and 3.6.11?
I think that error was introduced somewhere in that change.
For about half a year now i was using 3.2.27(from https://github.com/raspberrypi/linux/tree/rpi-3.2.27), without any kind of problems related to sdcard, recently i tried to update it to newer ones to check whats going on (is it any better,faster etc. and also update rest of my system),
so i tried 3.9 (from https://github.com/raspberrypi/linux/tree/rpi-3.9.y) it started to show those errors right after boot(even though i disabled overclocking and was trying all different commandline options),
so i switched to 3.6(from https://github.com/raspberrypi/linux) it worked for about 10h then the errors started to show up (although after reboots etc. they started showing earlier but still not as fast as on 3.9).
so i switched back again to 3.2.27 and it works perfectly without any of those errors again.
I could provide more infomation(config etc.) if its necessary/useful at all.

@Ferroin
Ferroin commented Jun 4, 2013

As I had said earlier, I have seen these errors very infrequently on 3.2.27, but it was only 2 errors in a 24 hour period after more than 72 hours of no errors. If i remember correctly, I was building LLVM and clang at the time, which is an almost 48 hour process on the pi with the overclocking I use.

@Ferroin
Ferroin commented Jun 4, 2013

@macerg do you have any other Linux (or other *nix) system that you could try running badblocks on the card from? I usually get errors like that either when the card isn't getting enough power (flash memory takes more energy to write than it does to read), or when the card is nearing the end of it's usable lifetime.

@ewheelerinc

@Neverous ,

The biggest change related to MMC was sdhci-bcm2708.enable_llm which attempts to decrease IO latency side effects for audio and such.

Try compiling without CONFIG_PREEMPT or set sdhci-bcm2708.enable_llm=0 on the command line. This fixed the problem on my end.

@ChristophHaag

@remsnet Thanks, I have tried a bit but it gets only worse.

For example, after a few tries with different parameters I am at

[    0.000000] Kernel command line: dma.dmachans=0x7f35 bcm2708_fb.fbwidth=1920 bcm2708_fb.fbheight=1080 bcm2708.boardrev=0xf bcm2708.serial=0x5a095a2a smsc95xx.macaddr=B8:27:EB:09:5A:2A sdhci-bcm2708.emmc_clock_freq=100000000 vc-cma-mem=0/0x1c00000@0x1d000000 mem=0x1c000000@0x0 mem=0x1c00000@0x1d000000 vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000  smsc95xx.turbo_mode=Y dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=noop sdhci-bcm2708.missing_status=0 sdhci-bcm2708.sync_after_dma=0 sdhci-bcm2708.enable_llm=0 rootwait

But this completeley broke the filesystem again I think.

[  432.589743] mmc0: Controller never released inhibit bit(s).
[  432.616597] mmcblk0: unknown error -5 sending read/write command, card status 0x900
[  432.649096] end_request: I/O error, dev mmcblk0, sector 191640
[  432.676154] Buffer I/O error on device mmcblk0p2, logical block 659
[  432.704513] lost page write due to I/O error on mmcblk0p2
[  440.986807] mmc0: Controller never released inhibit bit(s).
[  441.020302] mmcblk0: unknown error -5 sending read/write command, card status 0x900
[  441.052789] end_request: I/O error, dev mmcblk0, sector 449536
[  441.080042] end_request: I/O error, dev mmcblk0, sector 449544
[  441.107445] end_request: I/O error, dev mmcblk0, sector 449552
[  441.134766] Buffer I/O error on device mmcblk0p2, logical block 32896
[  441.163971] Buffer I/O error on device mmcblk0p2, logical block 32897
[  441.193202] Buffer I/O error on device mmcblk0p2, logical block 32898
[  441.222433] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:294: I/O error writing to inode 7780 (offset 0 size 12288 starting block 56195)
[  463.595652] mmc0: Controller never released inhibit bit(s).
[  463.629907] mmcblk0: unknown error -5 sending read/write command, card status 0x900
[  463.662804] end_request: I/O error, dev mmcblk0, sector 645296
[  463.690352] Buffer I/O error on device mmcblk0p2, logical block 57366
[  463.719880] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:294: I/O error writing to inode 18384 (offset 0 size 4096 starting block 80663)

etc.

Can PLEASE the manufacturer of the hardware step in and have a look why so many people can't write reliably to the intended device that holds the root file system?

@ghollingworth
Contributor

For the last two months I've been asking people who've reported a reproducible issue to send me their Pi, power supply and sdcard with instructions on how to best reproduce it...

Unfortunately nobody has so we cannot make any progress

Gordon

@ChristophHaag

Well, I'm in germany and I think sending it around half the world is not the best solution. If ssh access (with root access of course) would help I'd be happy to give it though.

As for the sd card, I only know that it's a sandisk ultra 16 gb sdhc with 30 MB/s but there are still different cards that fit that description, even with class 6 and class 10.

@licaon-kter

Gordon is UK based afaik :)

@ghollingworth
Contributor

Yes raspberry Pi is based in the UK

@ChristophHaag

UK sounds almost ok, but it's still an international shipping I'd only want to do if really necessary.

So... is ssh access really not enough, do you need to look at the hardware?

@ghollingworth
Contributor

Yes,

I need to plug in the JTAG debug for the GPU, to understand what is
happening and that can't be done over ssh.

I'll make sure someone sends you a complete new set back out so you can
get something that worksŠ

Gordon Hollingworth,
Raspberry Pi,
Mount Pleasant House,
Mount Pleasant,
Cambridge
CB3 0RN
UK

@ChristophHaag

Ok, I'll send you the whole package on monday, the usb hub I used, the sd card with my archlinux (using linux-raspberrypi-latest 3.9.4) and the passwords and the raspberry pi itself.
I don't use it for much currently and I have other devices so a few days or weeks without one are not a problem for me.

@ghollingworth
Contributor

Thanks,

Can you give expected time to failure and how you've reproduced the problem?

Gordon

@ChristophHaag

With the default configs I get this error:

[13288.492420] mmc0: final write to SD card still running
[13298.479242] mmc0: Timeout waiting for hardware interrupt - cmd12.
[13298.480375] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900

Often it appears already while booting. If not, it is provoked in a matter of seconds to a few minutes just doing some disk i/o, copying stuff or updating or something like that. I think this error results in mostly (?) slowdowns while it recovers, but other I/O errors almost surely lead to serious file system corruption.

I have tried some of the combinations of kernel parameters posted over the course of this thread and sometimes the errors got more rare, but I don't think they ever vanished. Only for some combinations that error was replaced with more serious I/O errors like the one I posted above (but I think I still had it overclocked at this point and pretty much forgot to test whether it would vanish with default frequencies and now it's already packed up). But anyway, I think they main point is that neither errors should appear and rather have workarounds implemented.

@hvenzke
hvenzke commented Jun 10, 2013

Hello fellows ,

my thinking are

  • no use of Overclock at all
  • no use preempt kernels for sd cards bigger then 4G

@popcornmix / @ghollingworth ,
may you able to provide an Non-preempt kernel.img with branch next please as an seperat image i.e as "server_kernel.img" PLEASE.

@ghollingworth @popcornmix

I have the 0x900 issue as well with Pi Model A with class 4 Transcend 16G SDHC cards .

@ghollingworth
Contributor

So have been looking into the problem from @ChristophHaag and it seems it's purely related to the card.

Imaging the card and copying to a class 4 card and the problem goes away... Just need to understand SDCard specification and the timing issues to find out what's happened now... Christoph, am returning your kit with replaced Class 4 4G card + camera (sorry don't have any 16G cards in the office) so thought a camera might be a good replacement

Thanks

Gordon

@oerik
oerik commented Jun 16, 2013

As creator of this ticket I wonder if this is the right time to close it. I was able to resolve my issue with the 3.8 and 3.9 kernels with 'sdhci_bcm2708.sync_after_dma=0' in cmdline.txt. I believe the phenomenon is not fully understood but the workarounds seem ok to me. I just rebooted a rpi after 15 days uptime (every hour there is data written) and no problems detected

@ghollingworth I have a 4Gb class 10 micro sd card that works oob with 3.6 and failed on 3.8 and 3.9 (multiple rPI's testen). Let me know if you are interested in this card.

@ottuzzi
ottuzzi commented Jun 18, 2013

@ghollingworth I think that in this thread we have different cases:

  • SD corruption
    • @ChristophHaag with a problematic SD
    • @oerik with another problematic SD but it looks like it is fixed using sdhci_bcm2708.sync_after_dma=0 in cmdline.txt
  • SD corruption under overclocking conditions (my own situation)

I think we should split all of these as they are probably different: I would propose to close this 'very rich' thread and move on single entity reporting.
I also think that it should be made clear if SD corruption under overclocking is a problem or not: as a user I would like my raspi can be overclocked at 1000MHz without problem but I can understand if you say 'if you overclock you are on your own'. To sum up: should you reject SD corruption under overclock? If it is so of all this thread I think the only real issue is @oerik's one.

Bye
Piero

@popcornmix
Contributor

My best guess is that overclock based sdcard corruption is probably determined by the quality of the power supply. If you can play with a variety of power supplies, I'd be interested in hearing results.

Other than that, if you are suffering sdcard corruption when overclocking, reduce the overclock, or try booting from a USB/NFS rootfs.

We are intending on spending some effort on understanding this fully, but resources are limited.

I'll close this issue, as oerik is happy. If anyone has new information, then continue to post here. If your issue is not covered here, then create a new issue.

@popcornmix popcornmix closed this Jun 18, 2013
@ottuzzi
ottuzzi commented Jun 18, 2013

Hi @popcornmix,

in my own setup (8GB Class 10 Samsung SD MB-SP8GA/EU, linux 3.6.11+) I tried two different power supplies (one is a 2A Nexus 7 charger from Asus) and in both cases I saw SD corruption when overclocking: the culprit in my setup seems to be overclocking.
I actually bought a second raspberry pi, a second power supply [1] and a new SD [2] and I'll experiment on this: I'll let you know if I find something interesting.

Thanks for your work
Bye
Piero

[1]http://raspberrypi.rsdelivers.com/product/rs/hnp10i-microusb/micro-usb-multi-head-power-supply-for-raspberry-pi/7653311.aspx
[2]http://www.sony.co.uk/product/sd-expert-uhs-94mbs-sdhc-cards/sf8ux

@Srokap
Srokap commented Aug 2, 2013

FYI I have this problem without overclocking, on original card, but when using power supply that gives only 550mA (700mA is theoretical absolute minimum requirement).

@popcornmix
Contributor

Yes, we believe overclocking related sdcard corruption is caused by voltage drops from power supply.
Your power supply is likely to drop voltage when current reaches 550mA.

@natevw
natevw commented Oct 29, 2013

I've been powering my RasPi off an Asus RT-N16 and dealing with all sorts of SD Card lockup and corruption issues. I'm not overclocking but am starting to suspect the router's USB port is only supplying ~500mA and this may still be the cause. I'll try a dedicated USB power transformer after the next system rebuild.

@anholt anholt pushed a commit to anholt/linux that referenced this issue Oct 12, 2015
Geert Uytterhoeven + Thomas Gleixner irqchip/renesas-irqc: Use a separate lockdep class
The renesas-irqc interrupt controller is cascaded to the GIC. Hence when
propagating wake-up settings to its parent interrupt controller, the
following lockdep warning is printed:

    =============================================
    [ INFO: possible recursive locking detected ]
    4.2.0-ape6evm-10725-g50fcd7643c034198 #280 Not tainted
    ---------------------------------------------
    s2ram/1072 is trying to acquire lock:
    (&irq_desc_lock_class){-.-...}, at: [<c008d3fc>] __irq_get_desc_lock+0x58/0x98

    but task is already holding lock:
    (&irq_desc_lock_class){-.-...}, at: [<c008d3fc>] __irq_get_desc_lock+0x58/0x98

    other info that might help us debug this:
    Possible unsafe locking scenario:

	  CPU0
	  ----
     lock(&irq_desc_lock_class);
     lock(&irq_desc_lock_class);

    *** DEADLOCK ***

    May be due to missing lock nesting notation

    6 locks held by s2ram/1072:
    #0:  (sb_writers#7){.+.+.+}, at: [<c012eb14>] __sb_start_write+0xa0/0xa8
    #1:  (&of->mutex){+.+.+.}, at: [<c019396c>] kernfs_fop_write+0x4c/0x1bc
    #2:  (s_active#24){.+.+.+}, at: [<c0193974>] kernfs_fop_write+0x54/0x1bc
    #3:  (pm_mutex){+.+.+.}, at: [<c008213c>] pm_suspend+0x10c/0x510
    #4:  (&dev->mutex){......}, at: [<c02af3c4>] __device_suspend+0xdc/0x2cc
    #5:  (&irq_desc_lock_class){-.-...}, at: [<c008d3fc>] __irq_get_desc_lock+0x58/0x98

    stack backtrace:
    CPU: 0 PID: 1072 Comm: s2ram Not tainted 4.2.0-ape6evm-10725-g50fcd7643c034198 #280
    Hardware name: Generic R8A73A4 (Flattened Device Tree)
    [<c0018078>] (unwind_backtrace) from [<c00144f0>] (show_stack+0x10/0x14)
    [<c00144f0>] (show_stack) from [<c0451f14>] (dump_stack+0x88/0x98)
    [<c0451f14>] (dump_stack) from [<c007b29c>] (__lock_acquire+0x15cc/0x20e4)
    [<c007b29c>] (__lock_acquire) from [<c007c6e0>] (lock_acquire+0xac/0x12c)
    [<c007c6e0>] (lock_acquire) from [<c0457c00>] (_raw_spin_lock_irqsave+0x40/0x54)
    [<c0457c00>] (_raw_spin_lock_irqsave) from [<c008d3fc>] (__irq_get_desc_lock+0x58/0x98)
    [<c008d3fc>] (__irq_get_desc_lock) from [<c008ebbc>] (irq_set_irq_wake+0x20/0xf8)
    [<c008ebbc>] (irq_set_irq_wake) from [<c0260770>] (irqc_irq_set_wake+0x20/0x4c)
    [<c0260770>] (irqc_irq_set_wake) from [<c008ec28>] (irq_set_irq_wake+0x8c/0xf8)
    [<c008ec28>] (irq_set_irq_wake) from [<c02cb8c0>] (gpio_keys_suspend+0x74/0xc0)
    [<c02cb8c0>] (gpio_keys_suspend) from [<c02ae8cc>] (dpm_run_callback+0x54/0x124)

Avoid this false positive by using a separate lockdep class for IRQC
interrupts.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
Cc: Grygorii Strashko <grygorii.strashko@ti.com>
Cc: Magnus Damm <magnus.damm@gmail.com>
Cc: Jason Cooper <jason@lakedaemon.net>
Link: http://lkml.kernel.org/r/1441798974-25716-2-git-send-email-geert%2Brenesas@glider.be
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
b137065
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment