Reboot from USB stopped working on 3B+ now Pi 3B+ flashes 4 times after reboot #968
Comments
|
Your error code blink sequence looks odd - the known codes only have 0, 2 or 4 long flashes. Can you double check the pattern? Link to a short video if you aren't sure. |
|
Interesting AFTER shutdown I see 10 flashes consistently. But here are more specifics. I acquired another Pi 3B+ and it WAS able to reboot we'll need to see if it can maintain that ability. We'll call this the GoodPi. I have several Pi-Day Pi 3B+ that exhibit consistent reboot failure using the exact same USB devices of different types - we'll call those BadPi. Both GoodPi and BadPi show repeated four rapid flashes when booting from USB without an SD where the flashes start at OS boot, but with an a blank SD card both do not flash. What are the constant 4 rapid flashes when booting from USB when the boot works fine? Using only a clean 2018-03-13-raspbian-stretch image etched on a USB with no SD card, Both GoodPi and BadPi 3B+ reboot successfully the first time to expand storage, then once they fully boot subsequent reboots fail on the BadPI and it flashes the Green LED 4 times short-short-short-long (more regular flashes not the rapid flashes above). On the GoodPi reboot works as expected. What are the 4 regular flashes when reboot fails? On shutdown both GoodPi and BadPi turn off the Red LED after shutdown then back on and then flash the Green LED 10 times. What are the 10 regular flashes after shutdown? Booth GoodPi and BadPi boot just fine after a shutdown. I haven't sampled enough devices to see if there are other BadPi and GoodPi in the wild. |
|
Not sure if this is exactly the same as I see 4 very fast flashes (https://www.youtube.com/watch?v=dqm8poGcTJw 4x slow motion) but I've seen occasional problems rebooting from USB whilst testing other things. I setup the RPi3+ booting off USB with Raspbian lite to auto login and then reboot from bashrc. Sometimes I get a good run of over 1000 reboots before it fails but it can also be every reboot. Good reboot (serial output)
Bad reboot
So it gets as far as fetching and booting the kernel from USB flash but then the USB flash isn't detected by the kernel. I tried removing and replugging the USB flash (before the "random: crng init done" shows up) but that didn't help. |
|
Four fast flashes means "Start not found", as reported by bootcode.bin when the load fails for any reason. The 10 flashes on shutdown is just a visual indication that shutdown has completed successfully. Comparing the successful and unsuccessful boots, after removing the timestamps and a certain amount of reordering you get: --- good.txt 2018-04-11 14:32:21.130087600 +0100
+++ bad.txt 2018-04-11 14:32:18.506254500 +0100
@@ -200,41 +200,4 @@
usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
hub 1-1.1:1.0: USB hub found
hub 1-1.1:1.0: 3 ports detected
-dwc_otg_handle_wakeup_detected_intr lxstate = 2
-usb 1-1.1.1: new high-speed USB device number 4 using dwc_otg
-usb 1-1.1.1: New USB device found, idVendor=0424, idProduct=7800
-usb 1-1.1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
-libphy: lan78xx-mdiobus: probed
-usb 1-1.1.3: new high-speed USB device number 5 using dwc_otg
-usb 1-1.1.3: New USB device found, idVendor=0781, idProduct=5583
-usb 1-1.1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
-usb 1-1.1.3: Product: Ultra Fit
-usb 1-1.1.3: Manufacturer: SanDisk
-usb 1-1.1.3: SerialNumber: 4C530001220114110455
-usb-storage 1-1.1.3:1.0: USB Mass Storage device detected
-scsi host0: usb-storage 1-1.1.3:1.0
-scsi 0:0:0:0: Direct-Access SanDisk Ultra Fit 1.00 PQ: 0 ANSI: 6
-sd 0:0:0:0: [sda] 240254976 512-byte logical blocks: (123 GB/115 GiB)
-sd 0:0:0:0: [sda] Write Protect is off
-sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
-sda: sda1 sda2
-sd 0:0:0:0: [sda] Attached SCSI removable disk
-EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
-VFS: Mounted root (ext4 filesystem) readonly on device 8:2.
-devtmpfs: mounted
-Freeing unused kernel memory: 1024K
-systemd[1]: System time before build time, advancing clock.
-NET: Registered protocol family 10
-Segment Routing with IPv6
-ip_tables: (C) 2000-2006 Netfilter Core Team
-systemd[1]: systemd 232 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
-systemd[1]: Detected architecture arm.
-systemd[1]: Set hostname to .
-systemd[1]: Listening on udev Kernel Socket.
-systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
-systemd[1]: Listening on Syslog Socket.
-systemd[1]: Created slice User and Session Slice.
-systemd[1]: Started Forward Password Requests to Wall Directory Watch.
-systemd[1]: Reached target Swap.
-systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
random: crng init doneIn other words, the kernel just isn't seeing any USB devices, including the built-in Ethernet controller - just the hubs. |
|
Probably not related, but thought I would mention it, if I fail to update the modules to match the kernel when testing a new kernel, lack of USB is the major symptom. That would probably be indicated in the logs though. |
|
Good point, but I don't think it's relevant in this case because in @burtyb's case the failure is intermittent and for @hoopsurfer it is board- (not model-) specific. @hoopsurfer Try adding the following to config.txt: |
|
Swapping out a GoodPi with a BadPi (everything attached/inserted is identical) fails to reboot with BTW, the four rapid flashes with no SD card are very rapid compared with the 4 flashes on reboot failures, still not clear what those mean since the happen on both GoodPi and BadPi and everything is working fine aside from the repeating flashes. |
|
I have 2 Pi 3 B+ that are exhibiting the exact same 4 flash behavior when using any firmware from 2018. When booting 2017 versions, I can boot successfully to the GPU test screen with a lightning bolt indicating outdated firmware. I can successfully boot into older releases of NOOBS and Raspbian, but again only those on 2017 firmware. I have returned and replaced both Pis already and they have all exhibited the exact same behavior. All of them were sourced from the same supplier. |
|
After sitting idle for the better part of 3 days, GoodPi failed an attempted reboot, after a power cycle it did boot and then reboot and seems to consistently reboot successfully again (tried 5 times). Something is not stable about 3B+ reboot behavior at least when booting from a USB device. I am using a tested 2.5A power adapter. Maybe if I let BadPi sit idle for 3 days it will start to reboot again. : ) |
|
@hoopsurfer Just looking into this a little, but starting by adding a whole tonne of changes to get some logging out... To confirm your problems: When you say failed to reboot, can you be more specific: How far does it get? Can you get a video of the Pi rebooting and failing showing both the LEDs and the HDMI output? Just want to make sure it is failing at the same point. |
|
Fail to reboot from USB consistently on one of my 3B+ BadPi means:
Nothing shows on he screen. I'll capture a video Monday. |
|
I think this means it failed to find start.elf but the LED is inverted and therefore the dashes are actually the on and the flashes are the 'off' I'm just not sure about whether we've correctly inverted the flash in bootcode.bin Gordon |
|
When preping to create a video, it appears GoodPi has now become a BadPi much to my disappointment. Reboot fails with or without a blank SD now. IF we assume inverted LEDs causes the long flash. What is going to cause failure to find start.elf when it consistently IS found on boot from power-off? |
|
It's worth mentioning that USB booting is still officially considered an experimental feature, isn't it? |
|
Also, bad start.elf is short-short-short-short-long-long-long-long, is it not? Doesn't it do the same number of shorts and longs. It has in every instance for me. |
|
It sounds like an intermittent (or environmentally sensitive) failure of the USB subsystem to reset properly during a reboot. We'll try to reproduce. |
|
My comment above is possibly more to relevant to @burtyb. @ghollingworth is currently hard at work on |
|
After leaving that system off for the past day and restarting it, it now reboots consistently including multiple USB devices. Now we're back to GoodPi. I cannot explain the change. |
|
@Nilpo we're talking Pi 3B+, which boots from USB by default, no wonky bit setting required. For me when reboot fails it is short short short long (repeating). When no SD card is present boot works but I see blink blink blink blink (repeating flashes). When shutdown it is short short short short short short short short short short (once). Same on multiple boards however I still have two BadPi (will not reboot) and one GoodPi (mostly reboots). I did reboot testing for some benchmarks and GoodPi did 30-40 in a row no problems (with and without SD cards and with multiple USB mSATA SSDs). |
|
I suppose that the reboot issues when netbooting could be related. I too get the repeated sequence of 4 flashes -- 3 short and 1 long. /var/log/sysconfig says that start.elf is not found. |
|
My issues turned out to be related to HDMI. |
kernel: firmware/raspberrypi: Notify firmware of a reboot See: #968 kernel: config: Add CONFIG_DM_CACHE See: raspberrypi/linux#2553 kernel: Cleanup of bcm2708_fb file to kernel coding standards kernel: Add ability to export the pin used by the gpio-poweroff driver See: raspberrypi/linux#2552 kernel: Revert: Sets the BCDC priority to constant 0 firmware: vc_image: vc_image_fill_yuv_uv sometimes overfilled the first column See: raspberrypi/userland#465 firmware: arm_loader: Add SET_NOTIFY_REBOOT message See: #968 firmware: gencmd: Report temperature with read_ring_osc values
kernel: firmware/raspberrypi: Notify firmware of a reboot See: raspberrypi/firmware#968 kernel: config: Add CONFIG_DM_CACHE See: raspberrypi/linux#2553 kernel: Cleanup of bcm2708_fb file to kernel coding standards kernel: Add ability to export the pin used by the gpio-poweroff driver See: raspberrypi/linux#2552 kernel: Revert: Sets the BCDC priority to constant 0 firmware: vc_image: vc_image_fill_yuv_uv sometimes overfilled the first column See: raspberrypi/userland#465 firmware: arm_loader: Add SET_NOTIFY_REBOOT message See: raspberrypi/firmware#968 firmware: gencmd: Report temperature with read_ring_osc values
|
I have now spent some more time looking at the reboot issue using two 3B+ devices. From hours of reboot testing I would say that with a USB mSATA attached I get random results when rebooting - sometimes it works sometimes it fails, once it works it seems to work for a while, once it fails it fails for a while. I cannot predict when it will work or fail or find any series of steps that causes a consistent result. In my case both 3B+ devices ALWAYS boot correctly (from power off), but often reboot can fail with the four flashing LEDs. Reboot failure means the system effectively hangs, power needs to be cycled to reboot. Forcing fsck doesn't generally make a difference, running with or without a blank SD card seems to make no difference. I need help debugging reboot failures. |
|
I saw there had been a couple of commits referencing this issue so I just tested with the latest rpi-update kernel to see if there had been any improvements in reboots on USB devices but things are now worse (doesn't reboot at all) - not sure if it's this known/expected atm? Using a fresh Raspbian Lite 2018-04-18 on an 16GB Sandisk Ultra Fit in a Pi3+ Adding "uart_enable=1" to config.txt. It boots OK on 4.14.34-v7+ and does the normal resize and reboot.
Boots into new 4.14.34-v7+ kernel.
Again boots into 4.14.34-v7+ kernel OK.
It updates to "fe525d2be041c1a9b924824e430b5d51214315c4", "4.14.50-v7+ #1122 SMP Tue Jun 19 12:26:26 BST 2018"
Boots into new 4.14.50-v7+ kernel.
I see the following on the serial console as it restarts.
And then both the PWR and ACT LED go OFF and never come back on. There's no sign of the boot process starting on the serial and the USB device is never powered back on either (I can measure 5.1919v / 3.2982v on the gpio header so power looks OK). After cycling the power it boots up OK but again doesn't come back when rebooted. Repeating the above steps with a"Crucial Gizmo! USB Device 8GB" it also fails. But on an SD card it works OK. |
|
I think we need to look back into the gpio expander driver again, I don't think it is properly resetting the LAN9515 at boot time, which it should do via the LAN_RUN_BOOT pin. But this requires that the LAN_RUN pin on the gpio_expander is tri-stated. |
|
Oops... Didn't mean to close! |
|
I was looking at this before leaving yesterday, and I think the problem must be that the VPU code to restore the internal LAN_RUN GPIO (LAN_RUN_BOOT) is clearing the (default) pull down setting, causing it to float. Sometimes this works, and sometimes it doesn't. I'll build and link to a modified start.elf for testing. |
|
The change I mentioned above didn't work, so I didn't bother uploading it. We're puzzled, but actively investigating the possible differences between a first-time boot and a reboot. |
See: #968 firmware: Allow selection of DSI port for LCD
See: raspberrypi/firmware#968 firmware: Allow selection of DSI port for LCD
|
Fixed on latest build |
|
Reconfirmed the original failure on current public firmware from 2018-06-07 15:30 running Linux 4.14.50-v7+ 1122 SMP Tue Jun 19 12:26:26 BST 2018. Tested after rpi-update to latest build firmware from 2018-06-29 13:12 running Linux 4.14.52-v7+ 1123 SMP Wed Jun 27 17:35:49 BST 2018 for 4 hours of continuous reboots with 100% success. I am no longer able to reproduce the reboot failure - consider #968 tested fixed. Thanks! |
|
Hi all, Although 3 years have passed I am currently experiencing the same issue.
Cold boot always succeeds. Setup utilized as a low-power backup NAS in secondary location. Any help or debugging guidance appreciated.
|
|
Also, in case reboot fails:
|
Boot works fine always (both from SD and USB), reboot never works from USB, but works fine from SD.
Using fresh Raspian images with apt update/apt full-upgrade/rpi-update applied.
Current uname -a output:
Linux raspberrypi 4.14.32-v7+ #1107 SMP Fri Apr 6 16:36:46 BST 2018 armv71 GNU/Linux
On reboot system fails to reboot, the green LED blinks: "short short short long " over and over
Fails both with and without blank SD card installed
Fails on multiple power supplies, multiple USB devices, multiple Pi 3B+s
Full power-off/on always boots and same Pi 3B+ devices rebooted with original Pi-day bits
The text was updated successfully, but these errors were encountered: