New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Plugging in Sensors prevents boot. #623
Comments
Which sensors and motors (model numbers) do you have plugged into which port? Bluetooth not starting is a known problem (#572) and happens more often when more sensors are plugged in. I haven't ever had it not finish booting though. |
I don't know the motor model numbers, but they are all Lego ones. I have a |
So, I have tried it at least 10 times, and I successfully booted it 4 times in a row with the sensors plugged in, but then twice it got stuck during boot, and just the two leds on each side of the center button blinked every second, but it was stuck on the ev3dev booting screen. But it seemed the first time it failed that it did not connect with bluetooth to my computer or something similar on the previous boot. It was assigned a IP address showing that it connected at some point, but when I looked at it the bluetooth did not have the black background on it. I would conclude it has to do with the bluetooth connecting automatically and the sensors causing the bluetooth to have issues, although I have not really tested too much without the sensors plugged in. |
I think I have been seeing this lately as well with two of my EV3s - running the ckt-9 kernel right now. I'll try to see if the boot is more reliable with NO devices plugged in anywhere. |
Have you tried using an older version? |
I downgraded the kernel because of a motor issue, and it hasn't not booted, On Friday, May 6, 2016, markyi370 notifications@github.com wrote:
|
I have a suspicion that it has to do with this commit: ev3dev/lego-linux-drivers@fa0d807 In your original post, you say that the problem occurs with kernel version Can you confirm which kernel version has the problem and which does not? Also, can you confirm that the issue only happens when EV3/UART sensors are plugged in, but not when any other type of sensor is plugged in? If you have anything to say about Bluetooth, leave a comment in #572 and not here so that we don't confuse the two issues. |
I'm a bit confused now that I think about it, I upgraded my kernel after On Friday, May 6, 2016, David Lechner <notifications@github.com
|
Okay, the kernel version is: 3.16.7-ckt21-9-ev3dev-ev3 So I must have upgraded to 3.16.7-ckt21-10-ev3dev-ev3, and then when I discovered that motors using python no longer worked I went back down to 3.16.7-ckt21-9-ev3dev-ev3. So it has always booted so far since doing that for some reason, although I haven't booted it many times since then. The bluetooth seems to be semi-reliable, and usually I leave it on the whole time when I'm working on a program, so a quick reboot if its not working is fine till it starts working. I will report back if the ev3 does not boot, but so far it seems to be good. |
This reverts commit fa0d807. It is possibly causing deadlocks when multiple EV3/UART sensors are plugged in during boot. Issue ev3dev/ev3dev#623
This reverts commit 0a6dc8e. This is possibly causing deadlocks on boot. Issue ev3dev/ev3dev#623
I have just released kernel v4.4.9-11-ev3dev-ev3. Please give it at try. I was able to reproduce this problem when testing this kernel. Unfortunately, I was not able to get to the root cause. There seem to be some data corruption issues related to the UARTs (reminiscent of #47). However, I have done a number of things that seem to help.
I still had a crash on shutdown once related to EV3/UART sensors (the line discipline driver) but haven't been able to repeat it. So, I am afraid that there are still some bugs lurking about, but I think this issue is "fixed" well enough, I hope. |
Dang it. I just got a deadlock during boot of |
I have noticed similar problems, I think more often (only?) when devices are attached to the ev3. But I've just done a couple dozen boot cycles with the serial console attached (and many devices) and I have not had hang (this is after three hangs in the preceding hour - NXT temp sensor, 2 large motors, the LEGO IR and a medium motor). |
which kernel are you using? |
3.16.7-ckt26-10-ev3dev-ev3 and 4.4.9-11-ev3dev-ev3 both showed this behaviour. |
Setup: It occasionaly stops when booting. Interesting lines from dmesg (when it succeeds!) [ 2.402921] serial8250: too much work for irq53
[ 2.598897] serial8250: too much work for irq53
[ 2.794045] serial8250: too much work for irq53
[ 2.989177] serial8250: too much work for irq53
[ 3.185087] serial8250: too much work for irq53
[ 3.380985] serial8250: too much work for irq53
[ 3.576130] serial8250: too much work for irq53
[ 3.771258] serial8250: too much work for irq53
[ 3.967129] serial8250: too much work for irq53
[ 4.162236] serial8250: too much work for irq53
[ 4.910667] ttyS1: 1 input overrun(s)
[ 6.754650] ttyS1: 3 input overrun(s)
[ 7.512047] serial8250_interrupt: 11 callbacks suppressed
[ 7.516772] serial8250: too much work for irq53
[ 7.713071] serial8250: too much work for irq53
[ 7.910077] serial8250: too much work for irq53
[ 8.105974] serial8250: too much work for irq53
[ 8.301875] serial8250: too much work for irq53
[ 8.497750] serial8250: too much work for irq53
[ 8.693666] serial8250: too much work for irq53
[ 8.888793] serial8250: too much work for irq53
[ 9.084712] serial8250: too much work for irq53
[ 9.280622] serial8250: too much work for irq53
[ 9.482399] ttyS1: 1 input overrun(s)
#[...]
[ 13.102983] serial8250_interrupt: 3 callbacks suppressed
[ 13.107066] serial8250: too much work for irq53
[ 13.303575] serial8250: too much work for irq53
[ 13.499547] serial8250: too much work for irq53
[ 13.695461] serial8250: too much work for irq53
[ 13.890594] serial8250: too much work for irq53
[ 14.086511] serial8250: too much work for irq53
[ 14.281647] serial8250: too much work for irq53
[ 14.477522] serial8250: too much work for irq53
[ 14.672625] serial8250: too much work for irq53
[ 14.867742] serial8250: too much work for irq53
[ 18.457560] serial8250_interrupt: 16 callbacks suppressed
[ 18.461727] serial8250: too much work for irq53
[ 18.657644] serial8250: too much work for irq53
[ 18.853545] serial8250: too much work for irq53
[ 19.049429] serial8250: too much work for irq53
[ 19.245357] serial8250: too much work for irq53
[ 19.442005] serial8250: too much work for irq53
[ 19.637983] serial8250: too much work for irq53
[ 19.833906] serial8250: too much work for irq53
[ 20.029903] serial8250: too much work for irq53
[ 20.225062] serial8250: too much work for irq53
#[...]
[ 23.771228] serial8250_interrupt: 16 callbacks suppressed
[ 23.775403] serial8250: too much work for irq53
[ 23.971342] serial8250: too much work for irq53
[ 24.167508] serial8250: too much work for irq53
[ 24.363088] serial8250: too much work for irq53
[ 24.559083] serial8250: too much work for irq53
[ 24.755000] serial8250: too much work for irq53
[ 24.950883] serial8250: too much work for irq53
[ 25.146026] serial8250: too much work for irq53
[ 25.341186] serial8250: too much work for irq53
[ 25.536342] serial8250: too much work for irq53
[ 25.926385] systemd[1]: Started Journal Service.
[ 26.325724] EXT4-fs (mmcblk0p2): re-mounted. Opts: errors=remount-ro
[ 29.514846] systemd-udevd[107]: starting version 215
[ 32.691103] random: nonblocking pool is initialized
[ 32.789137] lego-port port0: Registered 'in1' on 'legoev3-ports'.
[ 32.789967] lego-port port1: Registered 'in2' on 'legoev3-ports'.
[ 32.790866] lego-port port2: Registered 'in3' on 'legoev3-ports'.
[ 32.791697] lego-port port3: Registered 'in4' on 'legoev3-ports'.
[ 32.792636] lego-port port4: Registered 'outA' on 'legoev3-ports'.
[ 32.793581] lego-port port5: Registered 'outB' on 'legoev3-ports'.
[ 32.950294] lego-port port6: Registered 'outC' on 'legoev3-ports'.
[ 32.951379] lego-port port7: Registered 'outD' on 'legoev3-ports'.
[ 33.340688] lego-port port0: Added new device 'in1:ev3-uart-host'
[ 33.516522] ti_omapl_pru_suart ti_omapl_pru_suart.1: fw size 3772. downloading...
[ 33.520692] ti_omapl_pru_suart.1: ttySU0 at MMIO 0x1d00000 (irq = 3, base_baud = 8250000) is a suart_tty
[ 33.582530] ti_omapl_pru_suart.1: ttySU1 at MMIO 0x1d00000 (irq = 4, base_baud = 8250000) is a suart_tty
[ 33.626532] ti_omapl_pru_suart ti_omapl_pru_suart.1: ti_omapl_pru_suart device registered(pru_clk=150000000, asp_clk=132000000)
#[...]
[ 47.673803] Registered EV3 UART sensor line discipline. (29)
[ 49.534774] lego-sensor sensor0: Registered 'lego-ev3-gyro' on 'in1'.
[ 49.904757] lego-sensor sensor1: Registered 'lego-ev3-us' on 'in2'.
Serial8250 is responsible for UART in Input Port 1 and 2. The interesting lines are those like this: [ 6.754650] ttyS1: 3 input overrun(s) I don't know how overruns are handled in the code but if it happened after [ 47.673803] Registered EV3 UART sensor line discipline. (29) And was unhandled it could lead to crash before recent @dlech fix: ev3dev/lego-linux-drivers@2fefd7d It would be interesting to see if it still crashes with most recent changes (kernel build from github master) |
I tried it, it didn't help. Setup: Still hanging occasionaly on boot. |
This is interesting. I am unable to reproduce this problem when the sensors are connected to port 3 and 4 (PRU SUARTS, software uarts, not hardware SoCs). 10/10 boots without a problem so far with Ultrasonic and gyro on port 3 and 4. |
It would be intesting to blacklist some modules, e.g. Unfortunately I have no more time today. |
@bmegli, Thanks for the
I don't think blacklisting any modules will make any difference since the error are happening in early boot before modules are loaded. Long ago, I tried disabling the tty on input port 1, but it had a weird interaction with FIQ - see #47. tty on input port 1 (ttyS1) needs to be disabled using the kernel command line. See |
If anyone wants to try this before I get around to it, replace
with
in Be aware that this file will be written over when the |
I suppose your explanation for messages is correct! I am not sure however if this is what stops boot process. I suppose that it's worth trying to disable tty on port 1 and check. Maybe tommorow (EV3 stayed at work...) Alternatively we could: |
I mean booting problem! ;-)
Overruns yes - before. Do we know when booting hangs? (I don't have a serial cable so I can't see) I meant blacklisting modules to pinpoint boot problem. |
@kortschak said
@kortschak do you remember if those earlier hangs were with some sensor plugged to port 1 or also the serial cable? |
This is why this problem is so hard to fix. If we attach a serial cable, it doesn't hang - at lease since we fixed #662. Maybe we can change it to |
Never when serial cable is connected. This is the classic Heisenbug. I'll have a chance to look at some of this this weekend. Sorry for the silence. |
@dlech Have you tried debugging both to 1 and 2 at the same time?
Is that possible? I belive we can still debug to both ttyS1 and lcd (tty0?)
Maybe we could debug to both ttyS1 and ttyUSB0. I don't fully understand what are the limitations here (from kernel.org docs). |
I think so, but in order to reproduce the problem, I have to have sensors on both port 1 and port 2, so no serial debugging.
Yes, however, Heissenbug strikes again. I have just tried this an replacing |
Hey, this bug is funny! ;-) For my EV3 it is enough to hang with debug set to port 1 and gyro plugged to port 1. I am searching the net, and original firmware EV3 also sometimes hangs at boot. Edit: I am not sure I have tried with original firmware and UART sensor plugged to port 1 many times |
you should paste the links here |
http://forums.usfirst.org/showthread.php?20835-EV3-lock-ups-and-rebooting The interesting fragment:
So it probably with multiple sensors/motors in when booting Edit: Another one:
|
I am thinking about just changing the kernel command line for everyone. It does have the effect of showing one line of text under the boot logo. It also shows messages when shutting down, which is actually kind of good if you are wondering why shutdown is taking so long. |
It could be good for troubleshooting boot problems for newcomers. Those that are unable to boot ev3dev at all. We can't even reproduce it. |
Yes, I will have to change the boot logo and font size though. Right now, it only has enough room to show |
Since we are going to enable kernel messages on tty0, we need a smaller font so that we can actually fit the messages on the screen. Also, we need a smaller logo so there is room for the messages on the screen. Issue ev3dev/ev3dev#623
Since we are going to enable kernel messages on tty0, we need a smaller font so that we can actually fit the messages on the screen. Also, we need a smaller logo so there is room for the messages on the screen. Issue ev3dev/ev3dev#623
Unfortunaly, this is done mainly to work around issues with hardware UARTS on EV3 that we can't figure out. The side effect is that console messages will print on the screen during boot, which can be useful for troubleshooting. Issue: ev3dev/ev3dev#623
Since we are going to enable kernel messages on tty0, we need a smaller font so that we can actually fit the messages on the screen. Also, we need a smaller logo so there is room for the messages on the screen. Issue ev3dev/ev3dev#623
I think I may be getting to the root cause or at least we are no longer "flying blind". Setup: When the booting freezees it's possible to:
Here's dmesg output from such scenario (expires in 30 days): Nothing obvious seems to be there. A lot of It's possible that we pause the boot process with some random sequence from UART sensor and resume with serial cable. Now, the interesting part is that I am unable to reproduce it (pausing boot with break, scrool lock, ctrl+s doesn't work for me) Other dmesg outputs were so flooded with There may be also another possiblity if uart sensor sends I once had a scenario where I ended in Edit:
Hmm, if I could do it from the serial console then in theory also UART sensor could do it (if sending such data) |
This is interesting because with the Please try upgrading |
Ok. In what scenario did you have the lockups? UART sensor plugged to ttyS1 or something different? For now, I was not setting console to both to pinpoint the boot problem (for which you have found workaround... but it's still good riddle) |
Even with console on both |
This is still causing problems with UART sensors on input port 1, so disable it completely by default Issue ev3dev/ev3dev#623
I haven't been able to reproduce lockup on shutdown but I can shed some light on bootup. Maybe shutdown is in some way similiar. Setup: Up to some point it's possible to recover by unplugging gyro, plugging serial and typing something. [ OK ] Mounted FUSE Control File System.
[ OK ] Mounted Configuration File System.
[ OK ] Started Load/Save Random Seed.
[ OK ] Started Apply Kernel Variables.
[ OK ] Started Create Static Device Nodes in /dev.
[ TIME ] Timed out waiting for device dev-mmcblk0p1.device.
[DEPEND] Dependency failed for /boot/flash.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for File System Check on /dev/mmcblk0p1.
[ OK ] Stopped LEGO MINDSTORMS EV3 LEDs.
[ OK ] Stopped target Graphical Interface.
[ OK ] Stopped Brick Manager.
[ OK ] Stopped target Multi-User System.
[ OK ] Stopped Enable support for additional executable binary formats.
[ OK ] Stopped OpenBSD Secure Shell server.
[ OK ] Stopped Login Service.
[ OK ] Stopped LSB: start Samba NetBIOS nameserver (nmbd).
[ OK ] Stopped LSB: Autogenerate and use a swap file.
[ OK ] Stopped LSB: Start NTP daemon.
[ OK ] Stopped Avahi mDNS/DNS-SD Stack.
[ OK ] Closed Avahi mDNS/DNS-SD Stack Activation Socket.
[ OK ] Stopped D-Bus System Message Bus.
[ OK ] Closed D-Bus System Message Bus Socket.
Starting Connection service...
[ OK ] Stopped Permit User Sessions.
Starting Set console font and keymap...
[ OK ] Reached target Remote File Systems.
Starting Trigger Flushing of Journal to Persistent Storage...
[ OK ] Stopped /etc/rc.local Compatibility.
[ OK ] Stopped Turn on global VT cursor.
[ OK ] Stopped getty on tty2-tty6 if dbus and logind are not available.
[ OK ] Reached target Login Prompts.
[ OK ] Stopped USB Gadget for LEGO MINDSTORMS EV3 hardware.
[ OK ] Stopped target Basic System.
[ OK ] Reached target Timers.
[ OK ] Reached target Sockets.
[ OK ] Stopped target System Initialization.
Starting Restore Sound Card State...
Starting Create Volatile Files and Directories...
Starting Emergency Shell...
[ OK ] Started Emergency Shell.
[ OK ] Reached target Emergency Mode.
Starting udev Kernel Device Manager...
[...] The timeout of The award winning boot that I was still able to recover from (by unplugging gyro and plugging serial) had this fragment: [ 233.322081] serial8250: too much work for irq53
[ 233.322667] ev3_uart_receive_buf: 3830 callbacks suppressed
[ 233.322715] ttyS1: buffer overrun So it seems that if |
Today, I have updated the |
FWIW, in the official LEGO firmware, they disable the interrupt on this port, I imagine to work around the same problems we are seeing here. https://github.com/mindboards/ev3sources/blob/master/lms2012/lms2012/source/lms2012.h#L155 |
Since we are going to enable kernel messages on tty0, we need a smaller font so that we can actually fit the messages on the screen. Also, we need a smaller logo so there is room for the messages on the screen. Issue ev3dev/ev3dev#623
Since we are going to enable kernel messages on tty0, we need a smaller font so that we can actually fit the messages on the screen. Also, we need a smaller logo so there is room for the messages on the screen. Issue ev3dev/ev3dev#623
Since we are going to enable kernel messages on tty0, we need a smaller font so that we can actually fit the messages on the screen. Also, we need a smaller logo so there is room for the messages on the screen. Issue ev3dev/ev3dev#623
No one has said this is still a problem, so closing. If you notice a similar problem, please open a new issue. |
System information:
About my issue:
Basically the ev3 will not boot if i keep my sensors and motors plugged in. This is kinda of annoying, but I was having an issue where the bluetooth wasn't starting, and then the ev3 was stuck on the ev3dev booting page and wouldn't go to brickman, so I unplugged everything and it booted. I'm not sure if this is a known bug or intended behavior but it is really annoying to me. I am going to test this more later, but I believe that it was the reason it wasn't booting, and in some cases just the bluetooth wouldn't show up oddly. This is odd because I was using it yesterday and booted it at least three times with everything plugged in, and it worked fine, so I am going to definitely test this because I was having issues with my computer's bluetooth setup which I have resolved now.
The text was updated successfully, but these errors were encountered: