New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting a lot of random kernel panics / spinlocks #122

Closed
wipeDx opened this Issue Mar 5, 2018 · 14 comments

Comments

Projects
None yet
3 participants
@wipeDx
Copy link

wipeDx commented Mar 5, 2018

Hello,
I'm getting a lot of kernel panics when doing specific as well as unspecific things.
I'm running a Rock64 with stable stretch minimal with 4Gigs of Ram and 64Gb eMMC module.
Kernel version is 4.4.77-rockchip-ayufan-136
Also, I'm running a btrfs setup on it. Though the issues were had when I first flashed dietpi on it.
On it is running a nextcloud server along with postgresql, plex and a jdownloader instance.

What I've noticed is I kernel panicked the rock twice while connecting to my nextcloud instance.
This was probably one of the instance where I did that:
https://gist.github.com/wipeDx/0b0a7b885f50f88815fda12642d3e020

The second kernelpanic that's still in my syslog seems to be caused by plex, even though it's just running in the background right now, not doing anything.

https://gist.github.com/wipeDx/498da998a870f4aa7a6535ba42490259

I'll update this as much as possible with recent kernelpanics and logs where it actually says spinlock on cpu.

@wipeDx

This comment has been minimized.

Copy link

wipeDx commented Mar 5, 2018

Just started it again and wanted to sync my nextcloud on a different device, then was on my terminal:

https://gist.github.com/wipeDx/1f241730d47adec8d5e691f2c63bac08
And the rock is unresponsive

@ayufan

This comment has been minimized.

Copy link
Member

ayufan commented Mar 5, 2018

@wipeDx

This comment has been minimized.

Copy link

wipeDx commented Mar 5, 2018

I tried kernel 4.16 -> bandwidth got below 100kbyte/s so I couldn't really do much with it. So I tried 4.15, which also had the same problem.
Now with 4.14, it stops booting (power LED blinks twice, then a short pause)

I'll debug that tomorrow with the serial console I guess

(On the bright side, I didn't have any kernel panics so far!)

@ayufan

This comment has been minimized.

Copy link
Member

ayufan commented Mar 5, 2018

Still 4.4, but with latest patches and slightly different config.

@wipeDx

This comment has been minimized.

Copy link

wipeDx commented Mar 5, 2018

Oh I see. Guess I'll still debug that very soon then.

Thanks for your amazingly fast support though!

@wipeDx

This comment has been minimized.

Copy link

wipeDx commented Mar 5, 2018

Now I can't read the eMMC module anymore. I just can't. Guess the power LED blinking twice means there's a problem with it

@wipeDx

This comment has been minimized.

Copy link

wipeDx commented Mar 6, 2018

Got the 4.4.114 finally installed on the eMMC (finally got it to show up again, if someone needs to know how:
if you can get into uboot somehow (via an SD for example), you can actually erase the whole eMMC via the mmc command)

But no dice. Here, in any way I try to create the btrfs partition and then boot, I end up with kernel panics.
I'm afraid this time I can't provide you any kernel panics, cause they're simply too huge. Overloading the serial terminal as well as (i think) the copy/paste buffer of my OS.

If I go: Flash on eMMC from SD -> convert mmcblk0p7 to btrfs and change the boot entry to look for a btrfs -> boot, I end up with a working system, but when doing a simple apt update and/or upgrade/install, I end up with a kernel panic caused by btrfs (if you wish, I can replicate that, it just takes a lot of time)

If I go: Flash on eMMC from SD -> boot eMMC -> reboot SD to convert btrfs and change the boot entry -> boot eMMC, I end up with a working system again, that kernel panics with a huge entry after just a few seconds.

I'm gonna try an earlier kernel now.

@ayufan

This comment has been minimized.

Copy link
Member

ayufan commented Mar 6, 2018

You can use screen with file logging.

@wipeDx

This comment has been minimized.

Copy link

wipeDx commented Mar 6, 2018

Okay, I've tried it with 4.4.191 now, same result.
This is approach 2 btw, first boot was after being converted to btrfs.
This is the kernel panic after apt upgrade

https://gist.github.com/wipeDx/806977e5b5c4d545b30d7c6cd5e615c5

the filesystem becomes read-only after that point only. It is writable upon boot

@wipeDx

This comment has been minimized.

Copy link

wipeDx commented Mar 6, 2018

I tried again with my SD, falling into the same problems. Also not sure if you got notified on the gist, so I'm linking it here again.

https://gist.github.com/wipeDx/806977e5b5c4d545b30d7c6cd5e615c5#file-kernel_panic_5_sd-txt

@xalius xalius added the bug label Mar 30, 2018

@ayufan

This comment has been minimized.

Copy link
Member

ayufan commented Apr 2, 2018

This should be fixed as of 0.6.30 (requires full install). We fixed some potential problem related to DRAM stability. Please comment if this is not solved and the issue will be re-opened.

@ayufan ayufan closed this Apr 2, 2018

@ayufan

This comment has been minimized.

Copy link
Member

ayufan commented Apr 2, 2018

It is also possible to follow this: #152 (comment)

@wipeDx

This comment has been minimized.

Copy link

wipeDx commented Apr 24, 2018

I'm not entirely sure if this is related to my old problem, but now I can't seem to boot any of the images anymore.
Following builds have been tested:
stretch minimal 0.6.25-193
stretch minimal 0.6.30-208
stretch minimal 0.6.31-309
stretch minimal 0.6.32-310
stretch openmediavault 0.6.32-310

All of them end up in the following scenario:
https://gist.github.com/wipeDx/d3c23f76523ba1c3e46124f8f0266e7a
This is with just an SD Card, the eMMC Module is shorted/disabled

Sometimes the trace ends with Bad PC value
Googling after that only brought some questionable results from 6-8 years ago, so I have no idea what to do anymore.

If you need any more information, I'll gladly provide them

(Edited a huge lot away from this post, since I thought the old stable isn't able to boot anymore, which was wrong, I still can boot it)

@wipeDx wipeDx referenced this issue Apr 25, 2018

Open

Can't boot #96

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment