Getting a lot of random kernel panics / spinlocks

[wipedx]

Hello,
I'm getting a lot of kernel panics when doing specific as well as unspecific things.
I'm running a Rock64 with stable stretch minimal with 4Gigs of Ram and 64Gb eMMC module.
Kernel version is 4.4.77-rockchip-ayufan-136
Also, I'm running a btrfs setup on it. Though the issues were had when I first flashed dietpi on it.
On it is running a nextcloud server along with postgresql, plex and a jdownloader instance.

What I've noticed is I kernel panicked the rock twice while connecting to my nextcloud instance.
This was probably one of the instance where I did that:
https://gist.github.com/wipeDx/0b0a7b885f50f88815fda12642d3e020

The second kernelpanic that's still in my syslog seems to be caused by plex, even though it's just running in the background right now, not doing anything.

https://gist.github.com/wipeDx/498da998a870f4aa7a6535ba42490259

I'll update this as much as possible with recent kernelpanics and logs where it actually says spinlock on cpu.

[wipedx]

Just started it again and wanted to sync my nextcloud on a different device, then was on my terminal:

https://gist.github.com/wipeDx/1f241730d47adec8d5e691f2c63bac08
And the rock is unresponsive

[ayufan]

Try to use the pre-release kernel and see if this helps.

…

On Mon, Mar 5, 2018 at 5:37 PM, Raoul H. ***@***.***> wrote: Just started it again and wanted to sync my nextcloud on a different device, then this happened: https://gist.github.com/wipeDx/1f241730d47adec8d5e691f2c63bac08 And the rock is unresponsive — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#122 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACTpQaa4hpwuFb9bVHfr5jWMLbbkpBQeks5tbWmygaJpZM4Scl3w> .

[wipedx]

I tried kernel 4.16 -> bandwidth got below 100kbyte/s so I couldn't really do much with it. So I tried 4.15, which also had the same problem.
Now with 4.14, it stops booting (power LED blinks twice, then a short pause)

I'll debug that tomorrow with the serial console I guess

(On the bright side, I didn't have any kernel panics so far!)

[ayufan]

Well, no. Try this: https://github.com/ayufan-rock64/linux-build/releases/download/0.6.25/linux-headers-4.4.114-rockchip-ayufan-193_0.6.25_arm64.deb.

[ayufan]

Still 4.4, but with latest patches and slightly different config.

[wipedx]

Oh I see. Guess I'll still debug that very soon then.

Thanks for your amazingly fast support though!

[wipedx]

Now I can't read the eMMC module anymore. I just can't. Guess the power LED blinking twice means there's a problem with it

[wipedx]

Got the 4.4.114 finally installed on the eMMC (finally got it to show up again, if someone needs to know how:
if you can get into uboot somehow (via an SD for example), you can actually erase the whole eMMC via the mmc command)

But no dice. Here, in any way I try to create the btrfs partition and then boot, I end up with kernel panics.
I'm afraid this time I can't provide you any kernel panics, cause they're simply too huge. Overloading the serial terminal as well as (i think) the copy/paste buffer of my OS.

If I go: Flash on eMMC from SD -> convert mmcblk0p7 to btrfs and change the boot entry to look for a btrfs -> boot, I end up with a working system, but when doing a simple apt update and/or upgrade/install, I end up with a kernel panic caused by btrfs (if you wish, I can replicate that, it just takes a lot of time)

If I go: Flash on eMMC from SD -> boot eMMC -> reboot SD to convert btrfs and change the boot entry -> boot eMMC, I end up with a working system again, that kernel panics with a huge entry after just a few seconds.

I'm gonna try an earlier kernel now.

[ayufan]

You can use screen with file logging.

[wipedx]

Okay, I've tried it with 4.4.191 now, same result.
This is approach 2 btw, first boot was after being converted to btrfs.
This is the kernel panic after apt upgrade

https://gist.github.com/wipeDx/806977e5b5c4d545b30d7c6cd5e615c5

the filesystem becomes read-only after that point only. It is writable upon boot

[wipedx]

I tried again with my SD, falling into the same problems. Also not sure if you got notified on the gist, so I'm linking it here again.

https://gist.github.com/wipeDx/806977e5b5c4d545b30d7c6cd5e615c5#file-kernel_panic_5_sd-txt

[ayufan]

This should be fixed as of 0.6.30 (requires full install). We fixed some potential problem related to DRAM stability. Please comment if this is not solved and the issue will be re-opened.

[ayufan]

It is also possible to follow this: #152 (comment)

[wipedx]

I'm not entirely sure if this is related to my old problem, but now I can't seem to boot any of the images anymore.
Following builds have been tested:
stretch minimal 0.6.25-193
stretch minimal 0.6.30-208
stretch minimal 0.6.31-309
stretch minimal 0.6.32-310
stretch openmediavault 0.6.32-310

All of them end up in the following scenario:
https://gist.github.com/wipeDx/d3c23f76523ba1c3e46124f8f0266e7a
This is with just an SD Card, the eMMC Module is shorted/disabled

Sometimes the trace ends with Bad PC value
Googling after that only brought some questionable results from 6-8 years ago, so I have no idea what to do anymore.

If you need any more information, I'll gladly provide them

(Edited a huge lot away from this post, since I thought the old stable isn't able to boot anymore, which was wrong, I still can boot it)