geth repeatedly crashes with SIGSEGV #20190
Comments
|
Fast sync block is Could you try it with some more detailed logging in place? |
|
@holiman Absolutely, here is the output from Click to expand |
|
Interesting. It looks like the block body for block |
|
@holiman LMK if I can provide any other data to help find the culprit here. Is there any way I can get through this impasse? |
One thing that might work is to use |
|
@m0ar the first crash you had -- do you know if that was the exact same crash as this one? I'm thinking that maybe the first crash left the db a bit corrupted, and now you're hitting this because some data is missing in the db. Also, what hardware are you running on ? What's the ram size? |
|
@holiman How can I set this before the client crashes? It fails within seconds. |
|
Got it done. It definitely ran for longer, but crashed again: |
I'm not sure, could be. Would a resync be the only solution then? It's this setup: https://kauri.io/article/9695fcca217f46feb355245275835fc0 |
|
Hi. I'm currently doing a fast sync on the Pi 4 and ran into this error (full log attached): It happened just once and the sync process seems normal (seems a pointer bug but I'm not sure if it is related to this issue). Just in case it helps. I will report back when the fast sync completes. Setup:
Geth: Official armv7 binary 1.9.6-stable @m0ar, if you do a resync, would you mind to try this image? https://ethraspbian.com/downloads/image_2019-10-20-EthRaspbian2.0-lite.zip Just flash it and the fast sync process will start in 2-3 minutes (keep in mind that it will wipe out your entire USB disk and reinstall everything). Login ethereum:ethereum. You will be prompted to change the password on first login. |
|
Hi. Fast sync process was completed in 4 days. But, I got lots of errors like this one: It turns out that setting the cache to 256 MB (it was 512MB) solved this and Geth runs fine. But, somehow this is weird as with other 2 ARM boards (ARM64) I was using the default cache value (a little more that 1GB) and never ran into this (and both devices have 4GB RAM as well). Is it possible that this errors are caused by some problem with Go's GC? |
|
Hi, got another error: I don't know if they are related. Let me know if you want me to open a separate issue. |
|
Hi, I did another Fast Sync with Geth 1.9.7. This time I only ran into an unallocated span issue (see attached). It seems that the out of memory issues were a Go problem, I will keep monitoring the process in the coming days though. |
|
Did a re-sync, this is the initial crash (that once again occured, now on v1.9.8): Run with |
|
Worth noting is also that I can get past these crashes when I restart the service, until it inevitably happens again. The amount of free memory is ~2GB at the time of crash. @diglos Did you get anywhere investigating GC issues? |
|
Hi
I'm currently doing another fast sync with 1.9.8 and already got a unallocated span error. Let's see how it ends but I think I will get the same errors. Yesterday I got a report from a Eth on ARM Raspi 4 user (same problem, I think). Please attached find his log. This doesn't happen at all on the NanoPC-T4 (4 GB RAM as well but ARM64) Regards! |
|
@diglos Yeah my Pi gets borked after a night of OOM crashes too, until it stops responding to SSH and needs a hard reboot |
|
Hi all. It seems that this is a 32bit kernel thing. There are lots of issues about out of memory and oom crashes on the Raspberry Pi forums [1]. Raspbian enabled recently a 64bit kernel for the Raspberry Pi 4 which seems to work great handling 4GB of RAM (this is just the kernel, the userspace is still on 32bit). Once enabled, I completed a Fast Sync and it’s been running for 1 day with no issues whatsoever (with 256MB cache). I’m releasing early next week a new image of Ethereum on ARM for the Raspberry Pi 4 with this kernel enabled by default. @moar, you can enable the 64bit kernel just by adding the proper option to config.txt (first, make sure you have the latest firmware): And reboot. This should fix the memory issues. [1] https://www.raspberrypi.org/forums/viewtopic.php?t=255433 |
still getting the error even with those fixes
|
|
Hi :-) It's been a while... We successfully synced a Raspberry Pi 4 running an Ubuntu 20.04 64 bit and we didn't get any crash or memory allocation problem. As we expected this seems to be a 32bit OS problem. @m0ar please, take a look at this: https://www.reddit.com/r/ethereum/comments/gf3nhg/ethereum_on_arm_raspberry_pi_4_images_release/ Regards! |
|
Thank you @diglos |
|
Hi @adamschmideg, just noting that the users moving to ARM64, but ARMv7 is still broken. This issue still persists and shouldn't be closed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
System information
Geth version: 1.9.6-stable
OS & Version: Linux 4.19.66-v7l+
Go: go1.13 linux/arm
Expected behaviour
Geth service syncs the chain.
Actual behaviour
Geth crashes repeatedly when service is started.
Steps to reproduce the behaviour
I'm not sure why this happens, but it did not appear during fast sync. When the node was (or almost) caught up with the chain, it started to crash repeatedly. Tried rebuilding geth today with the latest release (1.9.6), but crashes persist.
Backtrace
The text was updated successfully, but these errors were encountered: