Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

geth repeatedly crashes with SIGSEGV #20190

Open
m0ar opened this issue Oct 21, 2019 · 13 comments

Comments

@m0ar
Copy link

@m0ar m0ar commented Oct 21, 2019

System information

Geth version: 1.9.6-stable
OS & Version: Linux 4.19.66-v7l+
Go: go1.13 linux/arm

Expected behaviour

Geth service syncs the chain.

Actual behaviour

Geth crashes repeatedly when service is started.

Steps to reproduce the behaviour

I'm not sure why this happens, but it did not appear during fast sync. When the node was (or almost) caught up with the chain, it started to crash repeatedly. Tried rebuilding geth today with the latest release (1.9.6), but crashes persist.

Backtrace

Oct 21 11:33:04 evl-geth systemd[1]: Started Geth Node.
Oct 21 11:33:04 evl-geth geth[5793]: INFO [10-21|11:33:04.557] Maximum peer count                       ETH=50 LES=0 total=50
Oct 21 11:33:04 evl-geth geth[5793]: INFO [10-21|11:33:04.558] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
Oct 21 11:33:04 evl-geth geth[5793]: INFO [10-21|11:33:04.562] Starting peer-to-peer node               instance=Geth/v1.9.6-stable-bd059680/linux-arm/go1.13
Oct 21 11:33:04 evl-geth geth[5793]: INFO [10-21|11:33:04.563] Allocated trie memory caches             clean=64.00MiB dirty=64.00MiB
Oct 21 11:33:04 evl-geth geth[5793]: INFO [10-21|11:33:04.563] Allocated cache and file handles         database=/mnt/ssd/ethereum/geth/chaindata cache=128.00MiB handles=262144
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.000] Opened ancient database                  database=/mnt/ssd/ethereum/geth/chaindata/ancient
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.008] Initialised chain configuration          config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: <nil> Engine: ethash}"
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.008] Disk storage enabled for ethash caches   dir=/mnt/ssd/ethereum/geth/ethash count=3
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.008] Disk storage enabled for ethash DAGs     dir=/home/pi/.ethash              count=2
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.010] Initialising Ethereum protocol           versions=[63] network=1 dbversion=7
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.062] Loaded most recent local header          number=8714751 hash=398925…4639c7 td=12312815174212912772480 age=1w3d19h
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.062] Loaded most recent local full block      number=8714666 hash=6a936b…e922eb td=12312604899411814897379 age=1w3d19h
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.062] Loaded most recent local fast block      number=8714751 hash=398925…4639c7 td=12312815174212912772480 age=1w3d19h
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.064] Loaded local transaction journal         transactions=0 dropped=0
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.065] Regenerated local transaction journal    transactions=0 accounts=0
Oct 21 11:33:06 evl-geth geth[5793]: WARN [10-21|11:33:06.065] Switch sync mode from fast sync to full sync
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.159] New local node record                    seq=890 id=b0ae5886b9284f06 ip=127.0.0.1 udp=30303 tcp=30303
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.162] Started P2P networking                   self=enode://e64123ecdd615fa43ba43ff014f2802afd5b39c70d2b9b48d2a8358063bfecd171dadbf042fd8c08239257cfbc34c6b3fbe5f79021a395776e8ee3ab4f0bbc57@127.0.0.1:30303
Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.168] IPC endpoint opened                      url=/mnt/ssd/ethereum/geth.ipc
Oct 21 11:33:16 evl-geth geth[5793]: INFO [10-21|11:33:16.162] Block synchronisation started
Oct 21 11:33:17 evl-geth geth[5793]: panic: runtime error: invalid memory address or nil pointer dereference
Oct 21 11:33:17 evl-geth geth[5793]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x1b8 pc=0x63d828]
Oct 21 11:33:17 evl-geth geth[5793]: goroutine 263 [running]:
Oct 21 11:33:17 evl-geth geth[5793]: github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestor(0x24bac80, 0x2672480, 0x4a5d440, 0x0, 0x0, 0xa, 0xa)
Oct 21 11:33:17 evl-geth geth[5793]: #011/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:877 +0x1500
Oct 21 11:33:17 evl-geth geth[5793]: github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0x24bac80, 0x2672480, 0x5ca3af9d, 0x3c227bfb, 0xc13762f3, 0x5704518b, 0x7de03df6, 0xf0182a20, 0xc22975cd, 0x400256e9, ...)
Oct 21 11:33:17 evl-geth geth[5793]: #011/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:443 +0x2a0
Oct 21 11:33:17 evl-geth geth[5793]: github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0x24bac80, 0x86fe140, 0x10, 0x5ca3af9d, 0x3c227bfb, 0xc13762f3, 0x5704518b, 0x7de03df6, 0xf0182a20, 0xc22975cd, ...)
Oct 21 11:33:17 evl-geth geth[5793]: #011/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:411 +0x338
Oct 21 11:33:17 evl-geth geth[5793]: github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0x24bac80, 0x86fe140, 0x10, 0x5ca3af9d, 0x3c227bfb, 0xc13762f3, 0x5704518b, 0x7de03df6, 0xf0182a20, 0xc22975cd, ...)
Oct 21 11:33:17 evl-geth geth[5793]: #011/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:323 +0x5c
Oct 21 11:33:17 evl-geth geth[5793]: github.com/ethereum/go-ethereum/eth.(*ProtocolManager).synchronise(0x49a81b0, 0x8631500)
Oct 21 11:33:17 evl-geth geth[5793]: #011/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:190 +0x1b0
Oct 21 11:33:17 evl-geth geth[5793]: created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer
Oct 21 11:33:17 evl-geth geth[5793]: #011/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:155 +0x17c
Oct 21 11:33:17 evl-geth systemd[1]: geth.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 21 11:33:17 evl-geth systemd[1]: geth.service: Failed with result 'exit-code'.
@holiman

This comment has been minimized.

Copy link
Contributor

@holiman holiman commented Oct 21, 2019

Fast sync block is 8714751, full sync block is 8714666. Then it switches to full-sync mode, and tries to import blocks one by one...

Could you try it with some more detailed logging in place?

@m0ar

This comment has been minimized.

Copy link
Author

@m0ar m0ar commented Oct 21, 2019

@holiman Absolutely, here is the output from --verbosity 5:

Click to expand
pi@evl-geth:~/go-ethereum $ geth --verbosity 5 --syncmode fast --cache 256 --datadir /mnt/ssd/ethereum
DEBUG[10-21|14:53:24.978] Sanitizing Go's GC trigger               percent=100
INFO [10-21|14:53:24.981] Maximum peer count                       ETH=50 LES=0 total=50
INFO [10-21|14:53:24.982] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
DEBUG[10-21|14:53:24.982] FS scan times                            list=91.405µs set=2.741µs diff=4µs
TRACE[10-21|14:53:24.982] Started watching keystore folder         path=/mnt/ssd/ethereum/keystore
INFO [10-21|14:53:24.986] Starting peer-to-peer node               instance=Geth/v1.9.6-stable-bd059680/linux-arm/go1.13
INFO [10-21|14:53:24.986] Allocated trie memory caches             clean=64.00MiB dirty=64.00MiB
INFO [10-21|14:53:24.986] Allocated cache and file handles         database=/mnt/ssd/ethereum/geth/chaindata cache=128.00MiB handles=524288
DEBUG[10-21|14:53:26.418] Chain freezer table opened               database=/mnt/ssd/ethereum/geth/chaindata/ancient table=hashes items=8624751 size=263.21MiB
DEBUG[10-21|14:53:26.419] Chain freezer table opened               database=/mnt/ssd/ethereum/geth/chaindata/ancient table=bodies items=8624751 size=1.22GiB
DEBUG[10-21|14:53:26.419] Chain freezer table opened               database=/mnt/ssd/ethereum/geth/chaindata/ancient table=receipts items=8624751 size=1.73GiB
DEBUG[10-21|14:53:26.420] Chain freezer table opened               database=/mnt/ssd/ethereum/geth/chaindata/ancient table=diffs    items=8624751 size=83.48MiB
DEBUG[10-21|14:53:26.420] Chain freezer table opened               database=/mnt/ssd/ethereum/geth/chaindata/ancient table=headers  items=8624751 size=1.59GiB
INFO [10-21|14:53:26.420] Opened ancient database                  database=/mnt/ssd/ethereum/geth/chaindata/ancient
DEBUG[10-21|14:53:26.422] Ancient blocks frozen already            number=8714666 hash=6a936b…e922eb frozen=8624751
INFO [10-21|14:53:26.424] Initialised chain configuration          config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: <nil> Engine: ethash}"
INFO [10-21|14:53:26.424] Disk storage enabled for ethash caches   dir=/mnt/ssd/ethereum/geth/ethash count=3
INFO [10-21|14:53:26.424] Disk storage enabled for ethash DAGs     dir=/home/pi/.ethash              count=2
INFO [10-21|14:53:26.425] Initialising Ethereum protocol           versions=[63] network=1 dbversion=7
INFO [10-21|14:53:26.485] Loaded most recent local header          number=8714751 hash=398925…4639c7 td=12312815174212912772480 age=1w3d22h
INFO [10-21|14:53:26.485] Loaded most recent local full block      number=8714666 hash=6a936b…e922eb td=12312604899411814897379 age=1w3d23h
INFO [10-21|14:53:26.485] Loaded most recent local fast block      number=8714751 hash=398925…4639c7 td=12312815174212912772480 age=1w3d22h
DEBUG[10-21|14:53:26.487] Reinjecting stale transactions           count=0
INFO [10-21|14:53:26.487] Loaded local transaction journal         transactions=0 dropped=0
INFO [10-21|14:53:26.487] Regenerated local transaction journal    transactions=0 accounts=0
WARN [10-21|14:53:26.488] Switch sync mode from fast sync to full sync 
DEBUG[10-21|14:53:26.489] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
DEBUG[10-21|14:53:26.519] UDP listener up                          addr=[::]:30303
DEBUG[10-21|14:53:26.519] TCP listener up                          addr=[::]:30303
TRACE[10-21|14:53:26.569] Found seed node in database              id=15fd05f575b045c4 addr=45.77.242.229:30305 age=1h21m17.569187994s
INFO [10-21|14:53:26.572] New local node record                    seq=1350 id=b0ae5886b9284f06 ip=127.0.0.1 udp=30303 tcp=30303
TRACE[10-21|14:53:26.572] Found seed node in database              id=2135b4036f3c0ce6 addr=185.137.234.120:30303 age=1h29m42.572587546s
TRACE[10-21|14:53:26.572] Found seed node in database              id=27c4fbeca04fffa1 addr=45.77.187.182:30388   age=19m29.572800893s
TRACE[10-21|14:53:26.572] Found seed node in database              id=32d4407180141db5 addr=52.59.208.29:30303    age=4m18.572998185s
TRACE[10-21|14:53:26.573] Found seed node in database              id=42f90998419ffee0 addr=167.99.153.43:30303   age=3h21m34.573224476s
TRACE[10-21|14:53:26.573] Found seed node in database              id=47892be54ad19f4c addr=165.22.85.95:30303    age=1h46m22.57345523s
TRACE[10-21|14:53:26.573] Found seed node in database              id=4ab7d88b76997f33 addr=159.69.140.75:21226   age=6h50m12.573656336s
TRACE[10-21|14:53:26.573] Found seed node in database              id=4bf7b1616e7e3916 addr=167.71.216.34:30303   age=1h26m18.573848573s
TRACE[10-21|14:53:26.573] Found seed node in database              id=5b8494e9f8e1456b addr=103.29.71.224:28568   age=6h50m57.574037939s
TRACE[10-21|14:53:26.574] Found seed node in database              id=62c1f10a22bc411b addr=95.217.47.59:21236    age=1h52m24.574233527s
TRACE[10-21|14:53:26.574] Found seed node in database              id=6ffd7817bb47942b addr=80.158.38.240:30303   age=38m45.574427985s
TRACE[10-21|14:53:26.574] Found seed node in database              id=78bca176bd6061dc addr=34.237.26.75:30303    age=3h46m14.574655165s
TRACE[10-21|14:53:26.574] Found seed node in database              id=7ccb1a6b44086628 addr=68.183.181.212:30303  age=47m40.574847865s
TRACE[10-21|14:53:26.574] Found seed node in database              id=860f00e69799a87f addr=42.232.29.50:20182    age=64h27m57.57503986s
TRACE[10-21|14:53:26.575] Found seed node in database              id=8f1c6fc898f38a2d addr=174.138.22.71:30303   age=4h34m35.575235059s
TRACE[10-21|14:53:26.575] Found seed node in database              id=9359cec51ae10cdb addr=178.62.238.75:30303   age=2h8m52.575458221s
TRACE[10-21|14:53:26.575] Found seed node in database              id=9b51527bc7771288 addr=47.74.249.14:30303    age=5h30m54.57564805s
TRACE[10-21|14:53:26.575] Found seed node in database              id=a9f5120426e780ba addr=46.101.195.141:30303  age=1h22m20.57583899s
TRACE[10-21|14:53:26.576] Found seed node in database              id=aa2e2209dce8b6c8 addr=213.227.145.70:30303  age=46m54.576077596s
TRACE[10-21|14:53:26.576] Found seed node in database              id=b36064d9d880e159 addr=173.199.115.77:30303  age=6h39m54.576268054s
TRACE[10-21|14:53:26.576] Found seed node in database              id=bcf7c32b8bd78102 addr=54.92.202.189:30303   age=1h25m32.576446736s
TRACE[10-21|14:53:26.576] Found seed node in database              id=bfee2a3bd6b668e0 addr=104.251.214.135:30303 age=3h5m38.57665299s
TRACE[10-21|14:53:26.576] Found seed node in database              id=cd2162aa63111f2e addr=149.28.8.149:30303    age=6h45m39.576844023s
TRACE[10-21|14:53:26.577] Found seed node in database              id=d4aafc4523eb5646 addr=178.63.69.19:30306    age=2h22m35.577060629s
TRACE[10-21|14:53:26.577] Found seed node in database              id=d7237db08e3fc068 addr=52.28.245.66:21000    age=6h44m2.577246254s
TRACE[10-21|14:53:26.577] Found seed node in database              id=e131359c042cb4c4 addr=94.176.236.103:30303  age=1h38m34.577523044s
TRACE[10-21|14:53:26.577] Found seed node in database              id=edcab126072c2f95 addr=176.99.9.182:30334    age=3h46m49.577706985s
TRACE[10-21|14:53:26.577] Found seed node in database              id=eedd3ad43113da8f addr=35.240.35.207:30304   age=2h2m50.577893499s
TRACE[10-21|14:53:26.578] Found seed node in database              id=f1b907674c716bdd addr=75.152.97.59:30303    age=16m32.57807381s
TRACE[10-21|14:53:26.578] Found seed node in database              id=0c6253589a8d244d addr=104.42.217.25:30303   age=9m3.578265324s
TRACE[10-21|14:53:26.578] Found seed node in database              id=c845e51a5e470e44 addr=18.138.108.67:30303   age=3h45m42.578587853s
TRACE[10-21|14:53:26.578] Found seed node in database              id=f23ac6da7c02f84a addr=3.209.45.79:30303     age=3h45m46.578832663s
TRACE[10-21|14:53:26.578] Found seed node in database              id=71563f2370599df6 addr=34.255.23.113:30303   age=13m7.57907899s
TRACE[10-21|14:53:26.579] Found seed node in database              id=2d1f1ff277435247 addr=35.158.244.151:30303  age=3m29.579319503s
TRACE[10-21|14:53:26.579] Found seed node in database              id=1e1ea8b3d566d276 addr=52.187.207.27:30303   age=35m3.579655496s
TRACE[10-21|14:53:26.579] Found seed node in database              id=32f7e89d30e83b68 addr=191.234.162.198:30303 age=30m26.579969952s
TRACE[10-21|14:53:26.580] Found seed node in database              id=8d1da4c5bc682f8f addr=52.231.165.108:30303  age=6h44m30.580220798s
TRACE[10-21|14:53:26.580] Found seed node in database              id=0c6253589a8d244d addr=104.42.217.25:30303   age=9m3.580414108s
TRACE[10-21|14:53:26.580] Found seed node in database              id=29cca67d5db0fbb2 addr=5.1.83.226:30303      age=436573h53m26.580669213s
INFO [10-21|14:53:26.581] Started P2P networking                   self=enode://e64123ecdd615fa43ba43ff014f2802afd5b39c70d2b9b48d2a8358063bfecd171dadbf042fd8c08239257cfbc34c6b3fbe5f79021a395776e8ee3ab4f0bbc57@127.0.0.1:30303
TRACE[10-21|14:53:26.581] New dial task                            task="discovery lookup"
DEBUG[10-21|14:53:26.582] InProc registered                        namespace=admin
DEBUG[10-21|14:53:26.582] InProc registered                        namespace=admin
DEBUG[10-21|14:53:26.582] InProc registered                        namespace=debug
DEBUG[10-21|14:53:26.583] InProc registered                        namespace=web3
DEBUG[10-21|14:53:26.583] InProc registered                        namespace=eth
DEBUG[10-21|14:53:26.583] InProc registered                        namespace=eth
DEBUG[10-21|14:53:26.584] InProc registered                        namespace=eth
DEBUG[10-21|14:53:26.584] InProc registered                        namespace=txpool
DEBUG[10-21|14:53:26.584] InProc registered                        namespace=debug
DEBUG[10-21|14:53:26.584] InProc registered                        namespace=debug
DEBUG[10-21|14:53:26.584] InProc registered                        namespace=eth
DEBUG[10-21|14:53:26.585] InProc registered                        namespace=personal
DEBUG[10-21|14:53:26.585] InProc registered                        namespace=eth
DEBUG[10-21|14:53:26.585] InProc registered                        namespace=ethash
DEBUG[10-21|14:53:26.585] InProc registered                        namespace=eth
DEBUG[10-21|14:53:26.585] InProc registered                        namespace=eth
DEBUG[10-21|14:53:26.586] InProc registered                        namespace=eth
DEBUG[10-21|14:53:26.586] InProc registered                        namespace=miner
DEBUG[10-21|14:53:26.586] InProc registered                        namespace=eth
DEBUG[10-21|14:53:26.586] InProc registered                        namespace=admin
DEBUG[10-21|14:53:26.586] InProc registered                        namespace=debug
DEBUG[10-21|14:53:26.587] InProc registered                        namespace=debug
DEBUG[10-21|14:53:26.587] InProc registered                        namespace=net
DEBUG[10-21|14:53:26.587] IPC registered                           namespace=admin
DEBUG[10-21|14:53:26.588] IPC registered                           namespace=admin
DEBUG[10-21|14:53:26.588] IPC registered                           namespace=debug
DEBUG[10-21|14:53:26.588] IPC registered                           namespace=web3
DEBUG[10-21|14:53:26.588] IPC registered                           namespace=eth
DEBUG[10-21|14:53:26.588] IPC registered                           namespace=eth
DEBUG[10-21|14:53:26.589] IPC registered                           namespace=eth
DEBUG[10-21|14:53:26.589] IPC registered                           namespace=txpool
DEBUG[10-21|14:53:26.589] IPC registered                           namespace=debug
DEBUG[10-21|14:53:26.589] IPC registered                           namespace=debug
DEBUG[10-21|14:53:26.589] IPC registered                           namespace=eth
DEBUG[10-21|14:53:26.589] IPC registered                           namespace=personal
DEBUG[10-21|14:53:26.590] IPC registered                           namespace=eth
DEBUG[10-21|14:53:26.590] IPC registered                           namespace=ethash
DEBUG[10-21|14:53:26.590] IPC registered                           namespace=eth
DEBUG[10-21|14:53:26.590] IPC registered                           namespace=eth
DEBUG[10-21|14:53:26.590] IPC registered                           namespace=eth
DEBUG[10-21|14:53:26.590] IPC registered                           namespace=miner
DEBUG[10-21|14:53:26.591] IPC registered                           namespace=eth
DEBUG[10-21|14:53:26.591] IPC registered                           namespace=admin
DEBUG[10-21|14:53:26.591] IPC registered                           namespace=debug
DEBUG[10-21|14:53:26.591] IPC registered                           namespace=debug
DEBUG[10-21|14:53:26.591] IPC registered                           namespace=net
INFO [10-21|14:53:26.591] IPC endpoint opened                      url=/mnt/ssd/ethereum/geth.ipc
TRACE[10-21|14:53:26.593] >> FINDNODE/v4                           id=d4aafc4523eb5646 addr=178.63.69.19:30306    err=nil
TRACE[10-21|14:53:26.593] >> FINDNODE/v4                           id=c845e51a5e470e44 addr=18.138.108.67:30303   err=nil
TRACE[10-21|14:53:26.594] >> FINDNODE/v4                           id=d7237db08e3fc068 addr=52.28.245.66:21000    err=nil
TRACE[10-21|14:53:26.616] Found seed node in database              id=15fd05f575b045c4 addr=45.77.242.229:30305   age=1h21m17.61672728s
TRACE[10-21|14:53:26.616] Found seed node in database              id=1a1bf2b0678671a5 addr=5.39.18.8:30303       age=19m2.616883814s
TRACE[10-21|14:53:26.617] Found seed node in database              id=2135b4036f3c0ce6 addr=185.137.234.120:30303 age=1h29m42.617067513s
TRACE[10-21|14:53:26.617] Found seed node in database              id=27c4fbeca04fffa1 addr=45.77.187.182:30388   age=19m29.617200362s
TRACE[10-21|14:53:26.617] Found seed node in database              id=34df49bf45b4789c addr=167.114.153.134:30305 age=46m2.617343137s
TRACE[10-21|14:53:26.617] Found seed node in database              id=3859f96d7a948090 addr=47.75.251.64:30303    age=2h26m5.617477689s
TRACE[10-21|14:53:26.617] Found seed node in database              id=47892be54ad19f4c addr=165.22.85.95:30303    age=1h46m22.617615186s
TRACE[10-21|14:53:26.617] Found seed node in database              id=4ab7d88b76997f33 addr=159.69.140.75:21226   age=6h50m12.617745109s
TRACE[10-21|14:53:26.617] Found seed node in database              id=5b8494e9f8e1456b addr=103.29.71.224:28568   age=6h50m57.617853347s
TRACE[10-21|14:53:26.617] Found seed node in database              id=5f3e7d5be31c4bfa addr=5.9.28.9:30303        age=61h49m40.617962123s
TRACE[10-21|14:53:26.618] Found seed node in database              id=62c1f10a22bc411b addr=95.217.47.59:21236    age=1h52m24.61807799s
TRACE[10-21|14:53:26.618] Found seed node in database              id=6cac7a531cb10c55 addr=138.201.30.201:30303  age=6h51m8.618187414s
TRACE[10-21|14:53:26.618] Found seed node in database              id=7a3b129c12591094 addr=51.38.115.52:30303    age=34m42.618302393s
TRACE[10-21|14:53:26.618] Found seed node in database              id=83141c6224866e6a addr=93.189.47.189:30303   age=3h27m23.618459352s
TRACE[10-21|14:53:26.618] Found seed node in database              id=887e8f67b92e0550 addr=51.68.136.169:30303   age=6h49m16.618593534s
TRACE[10-21|14:53:26.618] Found seed node in database              id=945041c794fa52f6 addr=154.197.27.248:30303  age=1h15m42.618752864s
TRACE[10-21|14:53:26.618] Found seed node in database              id=94e7bb35e5cc4487 addr=165.22.16.119:30303   age=6h22m27.618868898s
TRACE[10-21|14:53:26.618] Found seed node in database              id=999d4a9364794c24 addr=172.93.189.174:30303  age=27m11.618989877s
TRACE[10-21|14:53:26.619] Found seed node in database              id=a62864a7d32f9860 addr=95.217.4.143:30303    age=3h16m20.619127226s
TRACE[10-21|14:53:26.619] Found seed node in database              id=ae9be3dbe71c5259 addr=31.13.217.52:30303    age=1h31m57.619261241s
TRACE[10-21|14:53:26.619] Found seed node in database              id=b43ec88b49748e65 addr=51.68.140.98:30303    age=38m9.619364869s
TRACE[10-21|14:53:26.619] Found seed node in database              id=ba118cf9b6072ae6 addr=18.224.2.102:30304    age=13m16.6194657s
TRACE[10-21|14:53:26.619] Found seed node in database              id=c0548e0d952703c0 addr=51.75.29.226:30303    age=6h34m53.619630696s
TRACE[10-21|14:53:26.619] Found seed node in database              id=c321dcf5ddaaa424 addr=88.99.164.186:21210   age=6h45m50.619767711s
TRACE[10-21|14:53:26.619] Found seed node in database              id=ce1d3ba7e9934f36 addr=54.37.85.202:30303    age=6h38m1.619884597s
TRACE[10-21|14:53:26.619] Found seed node in database              id=dc137e09cd3dcd0f addr=13.124.207.8:30303    age=6h44m55.620019742s
TRACE[10-21|14:53:26.620] Found seed node in database              id=e272f31a9ab9425a addr=157.245.94.240:30300  age=20m13.620132407s
TRACE[10-21|14:53:26.620] Found seed node in database              id=e70dedc66e915c77 addr=164.68.103.48:30303   age=59m27.620266848s
TRACE[10-21|14:53:26.620] Found seed node in database              id=e793c676acafd0a4 addr=207.246.124.59:30303  age=33m8.620370957s
TRACE[10-21|14:53:26.620] Found seed node in database              id=e939aa4210b366ec addr=144.217.68.211:30306  age=6h23m38.62049001s
TRACE[10-21|14:53:26.620] Found seed node in database              id=c845e51a5e470e44 addr=18.138.108.67:30303   age=3h45m42.620603322s
TRACE[10-21|14:53:26.620] Found seed node in database              id=f23ac6da7c02f84a addr=3.209.45.79:30303     age=3h45m46.62070132s
TRACE[10-21|14:53:26.619] << NEIGHBORS/v4                          id=d7237db08e3fc068 addr=52.28.245.66:21000    err=nil
TRACE[10-21|14:53:26.620] Found seed node in database              id=71563f2370599df6 addr=34.255.23.113:30303   age=13m7.620829206s
TRACE[10-21|14:53:26.620] Found seed node in database              id=2d1f1ff277435247 addr=35.158.244.151:30303  age=3m29.621050275s
TRACE[10-21|14:53:26.621] Found seed node in database              id=1e1ea8b3d566d276 addr=52.187.207.27:30303   age=35m3.621157568s
TRACE[10-21|14:53:26.621] Found seed node in database              id=32f7e89d30e83b68 addr=191.234.162.198:30303 age=30m26.621283528s
TRACE[10-21|14:53:26.621] Found seed node in database              id=8d1da4c5bc682f8f addr=52.231.165.108:30303  age=6h44m30.621400396s
TRACE[10-21|14:53:26.621] Found seed node in database              id=0c6253589a8d244d addr=104.42.217.25:30303   age=9m3.621520227s
TRACE[10-21|14:53:26.621] Found seed node in database              id=29cca67d5db0fbb2 addr=5.1.83.226:30303      age=436573h53m26.621633391s
TRACE[10-21|14:53:26.622] >> FINDNODE/v4                           id=ba118cf9b6072ae6 addr=18.224.2.102:30304    err=nil
TRACE[10-21|14:53:26.622] >> FINDNODE/v4                           id=b36064d9d880e159 addr=173.199.115.77:30303  err=nil
TRACE[10-21|14:53:26.623] << NEIGHBORS/v4                          id=d7237db08e3fc068 addr=52.28.245.66:21000    err=nil
TRACE[10-21|14:53:26.624] >> FINDNODE/v4                           id=b43ec88b49748e65 addr=51.68.140.98:30303    err=nil
TRACE[10-21|14:53:26.625] >> FINDNODE/v4                           id=d2a34a6e40eab598 addr=51.38.60.79:30303     err=nil
TRACE[10-21|14:53:26.640] << NEIGHBORS/v4                          id=d4aafc4523eb5646 addr=178.63.69.19:30306    err=nil
TRACE[10-21|14:53:26.642] << NEIGHBORS/v4                          id=d4aafc4523eb5646 addr=178.63.69.19:30306    err=nil
TRACE[10-21|14:53:26.643] >> PING/v4                               id=d29dc715c5f43d88 addr=67.205.142.63:50000   err=nil
TRACE[10-21|14:53:26.656] << NEIGHBORS/v4                          id=d2a34a6e40eab598 addr=51.38.60.79:30303     err=nil
TRACE[10-21|14:53:26.659] << NEIGHBORS/v4                          id=d2a34a6e40eab598 addr=51.38.60.79:30303     err=nil
TRACE[10-21|14:53:26.660] >> PING/v4                               id=d29bdc4db2e3dcb3 addr=182.61.108.243:21212  err=nil
TRACE[10-21|14:53:26.682] << NEIGHBORS/v4                          id=b43ec88b49748e65 addr=51.68.140.98:30303    err=nil
TRACE[10-21|14:53:26.685] << NEIGHBORS/v4                          id=b43ec88b49748e65 addr=51.68.140.98:30303    err=nil
TRACE[10-21|14:53:26.686] >> FINDNODE/v4                           id=b0ccc92109355e95 addr=140.112.28.132:30303  err=nil
TRACE[10-21|14:53:26.733] << NEIGHBORS/v4                          id=b36064d9d880e159 addr=173.199.115.77:30303  err=nil
TRACE[10-21|14:53:26.735] << NEIGHBORS/v4                          id=b36064d9d880e159 addr=173.199.115.77:30303  err=nil
TRACE[10-21|14:53:26.736] >> FINDNODE/v4                           id=b007271daaf38367 addr=52.36.69.215:30303    err=nil
TRACE[10-21|14:53:26.754] << NEIGHBORS/v4                          id=ba118cf9b6072ae6 addr=18.224.2.102:30304    err=nil
TRACE[10-21|14:53:26.756] << NEIGHBORS/v4                          id=ba118cf9b6072ae6 addr=18.224.2.102:30304    err=nil
TRACE[10-21|14:53:26.757] << PONG/v4                               id=1fa8969fb5d07f60 addr=67.205.142.63:50000   err="unsolicited reply"
TRACE[10-21|14:53:26.759] >> FINDNODE/v4                           id=b09d9f90cead8dcc addr=149.28.118.15:30303   err=nil
TRACE[10-21|14:53:26.850] << NEIGHBORS/v4                          id=c845e51a5e470e44 addr=18.138.108.67:30303   err=nil
TRACE[10-21|14:53:26.853] << NEIGHBORS/v4                          id=c845e51a5e470e44 addr=18.138.108.67:30303   err=nil
TRACE[10-21|14:53:26.854] >> PING/v4                               id=d29772b9b5f29c15 addr=128.199.41.168:50000  err=nil
TRACE[10-21|14:53:26.873] << PONG/v4                               id=40f17881667e7bff addr=128.199.41.168:50000  err="unsolicited reply"
TRACE[10-21|14:53:26.887] << NEIGHBORS/v4                          id=b09d9f90cead8dcc addr=149.28.118.15:30303   err=nil
TRACE[10-21|14:53:26.889] << NEIGHBORS/v4                          id=b09d9f90cead8dcc addr=149.28.118.15:30303   err=nil
TRACE[10-21|14:53:26.890] >> FINDNODE/v4                           id=b0b4b7b863af4a30 addr=35.171.255.98:30303   err=nil
TRACE[10-21|14:53:26.917] << PING/v4                               id=b007271daaf38367 addr=52.36.69.215:30303    err=nil
TRACE[10-21|14:53:26.918] >> PONG/v4                               id=b007271daaf38367 addr=52.36.69.215:30303    err=nil
TRACE[10-21|14:53:26.969] << NEIGHBORS/v4                          id=b0ccc92109355e95 addr=140.112.28.132:30303  err=nil
TRACE[10-21|14:53:26.971] << NEIGHBORS/v4                          id=b0ccc92109355e95 addr=140.112.28.132:30303  err=nil
TRACE[10-21|14:53:26.972] >> FINDNODE/v4                           id=b0a8a7fa57b40b1b addr=13.209.61.22:30303    err=nil
TRACE[10-21|14:53:26.998] << NEIGHBORS/v4                          id=b0b4b7b863af4a30 addr=35.171.255.98:30303   err=nil
TRACE[10-21|14:53:27.001] << NEIGHBORS/v4                          id=b0b4b7b863af4a30 addr=35.171.255.98:30303   err=nil
TRACE[10-21|14:53:27.002] >> FINDNODE/v4                           id=b0a6ffc3ac4688f3 addr=206.189.224.11:30305  err=nil
TRACE[10-21|14:53:27.104] << NEIGHBORS/v4                          id=b007271daaf38367 addr=52.36.69.215:30303    err=nil
TRACE[10-21|14:53:27.106] << NEIGHBORS/v4                          id=b007271daaf38367 addr=52.36.69.215:30303    err=nil
TRACE[10-21|14:53:27.112] << NEIGHBORS/v4                          id=b0a6ffc3ac4688f3 addr=206.189.224.11:30305  err=nil
TRACE[10-21|14:53:27.112] >> PING/v4                               id=b0a77c36e1e28893 addr=157.245.205.31:50000  err=nil
TRACE[10-21|14:53:27.114] << NEIGHBORS/v4                          id=b0a6ffc3ac4688f3 addr=206.189.224.11:30305  err=nil
TRACE[10-21|14:53:27.115] >> FINDNODE/v4                           id=b0a75525c39086c4 addr=104.214.217.23:30303  err=nil
TRACE[10-21|14:53:27.142] << NEIGHBORS/v4                          id=b0a75525c39086c4 addr=104.214.217.23:30303  err=nil
TRACE[10-21|14:53:27.145] << NEIGHBORS/v4                          id=b0a75525c39086c4 addr=104.214.217.23:30303  err=nil
TRACE[10-21|14:53:27.146] >> FINDNODE/v4                           id=b0a5ce1b3c3b951c addr=198.11.182.18:30303   err=nil
TRACE[10-21|14:53:27.283] << NEIGHBORS/v4                          id=b0a8a7fa57b40b1b addr=13.209.61.22:30303    err=nil
TRACE[10-21|14:53:27.285] << NEIGHBORS/v4                          id=b0a8a7fa57b40b1b addr=13.209.61.22:30303    err=nil
TRACE[10-21|14:53:27.286] >> PING/v4                               id=b0a3ba5318812bf7 addr=47.244.205.240:30303  err=nil
TRACE[10-21|14:53:27.397] << PONG/v4                               id=0cffad8b3bc8ee94 addr=157.245.205.31:50000  err="unsolicited reply"
TRACE[10-21|14:53:27.645] Findnode failed                          id=b0a5ce1b3c3b951c failcount=3 err="RPC timeout"
TRACE[10-21|14:53:27.645] >> FINDNODE/v4                           id=d29dc715c5f43d88 addr=67.205.142.63:50000   err=nil
TRACE[10-21|14:53:27.646] >> FINDNODE/v4                           id=b0bc0bae83bba53b addr=24.93.179.155:30303   err=nil
TRACE[10-21|14:53:27.661] >> FINDNODE/v4                           id=d29bdc4db2e3dcb3 addr=182.61.108.243:21212  err=nil
TRACE[10-21|14:53:27.811] << NEIGHBORS/v4                          id=b0bc0bae83bba53b addr=24.93.179.155:30303   err=nil
TRACE[10-21|14:53:27.818] << NEIGHBORS/v4                          id=b0bc0bae83bba53b addr=24.93.179.155:30303   err=nil
TRACE[10-21|14:53:27.820] >> FINDNODE/v4                           id=b0b35544b703ebff addr=95.168.186.250:30303  err=nil
TRACE[10-21|14:53:27.861] << NEIGHBORS/v4                          id=b0b35544b703ebff addr=95.168.186.250:30303  err=nil
TRACE[10-21|14:53:27.865] >> FINDNODE/v4                           id=d29772b9b5f29c15 addr=128.199.41.168:50000  err=nil
TRACE[10-21|14:53:27.867] << NEIGHBORS/v4                          id=b0b35544b703ebff addr=95.168.186.250:30303  err=nil
TRACE[10-21|14:53:27.869] >> PING/v4                               id=b0b04543ed500b0f addr=159.65.87.68:50000    err=nil
TRACE[10-21|14:53:27.899] << PONG/v4                               id=efe0b09dba278d6d addr=159.65.87.68:50000    err="unsolicited reply"
TRACE[10-21|14:53:28.113] >> FINDNODE/v4                           id=b0a77c36e1e28893 addr=157.245.205.31:50000  err=nil
TRACE[10-21|14:53:28.145] Findnode failed                          id=d29dc715c5f43d88 failcount=3 err="RPC timeout"
TRACE[10-21|14:53:28.147] >> PING/v4                               id=d296e4811d30d123 addr=164.15.10.134:1052    err=nil
TRACE[10-21|14:53:28.161] Findnode failed                          id=d29bdc4db2e3dcb3 failcount=4 err="RPC timeout"
TRACE[10-21|14:53:28.163] >> PING/v4                               id=d29587632448cb21 addr=167.99.82.0:50000     err=nil
TRACE[10-21|14:53:28.194] << PONG/v4                               id=828c8262be1d3c61 addr=167.99.82.0:50000     err="unsolicited reply"
TRACE[10-21|14:53:28.289] >> FINDNODE/v4                           id=b0a3ba5318812bf7 addr=47.244.205.240:30303  err=nil
TRACE[10-21|14:53:28.361] Findnode failed                          id=d29772b9b5f29c15 failcount=1 err="RPC timeout"
TRACE[10-21|14:53:28.363] >> PING/v4                               id=d294cb9bcfb89fce addr=168.61.161.73:30303   err=nil
TRACE[10-21|14:53:28.613] Findnode failed                          id=b0a77c36e1e28893 failcount=109 err="RPC timeout"
TRACE[10-21|14:53:28.613] Too many findnode failures, dropping     id=b0a77c36e1e28893 failcount=109
TRACE[10-21|14:53:28.616] >> FINDNODE/v4                           id=b08e5ca5007081f8 addr=95.216.34.99:20954    err=nil
DEBUG[10-21|14:53:28.619] Couldn't add port mapping                proto=tcp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
DEBUG[10-21|14:53:28.620] Couldn't add port mapping                proto=udp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
TRACE[10-21|14:53:28.647] << NEIGHBORS/v4                          id=b08e5ca5007081f8 addr=95.216.34.99:20954    err=nil
TRACE[10-21|14:53:28.654] << NEIGHBORS/v4                          id=b08e5ca5007081f8 addr=95.216.34.99:20954    err=nil
TRACE[10-21|14:53:28.656] >> FINDNODE/v4                           id=b08dd798c20605f9 addr=159.100.245.37:30303  err=nil
TRACE[10-21|14:53:28.709] << NEIGHBORS/v4                          id=b08dd798c20605f9 addr=159.100.245.37:30303  err=nil
TRACE[10-21|14:53:28.715] << NEIGHBORS/v4                          id=b08dd798c20605f9 addr=159.100.245.37:30303  err=nil
TRACE[10-21|14:53:28.717] >> PING/v4                               id=b0ae133fa9174778 addr=138.68.252.162:50000  err=nil
TRACE[10-21|14:53:28.787] Findnode failed                          id=b0a3ba5318812bf7 failcount=24  err="RPC timeout"
TRACE[10-21|14:53:28.787] Too many findnode failures, dropping     id=b0a3ba5318812bf7 failcount=24
TRACE[10-21|14:53:28.789] >> PING/v4                               id=b0aa8e13085a58f2 addr=157.245.206.138:50000 err=nil
TRACE[10-21|14:53:28.871] >> FINDNODE/v4                           id=b0b04543ed500b0f addr=159.65.87.68:50000    err=nil
TRACE[10-21|14:53:28.894] << PONG/v4                               id=ba1435cac4cef010 addr=138.68.252.162:50000  err="unsolicited reply"
TRACE[10-21|14:53:29.079] << PONG/v4                               id=29e22c15c08a6782 addr=157.245.206.138:50000 err="unsolicited reply"
TRACE[10-21|14:53:29.149] >> FINDNODE/v4                           id=d296e4811d30d123 addr=164.15.10.134:1052    err=nil
TRACE[10-21|14:53:29.164] >> FINDNODE/v4                           id=d29587632448cb21 addr=167.99.82.0:50000     err=nil
TRACE[10-21|14:53:29.365] >> FINDNODE/v4                           id=d294cb9bcfb89fce addr=168.61.161.73:30303   err=nil
TRACE[10-21|14:53:29.370] Findnode failed                          id=b0b04543ed500b0f failcount=27  err="RPC timeout"
TRACE[10-21|14:53:29.370] Too many findnode failures, dropping     id=b0b04543ed500b0f failcount=27
TRACE[10-21|14:53:29.372] >> PING/v4                               id=b0a854822e6e3ffe addr=164.15.10.134:1123    err=nil
TRACE[10-21|14:53:29.649] Findnode failed                          id=d296e4811d30d123 failcount=4   err="RPC timeout"
TRACE[10-21|14:53:29.651] >> PING/v4                               id=d293e23a0074d53f addr=159.89.20.82:30303    err=nil
TRACE[10-21|14:53:29.664] Findnode failed                          id=d29587632448cb21 failcount=2   err="RPC timeout"
TRACE[10-21|14:53:29.667] >> PING/v4                               id=d29377beb3841c83 addr=167.99.12.22:50000    err=nil
TRACE[10-21|14:53:29.675] << PONG/v4                               id=d293e23a0074d53f addr=159.89.20.82:30303    err=nil
TRACE[10-21|14:53:29.676] Accepted connection                      addr=31.164.124.43:40026
TRACE[10-21|14:53:29.719] >> FINDNODE/v4                           id=b0ae133fa9174778 addr=138.68.252.162:50000  err=nil
TRACE[10-21|14:53:29.774] << PONG/v4                               id=eb8686c8abc21417 addr=167.99.12.22:50000    err="unsolicited reply"
DEBUG[10-21|14:53:29.774] Adding p2p peer                          id=3b7062ace89b8232 conn=inbound addr=31.164.124.43:40026   peers=1 name=Geth/v1.9.3-unstable...
TRACE[10-21|14:53:29.774] Connection set up                        id=3b7062ace89b8232 addr=31.164.124.43:40026   conn=inbound inbound=true
TRACE[10-21|14:53:29.775] Starting protocol eth/63                 id=3b7062ace89b8232 conn=inbound
DEBUG[10-21|14:53:29.775] Ethereum peer connected                  id=3b7062ace89b8232 conn=inbound name=Geth/v1.9.3-unstable-a978adfd-20190826/linux-amd64/go1.9.3
DEBUG[10-21|14:53:29.776] Ethereum handshake failed                id=3b7062ace89b8232 conn=inbound err="Genesis block mismatch - b726b36f6d811b33 (!= d4e56740f876aef8)"
TRACE[10-21|14:53:29.776] Protocol eth/63 failed                   id=3b7062ace89b8232 conn=inbound err="Genesis block mismatch - b726b36f6d811b33 (!= d4e56740f876aef8)"
DEBUG[10-21|14:53:29.778] Removing p2p peer                        id=3b7062ace89b8232 conn=inbound addr=31.164.124.43:40026   peers=0 duration=4.110ms req=false err="Genesis block mismatch - b726b36f6d811b33 (!= d4e56740f876aef8)"
TRACE[10-21|14:53:29.791] >> FINDNODE/v4                           id=b0aa8e13085a58f2 addr=157.245.206.138:50000 err=nil
TRACE[10-21|14:53:29.864] Findnode failed                          id=d294cb9bcfb89fce failcount=2   err="RPC timeout"
TRACE[10-21|14:53:29.867] >> PING/v4                               id=d292d0bd601d724b addr=128.199.35.72:50000   err=nil
TRACE[10-21|14:53:29.887] << PONG/v4                               id=d83b157765b00b81 addr=128.199.35.72:50000   err="unsolicited reply"
TRACE[10-21|14:53:30.177] >> FINDNODE/v4                           id=d293e23a0074d53f addr=159.89.20.82:30303    err=nil
TRACE[10-21|14:53:30.201] << PING/v4                               id=d293e23a0074d53f addr=159.89.20.82:30303    err=nil
TRACE[10-21|14:53:30.203] >> PONG/v4                               id=d293e23a0074d53f addr=159.89.20.82:30303    err=nil
TRACE[10-21|14:53:30.218] Findnode failed                          id=b0ae133fa9174778 failcount=601 err="RPC timeout"
TRACE[10-21|14:53:30.218] Too many findnode failures, dropping     id=b0ae133fa9174778 failcount=601
TRACE[10-21|14:53:30.220] >> PING/v4                               id=b0a8ba2f2fc227e7 addr=178.62.40.90:50000    err=nil
TRACE[10-21|14:53:30.241] << NEIGHBORS/v4                          id=d293e23a0074d53f addr=159.89.20.82:30303    err=nil
TRACE[10-21|14:53:30.245] Invalid neighbor node received           ip=157.230.121.200 addr=159.89.20.82:30303    err="low port"
TRACE[10-21|14:53:30.246] << NEIGHBORS/v4                          id=d293e23a0074d53f addr=159.89.20.82:30303    err=nil
DEBUG[10-21|14:53:30.246] IP exceeds bucket limit                  ip=164.15.10.134
TRACE[10-21|14:53:30.249] >> FINDNODE/v4                           id=d29f82f0669d35b3 addr=164.15.10.134:1450    err=nil
TRACE[10-21|14:53:30.250] << PONG/v4                               id=624beed8fbe560c7 addr=178.62.40.90:50000    err="unsolicited reply"
TRACE[10-21|14:53:30.290] Accepted connection                      addr=175.126.163.8:39436
TRACE[10-21|14:53:30.291] Findnode failed                          id=b0aa8e13085a58f2 failcount=537 err="RPC timeout"
TRACE[10-21|14:53:30.291] Too many findnode failures, dropping     id=b0aa8e13085a58f2 failcount=537
TRACE[10-21|14:53:30.296] >> PING/v4                               id=b0a51f21dd698b51 addr=13.126.85.24:30303    err=nil
TRACE[10-21|14:53:30.374] >> FINDNODE/v4                           id=b0a854822e6e3ffe addr=164.15.10.134:1123    err=nil
DEBUG[10-21|14:53:30.606] Adding p2p peer                          id=8f112bd35cc4e73c conn=inbound addr=175.126.163.8:39436   peers=1 name=Geth/v1.8.23-stable-...
TRACE[10-21|14:53:30.607] Connection set up                        id=8f112bd35cc4e73c addr=175.126.163.8:39436   conn=inbound inbound=true
TRACE[10-21|14:53:30.607] Starting protocol eth/63                 id=8f112bd35cc4e73c conn=inbound
DEBUG[10-21|14:53:30.608] Ethereum peer connected                  id=8f112bd35cc4e73c conn=inbound name=Geth/v1.8.23-stable-c9427004/linux-amd64/go1.11.5
TRACE[10-21|14:53:30.610] Registering sync peer                    peer=8f112bd35cc4e73c
DEBUG[10-21|14:53:30.610] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8552447 skip=0 reverse=false
TRACE[10-21|14:53:30.668] >> FINDNODE/v4                           id=d29377beb3841c83 addr=167.99.12.22:50000    err=nil
TRACE[10-21|14:53:30.748] Findnode failed                          id=d29f82f0669d35b3 failcount=1   err="RPC timeout"
TRACE[10-21|14:53:30.750] >> PING/v4                               id=d290414037655377 addr=157.245.206.68:50000  err=nil
TRACE[10-21|14:53:30.869] >> FINDNODE/v4                           id=d292d0bd601d724b addr=128.199.35.72:50000   err=nil
TRACE[10-21|14:53:30.873] Findnode failed                          id=b0a854822e6e3ffe failcount=509 err="RPC timeout"
TRACE[10-21|14:53:30.873] Too many findnode failures, dropping     id=b0a854822e6e3ffe failcount=509
TRACE[10-21|14:53:30.876] >> PING/v4                               id=b0a0fb98bf2e3b3e addr=67.205.142.63:50000   err=nil
TRACE[10-21|14:53:30.982] << PONG/v4                               id=1fa8969fb5d07f60 addr=67.205.142.63:50000   err="unsolicited reply"
TRACE[10-21|14:53:31.038] << PONG/v4                               id=ddabfe72c4aeef0f addr=157.245.206.68:50000  err="unsolicited reply"
TRACE[10-21|14:53:31.167] Findnode failed                          id=d29377beb3841c83 failcount=2   err="RPC timeout"
TRACE[10-21|14:53:31.169] >> PING/v4                               id=d28f4d17a1daf965 addr=164.15.10.134:1370    err=nil
TRACE[10-21|14:53:31.222] >> FINDNODE/v4                           id=b0a8ba2f2fc227e7 addr=178.62.40.90:50000    err=nil
TRACE[10-21|14:53:31.298] >> FINDNODE/v4                           id=b0a51f21dd698b51 addr=13.126.85.24:30303    err=nil
TRACE[10-21|14:53:31.368] Findnode failed                          id=d292d0bd601d724b failcount=1   err="RPC timeout"
TRACE[10-21|14:53:31.370] >> PING/v4                               id=d28d9aa3da17090e addr=78.41.194.15:43026    err=nil
TRACE[10-21|14:53:31.721] Findnode failed                          id=b0a8ba2f2fc227e7 failcount=596 err="RPC timeout"
TRACE[10-21|14:53:31.721] Too many findnode failures, dropping     id=b0a8ba2f2fc227e7 failcount=596
TRACE[10-21|14:53:31.724] >> PING/v4                               id=b0b47c10f130cd71 addr=13.114.151.77:30303   err=nil
TRACE[10-21|14:53:31.752] >> FINDNODE/v4                           id=d290414037655377 addr=157.245.206.68:50000  err=nil
TRACE[10-21|14:53:31.798] Findnode failed                          id=b0a51f21dd698b51 failcount=743 err="RPC timeout"
TRACE[10-21|14:53:31.798] Too many findnode failures, dropping     id=b0a51f21dd698b51 failcount=743
TRACE[10-21|14:53:31.878] >> FINDNODE/v4                           id=b0a0fb98bf2e3b3e addr=67.205.142.63:50000   err=nil
TRACE[10-21|14:53:32.170] >> FINDNODE/v4                           id=d28f4d17a1daf965 addr=164.15.10.134:1370    err=nil
TRACE[10-21|14:53:32.251] Findnode failed                          id=d290414037655377 failcount=4   err="RPC timeout"
TRACE[10-21|14:53:32.252] >> PING/v4                               id=d28d1c78c8340e36 addr=39.108.164.135:30303  err=nil
TRACE[10-21|14:53:32.371] >> FINDNODE/v4                           id=d28d9aa3da17090e addr=78.41.194.15:43026    err=nil
TRACE[10-21|14:53:32.377] Findnode failed                          id=b0a0fb98bf2e3b3e failcount=572 err="RPC timeout"
TRACE[10-21|14:53:32.377] Too many findnode failures, dropping     id=b0a0fb98bf2e3b3e failcount=572
TRACE[10-21|14:53:32.670] Findnode failed                          id=d28f4d17a1daf965 failcount=3   err="RPC timeout"
TRACE[10-21|14:53:32.671] >> PING/v4                               id=d28918e9b337c0b0 addr=85.250.222.230:30303  err=nil
TRACE[10-21|14:53:32.724] >> FINDNODE/v4                           id=b0b47c10f130cd71 addr=13.114.151.77:30303   err=nil
TRACE[10-21|14:53:32.790] Accepted connection                      addr=13.125.4.161:59046
TRACE[10-21|14:53:32.871] Findnode failed                          id=d28d9aa3da17090e failcount=12  err="RPC timeout"
TRACE[10-21|14:53:32.871] Too many findnode failures, dropping     id=d28d9aa3da17090e failcount=12
TRACE[10-21|14:53:32.872] >> PING/v4                               id=d28677e1d08b5cbd addr=47.56.185.214:30303   err=nil
DEBUG[10-21|14:53:33.099] Adding p2p peer                          id=649eb641681bc33b conn=inbound addr=13.125.4.161:59046    peers=2 name=Geth/v1.9.4-unstable...
TRACE[10-21|14:53:33.100] Connection set up                        id=649eb641681bc33b addr=13.125.4.161:59046    conn=inbound inbound=true
TRACE[10-21|14:53:33.100] Starting protocol eth/63                 id=649eb641681bc33b conn=inbound
DEBUG[10-21|14:53:33.100] Ethereum peer connected                  id=649eb641681bc33b conn=inbound name=Geth/v1.9.4-unstable-b4a4a4db-20190903/linux-amd64/go1.12.7
DEBUG[10-21|14:53:33.101] Ethereum handshake failed                id=649eb641681bc33b conn=inbound err="Genesis block mismatch - 71e0e1fcf9967616 (!= d4e56740f876aef8)"
TRACE[10-21|14:53:33.101] Protocol eth/63 failed                   id=649eb641681bc33b conn=inbound err="Genesis block mismatch - 71e0e1fcf9967616 (!= d4e56740f876aef8)"
DEBUG[10-21|14:53:33.103] Removing p2p peer                        id=649eb641681bc33b conn=inbound addr=13.125.4.161:59046    peers=1 duration=3.470ms req=false err="Genesis block mismatch - 71e0e1fcf9967616 (!= d4e56740f876aef8)"
TRACE[10-21|14:53:33.224] Findnode failed                          id=b0b47c10f130cd71 failcount=440 err="RPC timeout"
TRACE[10-21|14:53:33.224] Too many findnode failures, dropping     id=b0b47c10f130cd71 failcount=440
TRACE[10-21|14:53:33.227] >> FINDNODE/v4                           id=7ccb1a6b44086628 addr=68.183.181.212:30303  err=nil
TRACE[10-21|14:53:33.227] >> FINDNODE/v4                           id=71563f2370599df6 addr=34.255.23.113:30303   err=nil
TRACE[10-21|14:53:33.229] >> FINDNODE/v4                           id=78bca176bd6061dc addr=34.237.26.75:30303    err=nil
TRACE[10-21|14:53:33.255] >> FINDNODE/v4                           id=d28d1c78c8340e36 addr=39.108.164.135:30303  err=nil
TRACE[10-21|14:53:33.295] << NEIGHBORS/v4                          id=71563f2370599df6 addr=34.255.23.113:30303   err=nil
TRACE[10-21|14:53:33.301] << NEIGHBORS/v4                          id=71563f2370599df6 addr=34.255.23.113:30303   err=nil
TRACE[10-21|14:53:33.304] >> FINDNODE/v4                           id=72d48f27cd1ee7f6 addr=54.68.121.166:20203   err=nil
TRACE[10-21|14:53:33.367] << NEIGHBORS/v4                          id=78bca176bd6061dc addr=34.237.26.75:30303    err=nil
TRACE[10-21|14:53:33.373] << NEIGHBORS/v4                          id=78bca176bd6061dc addr=34.237.26.75:30303    err=nil
TRACE[10-21|14:53:33.376] >> FINDNODE/v4                           id=72a321369de5e8fb addr=183.82.123.102:30304  err=nil
TRACE[10-21|14:53:33.460] >> PING/v4                               id=999d4a9364794c24 addr=172.93.189.174:30303  err=nil
TRACE[10-21|14:53:33.497] << NEIGHBORS/v4                          id=72d48f27cd1ee7f6 addr=54.68.121.166:20203   err=nil
TRACE[10-21|14:53:33.504] << NEIGHBORS/v4                          id=72d48f27cd1ee7f6 addr=54.68.121.166:20203   err=nil
TRACE[10-21|14:53:33.507] >> PING/v4                               id=72fb90e2e8cc8697 addr=35.158.55.23:30303    err=nil
TRACE[10-21|14:53:33.531] << NEIGHBORS/v4                          id=72a321369de5e8fb addr=183.82.123.102:30304  err=nil
TRACE[10-21|14:53:33.538] << NEIGHBORS/v4                          id=72a321369de5e8fb addr=183.82.123.102:30304  err=nil
TRACE[10-21|14:53:33.540] >> PING/v4                               id=72f8970d203a3324 addr=74.125.182.17:64744   err=nil
TRACE[10-21|14:53:33.552] << NEIGHBORS/v4                          id=7ccb1a6b44086628 addr=68.183.181.212:30303  err=nil
TRACE[10-21|14:53:33.559] << NEIGHBORS/v4                          id=7ccb1a6b44086628 addr=68.183.181.212:30303  err=nil
TRACE[10-21|14:53:33.561] << PONG/v4                               id=72fb90e2e8cc8697 addr=35.158.55.23:30303    err=nil
TRACE[10-21|14:53:33.561] >> PING/v4                               id=72fe469d4a04c525 addr=107.172.208.100:30303 err=nil
TRACE[10-21|14:53:33.576] Accepted connection                      addr=35.224.108.27:33680
TRACE[10-21|14:53:33.602] Accepted connection                      addr=147.135.210.87:37144
DEBUG[10-21|14:53:33.668] Adding p2p peer                          id=b5549983afa1810a conn=inbound addr=147.135.210.87:37144  peers=2 name=Geth/v1.8.13-unstabl...
TRACE[10-21|14:53:33.669] Connection set up                        id=b5549983afa1810a addr=147.135.210.87:37144  conn=inbound inbound=true
TRACE[10-21|14:53:33.669] Starting protocol eth/63                 id=b5549983afa1810a conn=inbound
DEBUG[10-21|14:53:33.670] Ethereum peer connected                  id=b5549983afa1810a conn=inbound name=Geth/v1.8.13-unstable-3b074515/linux-amd64/go1.10.3
DEBUG[10-21|14:53:33.672] Ethereum handshake failed                id=b5549983afa1810a conn=inbound err="Genesis block mismatch - 325d96e6a3950ee7 (!= d4e56740f876aef8)"
TRACE[10-21|14:53:33.672] Protocol eth/63 failed                   id=b5549983afa1810a conn=inbound err="Genesis block mismatch - 325d96e6a3950ee7 (!= d4e56740f876aef8)"
DEBUG[10-21|14:53:33.673] Removing p2p peer                        id=b5549983afa1810a conn=inbound addr=147.135.210.87:37144  peers=1 duration=4.425ms req=false err="Genesis block mismatch - 325d96e6a3950ee7 (!= d4e56740f876aef8)"
TRACE[10-21|14:53:33.673] >> FINDNODE/v4                           id=d28918e9b337c0b0 addr=85.250.222.230:30303  err=nil
TRACE[10-21|14:53:33.687] << PONG/v4                               id=999d4a9364794c24 addr=172.93.189.174:30303  err=nil
DEBUG[10-21|14:53:33.687] Revalidated node                         b=14 id=999d4a9364794c24 checks=1
DEBUG[10-21|14:53:33.707] Adding p2p peer                          id=918bd3c3bcc5586f conn=inbound addr=35.224.108.27:33680   peers=2 name=Parity-Ethereum/v2.5...
TRACE[10-21|14:53:33.707] Connection set up                        id=918bd3c3bcc5586f addr=35.224.108.27:33680   conn=inbound inbound=true
TRACE[10-21|14:53:33.707] Starting protocol eth/63                 id=918bd3c3bcc5586f conn=inbound
DEBUG[10-21|14:53:33.707] Ethereum peer connected                  id=918bd3c3bcc5586f conn=inbound name=Parity-Ethereum/v2.5.8-stable-c52a6c8-20190916/x86_64-linux-gnu/rustc1.36.0
TRACE[10-21|14:53:33.753] Findnode failed                          id=d28d1c78c8340e36 failcount=14  err="RPC timeout"
TRACE[10-21|14:53:33.753] Too many findnode failures, dropping     id=d28d1c78c8340e36 failcount=14
TRACE[10-21|14:53:33.824] Registering sync peer                    peer=918bd3c3bcc5586f
DEBUG[10-21|14:53:33.824] Relaxed downloader QoS values            rtt=20s confidence=0.500 ttl=1m0s
DEBUG[10-21|14:53:33.824] Fetching batch of headers                id=918bd3c3bcc5586f conn=inbound count=1 fromnum=8552447 skip=0 reverse=false
TRACE[10-21|14:53:33.875] >> FINDNODE/v4                           id=d28677e1d08b5cbd addr=47.56.185.214:30303   err=nil
TRACE[10-21|14:53:33.895] Accepted connection                      addr=18.197.97.197:56940
DEBUG[10-21|14:53:33.938] Adding p2p peer                          id=d501e73fc2b6ba8e conn=inbound addr=18.197.97.197:56940   peers=3 name=Geth/v1.8.23-stable-...
TRACE[10-21|14:53:33.939] Connection set up                        id=d501e73fc2b6ba8e addr=18.197.97.197:56940   conn=inbound inbound=true
TRACE[10-21|14:53:33.939] Starting protocol eth/63                 id=d501e73fc2b6ba8e conn=inbound
DEBUG[10-21|14:53:33.939] Ethereum peer connected                  id=d501e73fc2b6ba8e conn=inbound name=Geth/v1.8.23-stable-c9427004/linux-amd64/go1.10.4
TRACE[10-21|14:53:33.944] Registering sync peer                    peer=d501e73fc2b6ba8e
DEBUG[10-21|14:53:33.944] Relaxed downloader QoS values            rtt=20s confidence=0.333 ttl=1m0s
DEBUG[10-21|14:53:33.945] Fetching batch of headers                id=d501e73fc2b6ba8e conn=inbound count=1 fromnum=8552447 skip=0 reverse=false
DEBUG[10-21|14:53:33.980] Failed to deliver headers                err="no sync active"
TRACE[10-21|14:53:34.063] >> FINDNODE/v4                           id=72fb90e2e8cc8697 addr=35.158.55.23:30303    err=nil
TRACE[10-21|14:53:34.160] << PING/v4                               id=72fb90e2e8cc8697 addr=35.158.55.23:30303    err=nil
TRACE[10-21|14:53:34.161] >> PONG/v4                               id=72fb90e2e8cc8697 addr=35.158.55.23:30303    err=nil
TRACE[10-21|14:53:34.172] Findnode failed                          id=d28918e9b337c0b0 failcount=3   err="RPC timeout"
TRACE[10-21|14:53:34.341] << NEIGHBORS/v4                          id=72fb90e2e8cc8697 addr=35.158.55.23:30303    err=nil
TRACE[10-21|14:53:34.347] << NEIGHBORS/v4                          id=72fb90e2e8cc8697 addr=35.158.55.23:30303    err=nil
TRACE[10-21|14:53:34.350] >> PING/v4                               id=72f8b65a77a01c7f addr=18.130.136.41:30303   err=nil
TRACE[10-21|14:53:34.374] Findnode failed                          id=d28677e1d08b5cbd failcount=3   err="RPC timeout"
TRACE[10-21|14:53:34.374] Dial task done                           task="discovery lookup (16 results)"
TRACE[10-21|14:53:34.375] New dial task                            task="dyndial d29f82f0669d35b3 164.15.10.134:1450"
TRACE[10-21|14:53:34.375] New dial task                            task="dyndial d29dc715c5f43d88 67.205.142.63:50000"
TRACE[10-21|14:53:34.375] New dial task                            task="dyndial d29bdc4db2e3dcb3 182.61.108.243:21212"
TRACE[10-21|14:53:34.375] New dial task                            task="dyndial d29772b9b5f29c15 128.199.41.168:50000"
TRACE[10-21|14:53:34.375] New dial task                            task="dyndial d296e4811d30d123 164.15.10.134:1052"
TRACE[10-21|14:53:34.375] New dial task                            task="dyndial d29587632448cb21 167.99.82.0:50000"
TRACE[10-21|14:53:34.376] New dial task                            task="dyndial d294cb9bcfb89fce 168.61.161.73:30303"
TRACE[10-21|14:53:34.376] New dial task                            task="dyndial d293e23a0074d53f 159.89.20.82:30303"
TRACE[10-21|14:53:34.376] New dial task                            task="dyndial d29377beb3841c83 167.99.12.22:50000"
TRACE[10-21|14:53:34.376] New dial task                            task="dyndial d292d0bd601d724b 128.199.35.72:50000"
TRACE[10-21|14:53:34.376] New dial task                            task="dyndial d290414037655377 157.245.206.68:50000"
TRACE[10-21|14:53:34.376] New dial task                            task="dyndial d28f4d17a1daf965 164.15.10.134:1370"
TRACE[10-21|14:53:34.376] New dial task                            task="dyndial d28d9aa3da17090e 78.41.194.15:43026"
TRACE[10-21|14:53:34.376] New dial task                            task="dyndial d28d1c78c8340e36 39.108.164.135:30303"
TRACE[10-21|14:53:34.377] New dial task                            task="dyndial d28918e9b337c0b0 85.250.222.230:30303"
TRACE[10-21|14:53:34.377] New dial task                            task="dyndial d28677e1d08b5cbd 47.56.185.214:30303"
TRACE[10-21|14:53:34.542] >> FINDNODE/v4                           id=72f8970d203a3324 addr=74.125.182.17:64744   err=nil
TRACE[10-21|14:53:34.552] Dial error                               task="dyndial d29772b9b5f29c15 128.199.41.168:50000" err="dial tcp 128.199.41.168:50000: connect: connection refused"
TRACE[10-21|14:53:34.552] Dial task done                           task="dyndial d29772b9b5f29c15 128.199.41.168:50000"
TRACE[10-21|14:53:34.553] New dial task                            task="discovery lookup"
TRACE[10-21|14:53:34.555] >> FINDNODE/v4                           id=6ffd7817bb47942b addr=80.158.38.240:30303   err=nil
TRACE[10-21|14:53:34.557] >> FINDNODE/v4                           id=62c1f10a22bc411b addr=95.217.47.59:21236    err=nil
TRACE[10-21|14:53:34.558] Dial error                               task="dyndial d292d0bd601d724b 128.199.35.72:50000" err="dial tcp 128.199.35.72:50000: connect: connection refused"
TRACE[10-21|14:53:34.558] Dial task done                           task="dyndial d292d0bd601d724b 128.199.35.72:50000"
TRACE[10-21|14:53:34.558] >> FINDNODE/v4                           id=7ccb1a6b44086628 addr=68.183.181.212:30303  err=nil
TRACE[10-21|14:53:34.563] >> FINDNODE/v4                           id=72fe469d4a04c525 addr=107.172.208.100:30303 err=nil
TRACE[10-21|14:53:34.565] Dial error                               task="dyndial d29587632448cb21 167.99.82.0:50000" err="dial tcp 167.99.82.0:50000: connect: connection refused"
TRACE[10-21|14:53:34.565] Dial task done                           task="dyndial d29587632448cb21 167.99.82.0:50000"
TRACE[10-21|14:53:34.642] Dial error                               task="dyndial d29377beb3841c83 167.99.12.22:50000" err="dial tcp 167.99.12.22:50000: connect: connection refused"
TRACE[10-21|14:53:34.642] Dial task done                           task="dyndial d29377beb3841c83 167.99.12.22:50000"
TRACE[10-21|14:53:34.644] Dial error                               task="dyndial d29dc715c5f43d88 67.205.142.63:50000" err="dial tcp 67.205.142.63:50000: connect: connection refused"
TRACE[10-21|14:53:34.644] Dial task done                           task="dyndial d29dc715c5f43d88 67.205.142.63:50000"
TRACE[10-21|14:53:34.645] Dial error                               task="dyndial d28918e9b337c0b0 85.250.222.230:30303" err="dial tcp 85.250.222.230:30303: connect: connection refused"
TRACE[10-21|14:53:34.645] Dial task done                           task="dyndial d28918e9b337c0b0 85.250.222.230:30303"
TRACE[10-21|14:53:34.655] Accepted connection                      addr=86.107.243.88:57700
TRACE[10-21|14:53:34.662] Dial error                               task="dyndial d290414037655377 157.245.206.68:50000" err="dial tcp 157.245.206.68:50000: connect: connection refused"
TRACE[10-21|14:53:34.663] Dial task done                           task="dyndial d290414037655377 157.245.206.68:50000"
TRACE[10-21|14:53:34.666] << NEIGHBORS/v4                          id=62c1f10a22bc411b addr=95.217.47.59:21236    err=nil
TRACE[10-21|14:53:34.673] << NEIGHBORS/v4                          id=62c1f10a22bc411b addr=95.217.47.59:21236    err=nil
DEBUG[10-21|14:53:34.675] Adding p2p peer                          id=d293e23a0074d53f conn=dyndial addr=159.89.20.82:30303    peers=4 name=Parity-Ethereum/v2.3...
TRACE[10-21|14:53:34.676] Connection set up                        id=d293e23a0074d53f addr=159.89.20.82:30303    conn=dyndial inbound=false
TRACE[10-21|14:53:34.676] Starting protocol eth/63                 id=d293e23a0074d53f conn=dyndial
DEBUG[10-21|14:53:34.676] Ethereum peer connected                  id=d293e23a0074d53f conn=dyndial name=Parity-Ethereum/v2.3.4-beta-0e95db1-20190220/x86_64-linux-gnu/rustc1.32.0
TRACE[10-21|14:53:34.677] >> FINDNODE/v4                           id=6dbec632440b9806 addr=164.68.121.126:30303  err=nil
TRACE[10-21|14:53:34.676] Dial task done                           task="dyndial d293e23a0074d53f 159.89.20.82:30303"
DEBUG[10-21|14:53:34.682] Ethereum handshake failed                id=d293e23a0074d53f conn=dyndial err=EOF
DEBUG[10-21|14:53:34.683] Removing p2p peer                        id=d293e23a0074d53f conn=dyndial addr=159.89.20.82:30303    peers=3 duration=7.619ms req=true  err="too many peers"
TRACE[10-21|14:53:34.691] Invalid neighbor node received           ip=39.100.242.121  addr=80.158.38.240:30303   err="low port"
TRACE[10-21|14:53:34.692] << NEIGHBORS/v4                          id=6ffd7817bb47942b addr=80.158.38.240:30303   err=nil
TRACE[10-21|14:53:34.698] << NEIGHBORS/v4                          id=6ffd7817bb47942b addr=80.158.38.240:30303   err=nil
DEBUG[10-21|14:53:34.698] IP exceeds bucket limit                  ip=128.199.36.156
TRACE[10-21|14:53:34.701] >> PING/v4                               id=6de702fafe659188 addr=178.130.44.248:30303  err=nil
TRACE[10-21|14:53:34.788] Invalid neighbor node received           ip=178.130.44.248  addr=164.68.121.126:30303  err="low port"
TRACE[10-21|14:53:34.793] Invalid neighbor node received           ip=39.100.242.121  addr=164.68.121.126:30303  err="low port"
TRACE[10-21|14:53:34.799] << NEIGHBORS/v4                          id=6dbec632440b9806 addr=164.68.121.126:30303  err=nil
TRACE[10-21|14:53:34.804] << NEIGHBORS/v4                          id=6dbec632440b9806 addr=164.68.121.126:30303  err=nil
TRACE[10-21|14:53:34.807] >> PING/v4                               id=6de7d6b8d8cf03dc addr=141.20.21.166:34677   err=nil
TRACE[10-21|14:53:34.868] << NEIGHBORS/v4                          id=7ccb1a6b44086628 addr=68.183.181.212:30303  err=nil
TRACE[10-21|14:53:34.874] << NEIGHBORS/v4                          id=7ccb1a6b44086628 addr=68.183.181.212:30303  err=nil
TRACE[10-21|14:53:34.877] >> PING/v4                               id=6de26d5a9c625384 addr=128.199.36.156:50000  err=nil
TRACE[10-21|14:53:34.887] Rejected peer                            id=d29bdc4db2e3dcb3 addr=182.61.108.243:21212  conn=dyndial err="useless peer"
TRACE[10-21|14:53:34.888] Setting up connection failed             addr=182.61.108.243:21212  err="useless peer"
TRACE[10-21|14:53:34.888] Dial error                               task="dyndial d29bdc4db2e3dcb3 182.61.108.243:21212" err="useless peer"
TRACE[10-21|14:53:34.888] Dial task done                           task="dyndial d29bdc4db2e3dcb3 182.61.108.243:21212"
TRACE[10-21|14:53:35.042] Findnode failed                          id=72f8970d203a3324 failcount=1   err="RPC timeout"
TRACE[10-21|14:53:35.044] >> PING/v4                               id=72ff347f38bb4e88 addr=54.236.62.205:30303   err=nil
TRACE[10-21|14:53:35.062] Findnode failed                          id=72fe469d4a04c525 failcount=9   err="RPC timeout"
TRACE[10-21|14:53:35.063] Too many findnode failures, dropping     id=72fe469d4a04c525 failcount=9
TRACE[10-21|14:53:35.066] >> PING/v4                               id=72fd17f82748ab4b addr=138.68.250.147:50000  err=nil
TRACE[10-21|14:53:35.237] << PONG/v4                               id=6cd2a37407a80222 addr=138.68.250.147:50000  err="unsolicited reply"
TRACE[10-21|14:53:35.352] >> FINDNODE/v4                           id=72f8b65a77a01c7f addr=18.130.136.41:30303   err=nil
TRACE[10-21|14:53:35.703] >> FINDNODE/v4                           id=6de702fafe659188 addr=178.130.44.248:30303  err=nil
TRACE[10-21|14:53:35.808] >> FINDNODE/v4                           id=6de7d6b8d8cf03dc addr=141.20.21.166:34677   err=nil
TRACE[10-21|14:53:35.851] Findnode failed                          id=72f8b65a77a01c7f failcount=2   err="RPC timeout"
TRACE[10-21|14:53:35.854] >> PING/v4                               id=72f2c59ed464e9f6 addr=167.99.12.22:50000    err=nil
TRACE[10-21|14:53:35.880] >> FINDNODE/v4                           id=6de26d5a9c625384 addr=128.199.36.156:50000  err=nil
TRACE[10-21|14:53:35.959] << PONG/v4                               id=eb8686c8abc21417 addr=167.99.12.22:50000    err="unsolicited reply"
TRACE[10-21|14:53:36.046] >> FINDNODE/v4                           id=72ff347f38bb4e88 addr=54.236.62.205:30303   err=nil
TRACE[10-21|14:53:36.068] >> FINDNODE/v4                           id=72fd17f82748ab4b addr=138.68.250.147:50000  err=nil
TRACE[10-21|14:53:36.202] Findnode failed                          id=6de702fafe659188 failcount=4   err="RPC timeout"
TRACE[10-21|14:53:36.204] >> PING/v4                               id=6de32e35712c8605 addr=128.199.36.156:50000  err=nil
TRACE[10-21|14:53:36.308] Findnode failed                          id=6de7d6b8d8cf03dc failcount=1   err="RPC timeout"
TRACE[10-21|14:53:36.310] >> PING/v4                               id=6de35c272c6b3bd4 addr=167.71.61.184:30303   err=nil
TRACE[10-21|14:53:36.378] Findnode failed                          id=6de26d5a9c625384 failcount=1   err="RPC timeout"
TRACE[10-21|14:53:36.380] >> PING/v4                               id=6de04dbe582d08c9 addr=164.15.10.134:1467    err=nil
TRACE[10-21|14:53:36.545] Findnode failed                          id=72ff347f38bb4e88 failcount=3   err="RPC timeout"
TRACE[10-21|14:53:36.548] >> FINDNODE/v4                           id=72f03b9acc589a72 addr=178.238.225.47:30303  err=nil
TRACE[10-21|14:53:36.567] Findnode failed                          id=72fd17f82748ab4b failcount=3   err="RPC timeout"
TRACE[10-21|14:53:36.569] >> PING/v4                               id=72f7a4f06b1c4b6a addr=34.250.50.222:30303   err=nil
INFO [10-21|14:53:36.582] Block synchronisation started 
DEBUG[10-21|14:53:36.582] Synchronising with the network           peer=8f112bd35cc4e73c eth=63 head=a9aa9f…401aba td=12483987806983869307610 mode=full
DEBUG[10-21|14:53:36.582] Retrieving remote chain height           peer=8f112bd35cc4e73c
DEBUG[10-21|14:53:36.582] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1 fromhash=a9aa9f…401aba skip=0 reverse=false
TRACE[10-21|14:53:36.591] << NEIGHBORS/v4                          id=72f03b9acc589a72 addr=178.238.225.47:30303  err=nil
TRACE[10-21|14:53:36.597] << NEIGHBORS/v4                          id=72f03b9acc589a72 addr=178.238.225.47:30303  err=nil
TRACE[10-21|14:53:36.600] >> PING/v4                               id=72f2b457c05aad17 addr=128.199.35.72:50000   err=nil
TRACE[10-21|14:53:36.636] >> PING/v4                               id=b7cfb51b7122692d addr=144.76.162.156:30303  err=nil
TRACE[10-21|14:53:36.651] << PONG/v4                               id=d83b157765b00b81 addr=128.199.35.72:50000   err="unsolicited reply"
TRACE[10-21|14:53:36.683] << PONG/v4                               id=b7cfb51b7122692d addr=144.76.162.156:30303  err=nil
DEBUG[10-21|14:53:36.683] Revalidated node                         b=11 id=b7cfb51b7122692d checks=1
TRACE[10-21|14:53:36.855] >> FINDNODE/v4                           id=72f2c59ed464e9f6 addr=167.99.12.22:50000    err=nil
TRACE[10-21|14:53:36.881] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:36.881] Remote head header identified            peer=8f112bd35cc4e73c number=8784300 hash=a9aa9f…401aba
DEBUG[10-21|14:53:36.881] Looking for common ancestor              peer=8f112bd35cc4e73c local=8714666 remote=8784300
TRACE[10-21|14:53:36.882] Span searching for common ancestor       peer=8f112bd35cc4e73c count=12 from=8784123 skip=15
DEBUG[10-21|14:53:36.882] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=12 fromnum=8784123 skip=15 reverse=false
TRACE[10-21|14:53:37.200] Binary searching for common ancestor     peer=8f112bd35cc4e73c start=8624666 end=8784300
DEBUG[10-21|14:53:37.201] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8704483 skip=0  reverse=false
TRACE[10-21|14:53:37.206] >> FINDNODE/v4                           id=6de32e35712c8605 addr=128.199.36.156:50000  err=nil
TRACE[10-21|14:53:37.312] >> FINDNODE/v4                           id=6de35c272c6b3bd4 addr=167.71.61.184:30303   err=nil
TRACE[10-21|14:53:37.355] Findnode failed                          id=72f2c59ed464e9f6 failcount=1   err="RPC timeout"
TRACE[10-21|14:53:37.357] >> PING/v4                               id=72f76c90ea44991f addr=94.79.55.28:30000     err=nil
TRACE[10-21|14:53:37.382] >> FINDNODE/v4                           id=6de04dbe582d08c9 addr=164.15.10.134:1467    err=nil
TRACE[10-21|14:53:37.500] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:37.507] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8744391 skip=0  reverse=false
TRACE[10-21|14:53:37.572] >> FINDNODE/v4                           id=72f7a4f06b1c4b6a addr=34.250.50.222:30303   err=nil
TRACE[10-21|14:53:37.601] >> FINDNODE/v4                           id=72f2b457c05aad17 addr=128.199.35.72:50000   err=nil
TRACE[10-21|14:53:37.708] Findnode failed                          id=6de32e35712c8605 failcount=2   err="RPC timeout"
TRACE[10-21|14:53:37.711] >> PING/v4                               id=6dee1df50008fa13 addr=128.199.36.156:50000  err=nil
TRACE[10-21|14:53:37.806] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:37.807] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8724437 skip=0  reverse=false
TRACE[10-21|14:53:37.810] Findnode failed                          id=6de35c272c6b3bd4 failcount=12  err="RPC timeout"
TRACE[10-21|14:53:37.811] Too many findnode failures, dropping     id=6de35c272c6b3bd4 failcount=12
TRACE[10-21|14:53:37.813] >> PING/v4                               id=6defe48b406759cf addr=157.245.206.138:50000 err=nil
TRACE[10-21|14:53:37.845] Accepted connection                      addr=40.68.81.47:34440
TRACE[10-21|14:53:37.881] Findnode failed                          id=6de04dbe582d08c9 failcount=3   err="RPC timeout"
DEBUG[10-21|14:53:37.882] Adding p2p peer                          id=d69a9fec3befba00 conn=inbound addr=40.68.81.47:34440     peers=4 name=Geth/v1.8.27-stable-...
TRACE[10-21|14:53:37.882] Connection set up                        id=d69a9fec3befba00 addr=40.68.81.47:34440     conn=inbound inbound=true
TRACE[10-21|14:53:37.882] Starting protocol eth/63                 id=d69a9fec3befba00 conn=inbound
DEBUG[10-21|14:53:37.883] Ethereum peer connected                  id=d69a9fec3befba00 conn=inbound name=Geth/v1.8.27-stable-4bcc0a37/linux-amd64/go1.11.9
TRACE[10-21|14:53:37.884] >> FINDNODE/v4                           id=6deb29c84e2b3758 addr=188.166.90.213:30303  err=nil
DEBUG[10-21|14:53:37.885] Ethereum handshake failed                id=d69a9fec3befba00 conn=inbound err="Genesis block mismatch - 33477ffd19d181cb (!= d4e56740f876aef8)"
TRACE[10-21|14:53:37.885] Protocol eth/63 failed                   id=d69a9fec3befba00 conn=inbound err="Genesis block mismatch - 33477ffd19d181cb (!= d4e56740f876aef8)"
DEBUG[10-21|14:53:37.886] Removing p2p peer                        id=d69a9fec3befba00 conn=inbound addr=40.68.81.47:34440     peers=3 duration=4.344ms req=false err="Genesis block mismatch - 33477ffd19d181cb (!= d4e56740f876aef8)"
TRACE[10-21|14:53:37.919] << NEIGHBORS/v4                          id=6deb29c84e2b3758 addr=188.166.90.213:30303  err=nil
TRACE[10-21|14:53:37.925] << NEIGHBORS/v4                          id=6deb29c84e2b3758 addr=188.166.90.213:30303  err=nil
TRACE[10-21|14:53:37.928] >> PING/v4                               id=6deb2fa8db66d98c addr=176.9.155.139:30303   err=nil
TRACE[10-21|14:53:38.070] Findnode failed                          id=72f7a4f06b1c4b6a failcount=8   err="RPC timeout"
TRACE[10-21|14:53:38.070] Too many findnode failures, dropping     id=72f7a4f06b1c4b6a failcount=8
TRACE[10-21|14:53:38.072] >> PING/v4                               id=72f5a6e13d21d938 addr=93.57.86.57:37111     err=nil
TRACE[10-21|14:53:38.101] Findnode failed                          id=72f2b457c05aad17 failcount=1   err="RPC timeout"
TRACE[10-21|14:53:38.102] >> PING/v4                               id=72f52953ec145c5d addr=178.62.235.175:50000  err=nil
TRACE[10-21|14:53:38.109] << PONG/v4                               id=29e22c15c08a6782 addr=157.245.206.138:50000 err="unsolicited reply"
TRACE[10-21|14:53:38.216] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:38.217] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8714460 skip=0  reverse=false
TRACE[10-21|14:53:38.358] >> FINDNODE/v4                           id=72f76c90ea44991f addr=94.79.55.28:30000     err=nil
TRACE[10-21|14:53:38.429] Failed RLPx handshake                    addr=86.107.243.88:57700   conn=inbound err=EOF
TRACE[10-21|14:53:38.429] Setting up connection failed             addr=86.107.243.88:57700   err=EOF
TRACE[10-21|14:53:38.534] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:38.542] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8719448 skip=0  reverse=false
TRACE[10-21|14:53:38.712] >> FINDNODE/v4                           id=6dee1df50008fa13 addr=128.199.36.156:50000  err=nil
TRACE[10-21|14:53:38.815] >> FINDNODE/v4                           id=6defe48b406759cf addr=157.245.206.138:50000 err=nil
TRACE[10-21|14:53:38.841] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:38.842] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8716954 skip=0  reverse=false
TRACE[10-21|14:53:38.858] Findnode failed                          id=72f76c90ea44991f failcount=1   err="RPC timeout"
TRACE[10-21|14:53:38.860] >> PING/v4                               id=72eb9a8da7036fb8 addr=24.125.193.190:30303  err=nil
TRACE[10-21|14:53:38.930] >> FINDNODE/v4                           id=6deb2fa8db66d98c addr=176.9.155.139:30303   err=nil
TRACE[10-21|14:53:39.074] >> FINDNODE/v4                           id=72f5a6e13d21d938 addr=93.57.86.57:37111     err=nil
TRACE[10-21|14:53:39.104] >> FINDNODE/v4                           id=72f52953ec145c5d addr=178.62.235.175:50000  err=nil
TRACE[10-21|14:53:39.143] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:39.144] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8715707 skip=0  reverse=false
TRACE[10-21|14:53:39.212] Findnode failed                          id=6dee1df50008fa13 failcount=2   err="RPC timeout"
TRACE[10-21|14:53:39.214] >> PING/v4                               id=6df725320987e763 addr=178.62.40.90:50000    err=nil
TRACE[10-21|14:53:39.314] Findnode failed                          id=6defe48b406759cf failcount=2   err="RPC timeout"
TRACE[10-21|14:53:39.316] >> PING/v4                               id=6df50697c60bc573 addr=178.62.27.60:50000    err=nil
TRACE[10-21|14:53:39.352] << PONG/v4                               id=c0fffc49cb1a4b89 addr=178.62.27.60:50000    err="unsolicited reply"
TRACE[10-21|14:53:39.429] Findnode failed                          id=6deb2fa8db66d98c failcount=12  err="RPC timeout"
TRACE[10-21|14:53:39.429] Too many findnode failures, dropping     id=6deb2fa8db66d98c failcount=12
TRACE[10-21|14:53:39.431] >> PING/v4                               id=6df2383c819b9098 addr=128.199.55.156:50000  err=nil
TRACE[10-21|14:53:39.454] << PONG/v4                               id=bc9e903f92ae4958 addr=128.199.55.156:50000  err="unsolicited reply"
TRACE[10-21|14:53:39.481] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:39.488] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8716330 skip=0  reverse=false
TRACE[10-21|14:53:39.573] Findnode failed                          id=72f5a6e13d21d938 failcount=6   err="RPC timeout"
TRACE[10-21|14:53:39.573] Too many findnode failures, dropping     id=72f5a6e13d21d938 failcount=6
TRACE[10-21|14:53:39.576] >> PING/v4                               id=72ef0f14b4b58e13 addr=35.195.106.62:20758   err=nil
TRACE[10-21|14:53:39.603] Findnode failed                          id=72f52953ec145c5d failcount=2   err="RPC timeout"
TRACE[10-21|14:53:39.605] >> PING/v4                               id=72ec039292142ee2 addr=206.189.157.63:30303  err=nil
TRACE[10-21|14:53:39.795] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:39.796] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8716018 skip=0  reverse=false
TRACE[10-21|14:53:39.862] >> FINDNODE/v4                           id=72eb9a8da7036fb8 addr=24.125.193.190:30303  err=nil
TRACE[10-21|14:53:40.094] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:40.100] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8716174 skip=0  reverse=false
TRACE[10-21|14:53:40.216] >> FINDNODE/v4                           id=6df725320987e763 addr=178.62.40.90:50000    err=nil
TRACE[10-21|14:53:40.318] >> FINDNODE/v4                           id=6df50697c60bc573 addr=178.62.27.60:50000    err=nil
TRACE[10-21|14:53:40.361] Findnode failed                          id=72eb9a8da7036fb8 failcount=3   err="RPC timeout"
TRACE[10-21|14:53:40.399] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:40.400] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8716096 skip=0  reverse=false
TRACE[10-21|14:53:40.433] >> FINDNODE/v4                           id=6df2383c819b9098 addr=128.199.55.156:50000  err=nil
TRACE[10-21|14:53:40.578] >> FINDNODE/v4                           id=72ef0f14b4b58e13 addr=35.195.106.62:20758   err=nil
TRACE[10-21|14:53:40.607] >> FINDNODE/v4                           id=72ec039292142ee2 addr=206.189.157.63:30303  err=nil
TRACE[10-21|14:53:40.715] Findnode failed                          id=6df725320987e763 failcount=5   err="RPC timeout"
TRACE[10-21|14:53:40.715] Too many findnode failures, dropping     id=6df725320987e763 failcount=5
TRACE[10-21|14:53:40.717] >> PING/v4                               id=6dfad945754fe00b addr=13.233.201.54:23000   err=nil
TRACE[10-21|14:53:40.731] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:40.732] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8716057 skip=0  reverse=false
TRACE[10-21|14:53:40.770] << PING/v4                               id=b0a6ffc3ac4688f3 addr=206.189.224.11:30305  err=nil
TRACE[10-21|14:53:40.772] >> PONG/v4                               id=b0a6ffc3ac4688f3 addr=206.189.224.11:30305  err=nil
TRACE[10-21|14:53:40.817] Findnode failed                          id=6df50697c60bc573 failcount=5   err="RPC timeout"
TRACE[10-21|14:53:40.817] Too many findnode failures, dropping     id=6df50697c60bc573 failcount=5
TRACE[10-21|14:53:40.828] >> PING/v4                               id=6dc60a052cc54396 addr=35.226.157.25:30303   err=nil
TRACE[10-21|14:53:40.830] << PING/v4                               id=b0b35544b703ebff addr=95.168.186.250:30303  err=nil
TRACE[10-21|14:53:40.832] >> PONG/v4                               id=b0b35544b703ebff addr=95.168.186.250:30303  err=nil
TRACE[10-21|14:53:40.858] << ENRREQUEST/v4                         id=b0b35544b703ebff addr=95.168.186.250:30303  err=nil
TRACE[10-21|14:53:40.860] >> ENRRESPONSE/v4                        id=b0b35544b703ebff addr=95.168.186.250:30303  err=nil
TRACE[10-21|14:53:40.932] Findnode failed                          id=6df2383c819b9098 failcount=5   err="RPC timeout"
TRACE[10-21|14:53:40.932] Too many findnode failures, dropping     id=6df2383c819b9098 failcount=5
TRACE[10-21|14:53:40.934] >> FINDNODE/v4                           id=6dc6cf8757bad9c8 addr=54.162.153.64:30303   err=nil
TRACE[10-21|14:53:41.031] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:41.032] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8716037 skip=0  reverse=false
TRACE[10-21|14:53:41.042] << NEIGHBORS/v4                          id=252cecb6dfba92ae addr=54.162.153.64:30303   err="unsolicited reply"
TRACE[10-21|14:53:41.045] << NEIGHBORS/v4                          id=252cecb6dfba92ae addr=54.162.153.64:30303   err="unsolicited reply"
TRACE[10-21|14:53:41.077] Findnode failed                          id=72ef0f14b4b58e13 failcount=9   err="RPC timeout"
TRACE[10-21|14:53:41.077] Too many findnode failures, dropping     id=72ef0f14b4b58e13 failcount=9
TRACE[10-21|14:53:41.106] Findnode failed                          id=72ec039292142ee2 failcount=1   err="RPC timeout"
TRACE[10-21|14:53:41.109] >> PING/v4                               id=860f00e69799a87f addr=42.232.29.50:20182    err=nil
TRACE[10-21|14:53:41.109] >> FINDNODE/v4                           id=887e8f67b92e0550 addr=51.68.136.169:30303   err=nil
TRACE[10-21|14:53:41.110] >> FINDNODE/v4                           id=83141c6224866e6a addr=93.189.47.189:30303   err=nil
TRACE[10-21|14:53:41.178] << NEIGHBORS/v4                          id=887e8f67b92e0550 addr=51.68.136.169:30303   err=nil
TRACE[10-21|14:53:41.181] << NEIGHBORS/v4                          id=887e8f67b92e0550 addr=51.68.136.169:30303   err=nil
TRACE[10-21|14:53:41.182] >> FINDNODE/v4                           id=80807c1c7b15253e addr=45.76.62.76:30305     err=nil
TRACE[10-21|14:53:41.195] Accepted connection                      addr=206.189.224.11:49234
TRACE[10-21|14:53:41.247] << NEIGHBORS/v4                          id=83141c6224866e6a addr=93.189.47.189:30303   err=nil
TRACE[10-21|14:53:41.250] << NEIGHBORS/v4                          id=83141c6224866e6a addr=93.189.47.189:30303   err=nil
TRACE[10-21|14:53:41.251] >> FINDNODE/v4                           id=80ebec7d03036f90 addr=138.201.248.31:30303  err=nil
TRACE[10-21|14:53:41.300] << NEIGHBORS/v4                          id=80ebec7d03036f90 addr=138.201.248.31:30303  err=nil
TRACE[10-21|14:53:41.303] << NEIGHBORS/v4                          id=80ebec7d03036f90 addr=138.201.248.31:30303  err=nil
TRACE[10-21|14:53:41.304] >> PING/v4                               id=80fe7bee1d8dc690 addr=145.239.133.36:30303  err=nil
DEBUG[10-21|14:53:41.305] Adding p2p peer                          id=b0a6ffc3ac4688f3 conn=inbound addr=206.189.224.11:49234  peers=4 name=Geth/v0.0.9.1-Ether1...
TRACE[10-21|14:53:41.305] Connection set up                        id=b0a6ffc3ac4688f3 addr=206.189.224.11:49234  conn=inbound inbound=true
TRACE[10-21|14:53:41.305] Starting protocol eth/63                 id=b0a6ffc3ac4688f3 conn=inbound
DEBUG[10-21|14:53:41.306] Ethereum peer connected                  id=b0a6ffc3ac4688f3 conn=inbound name=Geth/v0.0.9.1-Ether1_MN_SN-Security-Update/linux-amd64/go1.11.5
DEBUG[10-21|14:53:41.307] Ethereum handshake failed                id=b0a6ffc3ac4688f3 conn=inbound err="Genesis block mismatch - 70cefc67ff52eb3e (!= d4e56740f876aef8)"
TRACE[10-21|14:53:41.307] Protocol eth/63 failed                   id=b0a6ffc3ac4688f3 conn=inbound err="Genesis block mismatch - 70cefc67ff52eb3e (!= d4e56740f876aef8)"
DEBUG[10-21|14:53:41.308] Removing p2p peer                        id=b0a6ffc3ac4688f3 conn=inbound addr=206.189.224.11:49234  peers=3 duration=2.347ms req=false err="Genesis block mismatch - 70cefc67ff52eb3e (!= d4e56740f876aef8)"
TRACE[10-21|14:53:41.309] << NEIGHBORS/v4                          id=80807c1c7b15253e addr=45.76.62.76:30305     err=nil
TRACE[10-21|14:53:41.312] << NEIGHBORS/v4                          id=80807c1c7b15253e addr=45.76.62.76:30305     err=nil
TRACE[10-21|14:53:41.313] >> PING/v4                               id=80ff25e95839d737 addr=123.118.62.70:20182   err=nil
TRACE[10-21|14:53:41.331] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:41.332] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8716047 skip=0  reverse=false
TRACE[10-21|14:53:41.433] Findnode failed                          id=6dc6cf8757bad9c8 failcount=3   err="RPC timeout"
TRACE[10-21|14:53:41.631] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:41.631] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8716042 skip=0  reverse=false
TRACE[10-21|14:53:41.718] >> FINDNODE/v4                           id=6dfad945754fe00b addr=13.233.201.54:23000   err=nil
TRACE[10-21|14:53:41.829] >> FINDNODE/v4                           id=6dc60a052cc54396 addr=35.226.157.25:30303   err=nil
TRACE[10-21|14:53:41.977] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:41.978] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8716039 skip=0  reverse=false
TRACE[10-21|14:53:42.111] >> FINDNODE/v4                           id=860f00e69799a87f addr=42.232.29.50:20182    err=nil
TRACE[10-21|14:53:42.218] Findnode failed                          id=6dfad945754fe00b failcount=9   err="RPC timeout"
TRACE[10-21|14:53:42.218] Too many findnode failures, dropping     id=6dfad945754fe00b failcount=9
TRACE[10-21|14:53:42.276] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:42.279] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8716040 skip=0  reverse=false
TRACE[10-21|14:53:42.306] >> FINDNODE/v4                           id=80fe7bee1d8dc690 addr=145.239.133.36:30303  err=nil
TRACE[10-21|14:53:42.315] >> FINDNODE/v4                           id=80ff25e95839d737 addr=123.118.62.70:20182   err=nil
TRACE[10-21|14:53:42.329] Findnode failed                          id=6dc60a052cc54396 failcount=7   err="RPC timeout"
TRACE[10-21|14:53:42.329] Too many findnode failures, dropping     id=6dc60a052cc54396 failcount=7
TRACE[10-21|14:53:42.330] Dial task done                           task="discovery lookup (16 results)"
TRACE[10-21|14:53:42.330] New dial task                            task="dyndial 6de702fafe659188 178.130.44.248:30303"
TRACE[10-21|14:53:42.331] New dial task                            task="dyndial 6de7d6b8d8cf03dc 141.20.21.166:34677"
TRACE[10-21|14:53:42.331] New dial task                            task="dyndial 6de26d5a9c625384 128.199.36.156:50000"
TRACE[10-21|14:53:42.332] New dial task                            task="dyndial 6de32e35712c8605 128.199.36.156:50000"
TRACE[10-21|14:53:42.332] New dial task                            task="dyndial 6de35c272c6b3bd4 167.71.61.184:30303"
TRACE[10-21|14:53:42.333] New dial task                            task="dyndial 6de04dbe582d08c9 164.15.10.134:1467"
TRACE[10-21|14:53:42.333] New dial task                            task="dyndial 6dee1df50008fa13 128.199.36.156:50000"
TRACE[10-21|14:53:42.334] New dial task                            task="dyndial 6defe48b406759cf 157.245.206.138:50000"
TRACE[10-21|14:53:42.334] New dial task                            task="dyndial 6deb29c84e2b3758 188.166.90.213:30303"
TRACE[10-21|14:53:42.348] Dial error                               task="dyndial 6de26d5a9c625384 128.199.36.156:50000" err="dial tcp 128.199.36.156:50000: connect: connection refused"
TRACE[10-21|14:53:42.349] Dial task done                           task="dyndial 6de26d5a9c625384 128.199.36.156:50000"
TRACE[10-21|14:53:42.349] New dial task                            task="dyndial 6deb2fa8db66d98c 176.9.155.139:30303"
TRACE[10-21|14:53:42.349] Dial error                               task="dyndial 6de32e35712c8605 128.199.36.156:50000" err="dial tcp 128.199.36.156:50000: connect: connection refused"
TRACE[10-21|14:53:42.349] Dial task done                           task="dyndial 6de32e35712c8605 128.199.36.156:50000"
TRACE[10-21|14:53:42.349] New dial task                            task="dyndial 6df725320987e763 178.62.40.90:50000"
TRACE[10-21|14:53:42.351] Dial error                               task="dyndial 6dee1df50008fa13 128.199.36.156:50000" err="dial tcp 128.199.36.156:50000: connect: connection refused"
TRACE[10-21|14:53:42.351] Dial task done                           task="dyndial 6dee1df50008fa13 128.199.36.156:50000"
TRACE[10-21|14:53:42.351] New dial task                            task="dyndial 6df50697c60bc573 178.62.27.60:50000"
TRACE[10-21|14:53:42.376] Dial error                               task="dyndial 6df725320987e763 178.62.40.90:50000" err="dial tcp 178.62.40.90:50000: connect: connection refused"
TRACE[10-21|14:53:42.377] Dial task done                           task="dyndial 6df725320987e763 178.62.40.90:50000"
TRACE[10-21|14:53:42.377] New dial task                            task="dyndial 6df2383c819b9098 128.199.55.156:50000"
TRACE[10-21|14:53:42.377] Dial error                               task="dyndial 6df50697c60bc573 178.62.27.60:50000" err="dial tcp 178.62.27.60:50000: connect: connection refused"
TRACE[10-21|14:53:42.378] Dial task done                           task="dyndial 6df50697c60bc573 178.62.27.60:50000"
TRACE[10-21|14:53:42.378] New dial task                            task="dyndial 6dfad945754fe00b 13.233.201.54:23000"
TRACE[10-21|14:53:42.387] Failed proto handshake                   id=6deb29c84e2b3758 addr=188.166.90.213:30303  conn=dyndial err="too many peers"
TRACE[10-21|14:53:42.388] Setting up connection failed             addr=188.166.90.213:30303  err="too many peers"
TRACE[10-21|14:53:42.388] Dial error                               task="dyndial 6deb29c84e2b3758 188.166.90.213:30303" err="too many peers"
TRACE[10-21|14:53:42.388] Dial task done                           task="dyndial 6deb29c84e2b3758 188.166.90.213:30303"
TRACE[10-21|14:53:42.389] New dial task                            task="dyndial 6dc60a052cc54396 35.226.157.25:30303"
TRACE[10-21|14:53:42.395] Dial error                               task="dyndial 6df2383c819b9098 128.199.55.156:50000" err="dial tcp 128.199.55.156:50000: connect: connection refused"
TRACE[10-21|14:53:42.396] Dial task done                           task="dyndial 6df2383c819b9098 128.199.55.156:50000"
TRACE[10-21|14:53:42.396] New dial task                            task="dyndial 6dc6cf8757bad9c8 54.162.153.64:1028"
TRACE[10-21|14:53:42.511] Dial error                               task="dyndial 6dfad945754fe00b 13.233.201.54:23000" err="dial tcp 13.233.201.54:23000: connect: connection refused"
TRACE[10-21|14:53:42.512] Dial task done                           task="dyndial 6dfad945754fe00b 13.233.201.54:23000"
TRACE[10-21|14:53:42.512] New dial task                            task="discovery lookup"
TRACE[10-21|14:53:42.514] >> FINDNODE/v4                           id=e131359c042cb4c4 addr=94.176.236.103:30303  err=nil
TRACE[10-21|14:53:42.514] >> FINDNODE/v4                           id=e793c676acafd0a4 addr=207.246.124.59:30303  err=nil
TRACE[10-21|14:53:42.515] >> FINDNODE/v4                           id=e272f31a9ab9425a addr=157.245.94.240:30300  err=nil
TRACE[10-21|14:53:42.577] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:42.579] Fetching batch of headers                id=8f112bd35cc4e73c conn=inbound count=1  fromnum=8716041 skip=0  reverse=false
TRACE[10-21|14:53:42.610] Findnode failed                          id=860f00e69799a87f failcount=7   err="RPC timeout"
TRACE[10-21|14:53:42.610] Too many findnode failures, dropping     id=860f00e69799a87f failcount=7
TRACE[10-21|14:53:42.613] >> PING/v4                               id=80fd7359b4999319 addr=61.147.122.42:61913   err=nil
TRACE[10-21|14:53:42.628] Dial error                               task="dyndial 6defe48b406759cf 157.245.206.138:50000" err="dial tcp 157.245.206.138:50000: connect: connection refused"
TRACE[10-21|14:53:42.629] Dial task done                           task="dyndial 6defe48b406759cf 157.245.206.138:50000"
TRACE[10-21|14:53:42.633] << NEIGHBORS/v4                          id=e272f31a9ab9425a addr=157.245.94.240:30300  err=nil
TRACE[10-21|14:53:42.639] << NEIGHBORS/v4                          id=e272f31a9ab9425a addr=157.245.94.240:30300  err=nil
TRACE[10-21|14:53:42.653] << NEIGHBORS/v4                          id=e793c676acafd0a4 addr=207.246.124.59:30303  err=nil
TRACE[10-21|14:53:42.653] >> PING/v4                               id=e08d7bbb9abf4c3b addr=171.217.52.134:30303  err=nil
TRACE[10-21|14:53:42.660] << NEIGHBORS/v4                          id=e793c676acafd0a4 addr=207.246.124.59:30303  err=nil
TRACE[10-21|14:53:42.662] >> FINDNODE/v4                           id=e08a00c9ae922ce9 addr=51.79.27.131:30303    err=nil
TRACE[10-21|14:53:42.674] << NEIGHBORS/v4                          id=e131359c042cb4c4 addr=94.176.236.103:30303  err=nil
TRACE[10-21|14:53:42.681] << NEIGHBORS/v4                          id=e131359c042cb4c4 addr=94.176.236.103:30303  err=nil
TRACE[10-21|14:53:42.684] >> FINDNODE/v4                           id=e08a031a7bebb56b addr=3.208.69.150:30303    err=nil
TRACE[10-21|14:53:42.801] << NEIGHBORS/v4                          id=e08a00c9ae922ce9 addr=51.79.27.131:30303    err=nil
TRACE[10-21|14:53:42.808] << NEIGHBORS/v4                          id=e08a00c9ae922ce9 addr=51.79.27.131:30303    err=nil
TRACE[10-21|14:53:42.811] Findnode failed                          id=80fe7bee1d8dc690 failcount=11  err="RPC timeout"
TRACE[10-21|14:53:42.812] Too many findnode failures, dropping     id=80fe7bee1d8dc690 failcount=11
TRACE[10-21|14:53:42.823] << NEIGHBORS/v4                          id=e08a031a7bebb56b addr=3.208.69.150:30303    err=nil
TRACE[10-21|14:53:42.823] Findnode failed                          id=80ff25e95839d737 failcount=2   err="RPC timeout"
TRACE[10-21|14:53:42.823] >> PING/v4                               id=e08174b91ac5a9e7 addr=157.245.206.68:50000  err=nil
TRACE[10-21|14:53:42.824] >> PING/v4                               id=80fd30273665f2fa addr=138.68.246.170:50000  err=nil
TRACE[10-21|14:53:42.825] >> PING/v4                               id=80fa8a11d65b83ad addr=52.194.227.28:30303   err=nil
TRACE[10-21|14:53:42.832] << NEIGHBORS/v4                          id=e08a031a7bebb56b addr=3.208.69.150:30303    err=nil
TRACE[10-21|14:53:42.834] >> PING/v4                               id=e0815e37ee2e7f9b addr=91.157.100.100:30303  err=nil
TRACE[10-21|14:53:42.878] Filtering headers                        peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:42.881] Synchronisation terminated               elapsed=6.298s
DEBUG[10-21|14:53:42.881] Reset ancient limit to zero 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1b8 pc=0x63d828]

goroutine 266 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestor(0x20dab40, 0x21fc300, 0x8684240, 0x0, 0x0, 0xa, 0xa)
  /home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:877 +0x1500
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0x20dab40, 0x21fc300, 0xc69faaa9, 0x4ef38840, 0xb522428c, 0x16d006cd, 0x26f28c, 0x1620fb5d, 0x5991498f, 0xba1a4001, ...)
  /home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:443 +0x2a0
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0x20dab40, 0x800d6b0, 0x10, 0xc69faaa9, 0x4ef38840, 0xb522428c, 0x16d006cd, 0x26f28c, 0x1620fb5d, 0x5991498f, ...)
  /home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:411 +0x338
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0x20dab40, 0x800d6b0, 0x10, 0xc69faaa9, 0x4ef38840, 0xb522428c, 0x16d006cd, 0x26f28c, 0x1620fb5d, 0x5991498f, ...)
  /home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:323 +0x5c
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).synchronise(0x21e63f0, 0x8124480)
  /home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:190 +0x1b0
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer
  /home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:155 +0x17c
@holiman

This comment has been minimized.

Copy link
Contributor

@holiman holiman commented Oct 22, 2019

Interesting. It looks like the block body for block 8716041 exists, but the header seems to not be present.

@m0ar

This comment has been minimized.

Copy link
Author

@m0ar m0ar commented Oct 22, 2019

@holiman LMK if I can provide any other data to help find the culprit here.

Is there any way I can get through this impasse?

@holiman

This comment has been minimized.

Copy link
Contributor

@holiman holiman commented Oct 23, 2019

Is there any way I can get through this impasse?

One thing that might work is to use debug.setHead at 8714666.

@holiman

This comment has been minimized.

Copy link
Contributor

@holiman holiman commented Oct 23, 2019

@m0ar the first crash you had -- do you know if that was the exact same crash as this one? I'm thinking that maybe the first crash left the db a bit corrupted, and now you're hitting this because some data is missing in the db.

Also, what hardware are you running on ? What's the ram size?

@m0ar

This comment has been minimized.

Copy link
Author

@m0ar m0ar commented Oct 23, 2019

@holiman How can I set this before the client crashes? It fails within seconds.

@m0ar

This comment has been minimized.

Copy link
Author

@m0ar m0ar commented Oct 23, 2019

Got it done. It definitely ran for longer, but crashed again:

[...]
TRACE[10-23|09:43:56.593] >> PING/v4                               id=87f1a269da09b133 addr=164.15.10.134:1439    err=nil
TRACE[10-23|09:43:56.602] >> FINDNODE/v4                           id=87f2609389c3c75f addr=74.71.193.225:30303   err=nil
TRACE[10-23|09:43:56.683] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:56.684] Remote head header identified            peer=fbecffbe2c173d8a number=8795682 hash=8952d0…f3c6e0
DEBUG[10-23|09:43:56.684] Looking for common ancestor              peer=fbecffbe2c173d8a local=8714666 remote=8795682
TRACE[10-23|09:43:56.684] Span searching for common ancestor       peer=fbecffbe2c173d8a count=12 from=8795505 skip=15
DEBUG[10-23|09:43:56.684] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=12 fromnum=8795505 skip=15 reverse=false
DEBUG[10-23|09:43:56.693] Peer discarded announcement              peer=fbecffbe2c173d8a number=8795684 hash=710e5d…04f5f2 distance=81018
TRACE[10-23|09:43:56.748] >> FINDNODE/v4                           id=dc1e03b53752c6d7 addr=113.52.133.48:30303   err=nil
TRACE[10-23|09:43:56.771] Binary searching for common ancestor     peer=fbecffbe2c173d8a start=8624666 end=8795682
DEBUG[10-23|09:43:56.772] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8710174 skip=0  reverse=false
TRACE[10-23|09:43:56.936] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:56.944] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8752928 skip=0  reverse=false
TRACE[10-23|09:43:56.990] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:56.991] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8731551 skip=0  reverse=false
TRACE[10-23|09:43:57.083] >> FINDNODE/v4                           id=dc1d32b41bfa9f2c addr=165.22.28.47:50000    err=nil
TRACE[10-23|09:43:57.083] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.084] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8720862 skip=0  reverse=false
TRACE[10-23|09:43:57.101] Findnode failed                          id=87f2609389c3c75f failcount=1   err="RPC timeout"
TRACE[10-23|09:43:57.103] >> PING/v4                               id=87f4880a9638d756 addr=94.79.55.28:30000     err=nil
TRACE[10-23|09:43:57.154] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.155] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8715518 skip=0  reverse=false
TRACE[10-23|09:43:57.234] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.240] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8718190 skip=0  reverse=false
TRACE[10-23|09:43:57.247] Findnode failed                          id=dc1e03b53752c6d7 failcount=3   err="RPC timeout"
TRACE[10-23|09:43:57.249] >> PING/v4                               id=dc1d079a808feccf addr=91.55.92.55:30303     err=nil
TRACE[10-23|09:43:57.349] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.350] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716854 skip=0  reverse=false
TRACE[10-23|09:43:57.397] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.398] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716186 skip=0  reverse=false
TRACE[10-23|09:43:57.444] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.445] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8715852 skip=0  reverse=false
DEBUG[10-23|09:43:57.466] Peer discarded announcement              peer=fbecffbe2c173d8a number=8795684 hash=d9d7fb…bded03 distance=81018
TRACE[10-23|09:43:57.475] >> FINDNODE/v4                           id=87f1d99d692cc00e addr=116.253.203.71:5818   err=nil
TRACE[10-23|09:43:57.481] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.488] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716019 skip=0  reverse=false
TRACE[10-23|09:43:57.514] >> FINDNODE/v4                           id=dc1d3faf109a9f5b addr=157.245.93.35:50000   err=nil
TRACE[10-23|09:43:57.541] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.543] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716102 skip=0  reverse=false
TRACE[10-23|09:43:57.573] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.575] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716060 skip=0  reverse=false
TRACE[10-23|09:43:57.582] Findnode failed                          id=dc1d32b41bfa9f2c failcount=2   err="RPC timeout"
TRACE[10-23|09:43:57.584] >> PING/v4                               id=dc1cc69c6a05eb8f addr=13.230.252.33:30303   err=nil
TRACE[10-23|09:43:57.595] >> FINDNODE/v4                           id=87f1a269da09b133 addr=164.15.10.134:1439    err=nil
TRACE[10-23|09:43:57.661] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.661] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716039 skip=0  reverse=false
TRACE[10-23|09:43:57.697] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.700] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716049 skip=0  reverse=false
TRACE[10-23|09:43:57.753] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.754] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716044 skip=0  reverse=false
TRACE[10-23|09:43:57.787] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.787] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716041 skip=0  reverse=false
TRACE[10-23|09:43:57.847] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.851] Synchronisation terminated               elapsed=11.386s
DEBUG[10-23|09:43:57.851] Reset ancient limit to zero 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1b8 pc=0x63d828]

goroutine 376 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestor(0x24bac80, 0x266c6c0, 0x398b440, 0x0, 0x0, 0xa, 0xa)
	/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:877 +0x1500
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0x24bac80, 0x266c6c0, 0xc3d05289, 0x15726f7b, 0x93e43866, 0xc042d0bd, 0x75832ac, 0xc6690d42, 0xd013c950, 0xe0c6f3ee, ...)
	/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:443 +0x2a0
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0x24bac80, 0x89d1ca0, 0x10, 0xc3d05289, 0x15726f7b, 0x93e43866, 0xc042d0bd, 0x75832ac, 0xc6690d42, 0xd013c950, ...)
	/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:411 +0x338
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0x24bac80, 0x89d1ca0, 0x10, 0xc3d05289, 0x15726f7b, 0x93e43866, 0xc042d0bd, 0x75832ac, 0xc6690d42, 0xd013c950, ...)
	/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:323 +0x5c
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).synchronise(0x24921b0, 0x8a24e80)
	/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:190 +0x1b0
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer
	/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:155 +0x17c
@m0ar

This comment has been minimized.

Copy link
Author

@m0ar m0ar commented Oct 23, 2019

@m0ar the first crash you had -- do you know if that was the exact same crash as this one? I'm thinking that maybe the first crash left the db a bit corrupted, and now you're hitting this because some data is missing in the db.

Also, what hardware are you running on ? What's the ram size?

I'm not sure, could be. Would a resync be the only solution then?

It's this setup: https://kauri.io/article/9695fcca217f46feb355245275835fc0
(RaspberryPi 4B /w 4GB and a 500GB samsung SSD)

@diglos

This comment has been minimized.

Copy link

@diglos diglos commented Oct 24, 2019

Hi.

I'm currently doing a fast sync on the Pi 4 and ran into this error (full log attached):

Oct 21 22:47:58 ethnode-9efbf7ff geth[526]: runtime: pointer 0x8f8fe000 to unallocated span span.base()=0x8c1be000 span.limit=0x2761dfe0 span.state=3
Oct 21 22:47:58 ethnode-9efbf7ff geth[526]: runtime: found in object at *(0x3834c00+0x320)
Oct 21 22:47:58 ethnode-9efbf7ff geth[526]: object=0x3834c00 s.base()=0x3830000 s.limit=0x3835f00 s.spanclass=88 s.elemsize=4864 s.state=mSpanInUse

It happened just once and the sync process seems normal (seems a pointer bug but I'm not sure if it is related to this issue). Just in case it helps. I will report back when the fast sync completes.

Setup:

  • Raspberry Pi 4 4GB RAM
  • --cache 512
  • Swap: 4GB
  • Disk: SSD Kingston 480GB with SATA-USB case (ASMedia ASM1153E chipset)

Geth: Official armv7 binary 1.9.6-stable

@m0ar, if you do a resync, would you mind to try this image?

https://ethraspbian.com/downloads/image_2019-10-20-EthRaspbian2.0-lite.zip

Just flash it and the fast sync process will start in 2-3 minutes (keep in mind that it will wipe out your entire USB disk and reinstall everything). Login ethereum:ethereum. You will be prompted to change the password on first login.

geth-bad-pointer-error-rp4.txt

@diglos

This comment has been minimized.

Copy link

@diglos diglos commented Oct 26, 2019

Hi.

Fast sync process was completed in 4 days. But, I got lots of errors like this one:

Oct 26 09:19:28 ethnode-9efbf7ff geth[16298]: runtime: out of memory: cannot allocate 279183360-byte block (2246934528 in use)
Oct 26 09:19:28 ethnode-9efbf7ff geth[16298]: fatal error: out of memory
Oct 26 09:19:28 ethnode-9efbf7ff geth[16298]: runtime stack:
Oct 26 09:19:28 ethnode-9efbf7ff geth[16298]: runtime.throw(0xb64f9a, 0xd)

It turns out that setting the cache to 256 MB (it was 512MB) solved this and Geth runs fine. But, somehow this is weird as with other 2 ARM boards (ARM64) I was using the default cache value (a little more that 1GB) and never ran into this (and both devices have 4GB RAM as well). Is it possible that this errors are caused by some problem with Go's GC?

@diglos

This comment has been minimized.

Copy link

@diglos diglos commented Oct 30, 2019

Hi, got another error:

Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: INFO [10-29|21:40:06.773] Imported new block headers               count=1    elapsed=20.904ms     number=8836401 hash=9993f4…8f53dc age=1m28s
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: panic: runtime error: slice bounds out of range [:37:34]
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: goroutine 1320563 [running]:
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy.load32(...)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy/encode_other.go:10
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy.encodeBlock(0x50148141, 0x46, 0x46, 0x50aee200, 0x22, 0x200, 0x12a684c8)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy/encode_other.go:176 +0x900
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy.Encode(0x50148140, 0x47, 0x47, 0x0, 0x0, 0x0, 0x200, 0x0, 0x0)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy/encode.go:37 +0x260
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/p2p.(*rlpxFrameRW).WriteMsg(0x22038b80, 0x1f, 0x0, 0x22, 0xdf1f98, 0x506e4440, 0x0, 0x0, 0x0, 0x0, ...)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:601 +0xc8
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/p2p.(*rlpx).WriteMsg(0x20b7f280, 0x1f, 0x0, 0x22, 0xdf1f98, 0x506e4440, 0x0, 0x0, 0x0, 0x0, ...)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:102 +0xe8
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/p2p.(*protoRW).WriteMsg(0x21d8c1e0, 0x1f, 0x0, 0x22, 0xdf1f98, 0x506e4440, 0x0, 0x0, 0x0, 0x0, ...)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/p2p/peer.go:412 +0x15c
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/p2p.Send(0xa5820030, 0x21d8c1e0, 0xf, 0x0, 0xa09728, 0x5954c5a0, 0x0, 0x0)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/p2p/message.go:101 +0xa0
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/eth.(*peer).RequestReceipts(0x20cffd00, 0x1c8063a0, 0x1, 0x1, 0xa19b711e, 0xdfeb7436)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/peer.go:351 +0x128
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: created by github.com/ethereum/go-ethereum/eth/downloader.(*peerConnection).FetchReceipts
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/peer.go:203 +0x20c
Oct 29 21:40:07 ethnode-9efbf7ff systemd[1]: geth.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

I don't know if they are related. Let me know if you want me to open a separate issue.

@diglos

This comment has been minimized.

Copy link

@diglos diglos commented Nov 15, 2019

Hi,

I did another Fast Sync with Geth 1.9.7. This time I only ran into an unallocated span issue (see attached). It seems that the out of memory issues were a Go problem, I will keep monitoring the process in the coming days though.

geth-pointer-unallocated-span.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.