Release 1.1.0 stable cannot full sync on fast mode #329

dracmic · 2021-07-28T08:41:01Z

System information

Geth version:
geth version
Geth
Version: 1.1.0
Git Commit: 7822e9e
Architecture: amd64
Go Version: go1.16.3
Operating System: linux
GOPATH=
GOROOT=go

OS & Version: Linux Ubuntu 20.04
Commit hash : (if develop)

Expected behaviour

To be fully synced and to not be behind with 100 blocks

Actual behaviour

Pivot Stale, receipts behind with ~100 blocks and the issue was also in 1.1.0-beta and actually the issue started to happen about 1 week and half ago after I saw that my node gets behind I did restart it and started this behaviour. Deleted and resynced same issue. I tried 2-3 different server providers all with NVME and a lot of RAM and different location same issue: eth.blockNumber on 0 and pivot getting stalled so I get the issue with 3 minutes back blocks (~100blocks).

Steps to reproduce the behaviour

Start a full sync ( personally I've used servers from EU countries) by using fast as syncmode

Backtrace

[backtrace]

When submitting logs: please submit them as text and not screenshots.

The text was updated successfully, but these errors were encountered:

dwjpeng · 2021-07-28T09:15:29Z

same problem， I tried different syncmodes（snap and fast） but it didn't work；now more than 8,000 blocks behind；

maldiva · 2021-07-28T12:27:37Z

Having the same issue. Has anyone solved this problem? My hardware specs are more than enough to run a node, so it's not the problem. Changing sync modes didn't help, txlookuplimit didn't help as well

Crypto2 · 2021-07-28T14:24:25Z

I'm having this problem with snap sync, it's fully caught up on blocks but syncing won't end.

INFO [07-28|10:23:22.535] Downloader queue stats                   receiptTasks=0     blockTasks=0     itemSize=696.76KiB  throttle=95
WARN [07-28|10:23:22.535] Pivot became stale, moving               old=9,549,567 new=9,549,631
INFO [07-28|10:23:23.012] Imported new block receipts              count=64   elapsed=475.749ms   number=9,549,630 hash=f1ba4a..79446e age=3m22s      size=19.06MiB
INFO [07-28|10:23:23.012] State heal in progress                   accounts=1,336,240@66.72MiB   slots=10,615,893@741.15MiB codes=1649@14.33MiB      nodes=31,544,661@6.05GiB   pending=9232
INFO [07-28|10:23:25.891] Imported new block headers               count=1    elapsed=1.221ms     number=9,549,694 hash=ac0434..51228c
INFO [07-28|10:23:29.422] Imported new block headers               count=2    elapsed=1.062ms     number=9,549,696 hash=dd897f..2ad02e
INFO [07-28|10:23:31.031] State heal in progress                   accounts=1,336,245@66.72MiB   slots=10,615,893@741.15MiB codes=1649@14.33MiB      nodes=31,551,163@6.06GiB   pending=18434

cctanfujun · 2021-07-28T17:45:30Z

Having the same issue. knownStates and pulledStates increase but not sync finish

flonussi · 2021-07-29T06:45:36Z

Same here, synced fine before, now after the update I cannot get the node to sync. Hardware is more than enough: 12 core cpu, nvme ssd 980 pro, 128gb memory.

dracmic · 2021-07-29T07:39:17Z

@flonussi @cctanfujun @dwjpeng @Crypto2 I have a look here https://explorer.bitquery.io/bsc/calls maybe we can help does CryptoBlades (SKILL) to not do transactions for fights on blockchain :) (https://explorer.bitquery.io/bsc/smart_contract/0x39bea96e13453ed52a734b6aceed4c41f57b2271/methods)

For me new issue at last 2-3 minutes of full sync:
t=2021-07-29T07:16:03+0000 lvl=info msg="Sidechain written to disk" start=9,568,446 end=9,568,446 sidetd=19,060,848 localtd=19,061,334
t=2021-07-29T07:16:03+0000 lvl=info msg="Sidechain written to disk" start=9,568,446 end=9,568,446 sidetd=19,060,849 localtd=19,061,334
t=2021-07-29T07:16:03+0000 lvl=info msg="Sidechain written to disk" start=9,568,448 end=9,568,448 sidetd=19,060,853 localtd=19,061,334
t=2021-07-29T07:16:03+0000 lvl=info msg="Sidechain written to disk" start=9,568,448 end=9,568,448 sidetd=19,060,852 localtd=19,061,334
t=2021-07-29T07:16:03+0000 lvl=info msg="Sidechain written to disk" start=9,568,450 end=9,568,450 sidetd=19,060,857 localtd=19,061,334
t=2021-07-29T07:16:03+0000 lvl=info msg="Sidechain written to disk" start=9,568,451 end=9,568,451 sidetd=19,060,859 localtd=19,061,334

cctanfujun · 2021-07-29T07:58:23Z

Having the same issue. knownStates and pulledStates increase but not sync finish

cctanfujun · 2021-07-29T08:00:18Z

i ask some people, they said because of nft games,their are too many transcation,just wait about 48hours,if you are use nvme ssd

lovelyrrg51 · 2021-07-29T18:42:52Z

Same problems here.

lovelyrrg51 · 2021-07-29T18:43:08Z

when they can fix it?

dracmic · 2021-07-29T19:15:52Z

@lovelyrrg51 @cctanfujun they've released 1.1.1-beta which personally I am running and got synced. So give it a try :)

lovelyrrg51 · 2021-07-29T19:25:21Z

Really?
@dracmic so only update geth with 1.1.1?
And then resync?

dracmic · 2021-07-29T19:33:02Z

@lovelyrrg51 I have a full node started with snapshot ... which was almost synced ... I did a safe stop like crtl+c and replace the exec and I started and waited a bit ... but it got synced.

lovelyrrg51 · 2021-07-29T19:33:35Z

Ah, okay, got it.
Thanks for your advice.
@dracmic

Crypto2 · 2021-07-29T23:05:43Z

Is 1.1.1 working for everyone? Unfortunately no luck here with it :(

lovelyrrg51 · 2021-07-30T04:57:19Z

Yes
still syncing in 10 hours.

litebarb · 2021-07-30T05:22:31Z

@lovelyrrg51 do you mean your syncing has completed?

dracmic · 2021-07-30T05:23:59Z

problem with fast syncing persists in 1.1.1-beta ....
t=2021-07-30T07:12:55+0200 lvl=warn msg="Pivot became stale, moving" old=9,593,429 new=9,593,493
t=2021-07-30T07:16:07+0200 lvl=warn msg="Pivot became stale, moving" old=9,593,493 new=9,593,557
t=2021-07-30T07:19:20+0200 lvl=warn msg="Pivot became stale, moving" old=9,593,557 new=9,593,621

lovelyrrg51 · 2021-07-30T05:27:01Z

not completed.

litebarb · 2021-07-30T05:29:53Z

@lovelyrrg51 alrights let us know how it goes. I am on 1.1.0, stable. Around 11hours into syncing, not completed yet.

lovelyrrg51 · 2021-07-30T05:32:07Z

Yes, of course.
1.1.1-beta is still syncing, not completed too.
@litebarb

lovelyrrg51 · 2021-07-30T05:33:02Z

not sure which version should be use.

litebarb · 2021-07-30T05:34:55Z

@lovelyrrg51 is this your first time setting up a node? Were you successful in previous versions before? Seems like some ppl are able to sync on 1.1.0 beta and 1.0.7

lovelyrrg51 · 2021-07-30T05:36:55Z

Not first time.
About few days ago, it works well with 1.1.0 beta.
but not work well now.
Probably, because of CryptoBlades, too many transactions. :)

litebarb · 2021-07-30T05:38:28Z

problem with fast syncing persists in 1.1.1-beta ....
t=2021-07-30T07:12:55+0200 lvl=warn msg="Pivot became stale, moving" old=9,593,429 new=9,593,493
t=2021-07-30T07:16:07+0200 lvl=warn msg="Pivot became stale, moving" old=9,593,493 new=9,593,557
t=2021-07-30T07:19:20+0200 lvl=warn msg="Pivot became stale, moving" old=9,593,557 new=9,593,621

@dracmic I thought you managed to sync up? When you said you replaced the exec file, what do you mean exactly? Do you mean you only replaced the geth_linux file from https://github.com/binance-chain/bsc/releases/download/v1.1.1-beta/geth_linux? Also how many hours did it take for your node to be fully synced?

lovelyrrg51 · 2021-07-30T05:41:22Z

All of third parties including quicknode&getblock not sync complete too.

litebarb · 2021-07-30T05:41:45Z

Not first time.
About few days ago, it works well with 1.1.0 beta.
but not work well now.
Probably, because of CryptoBlades, too many transactions. :)

Oooo I see. Any reason you are upgrading to 1.1.0 stable? I thought the beta version was good enough?

litebarb · 2021-07-30T05:42:49Z

@lovelyrrg51 and how long did your 1.1.0 beta version took before sync completed?

lovelyrrg51 · 2021-07-30T05:43:03Z

not to 1.1.0 stable
updated geth to 1.1.1 beta.
but not working too.

lovelyrrg51 · 2021-07-30T05:43:34Z

@lovelyrrg51 and how long did your 1.1.0 beta version took before sync completed?

about 2 month

litebarb · 2021-07-30T05:44:58Z

@lovelyrrg51 and how long did your 1.1.0 beta version took before sync completed?

about 2 month

I mean how long did your server took from scratch to achieve full sync state? 10hours?

lovelyrrg51 · 2021-07-30T05:46:03Z

Ah, no
about 3 days

dracmic · 2021-07-30T06:02:27Z

problem with fast syncing persists in 1.1.1-beta ....
t=2021-07-30T07:12:55+0200 lvl=warn msg="Pivot became stale, moving" old=9,593,429 new=9,593,493
t=2021-07-30T07:16:07+0200 lvl=warn msg="Pivot became stale, moving" old=9,593,493 new=9,593,557
t=2021-07-30T07:19:20+0200 lvl=warn msg="Pivot became stale, moving" old=9,593,557 new=9,593,621

@dracmic I thought you managed to sync up? When you said you replaced the exec file, what do you mean exactly? Do you mean you only replaced the geth_linux file from https://github.com/binance-chain/bsc/releases/download/v1.1.1-beta/geth_linux? Also how many hours did it take for your node to be fully synced?

@litebarb
Yes ... But I tried on other server for fast sync ... without the snapshot ... just to test out the sync ...
Yes I just replaced the exec with the one from https://github.com/binance-chain/bsc/releases/download/v1.1.1-beta/geth_linux ...
and on the node with snapshot took about 3-4 hours to recover the last hour from states and receipts.

litebarb · 2021-07-30T07:21:58Z

@dracmic So in summary, your node with snapshot synced, and the other node without snapshot did not sync? Can you share with us the link you used for snapshot? Is this the one? https://github.com/binance-chain/bsc-snapshots/blob/main/README.md

dracmic · 2021-07-30T07:28:29Z

@litebarb in which area you have your server maybe I can share a faster way from the server which I build ... Yes I used the the snapshot from there with syncmode="snap" ...

litebarb · 2021-07-30T07:45:59Z

@dracmic It's a custom server I have at home, I am in South East Asia

dracmic · 2021-07-30T07:47:32Z

@litebarb then that s3 link from there I think is faster to download from what I have in Europe.

litebarb · 2021-07-30T07:53:57Z

@dracmic Man I have a good connection (1 gbps) but the speed it's giving me takes me another 9hours to complete. Lol. Where can I find alternative links?

j75689 · 2021-07-30T07:56:06Z

At present, fast/snap sync is very unstable, a large number of transactions will cause more state changes, and it will become very difficult to catch up with the current state with the current implementation.

We recommend that you download the snapshot we provide and run a full sync.

If there is a sync issue, please report it at #338 and we will continue to update.

seiyria · 2021-08-02T18:31:09Z

[...] I have a look here explorer.bitquery.io/bsc/calls maybe we can help does CryptoBlades (SKILL) to not do transactions for fights on blockchain :)

CryptoBlades is intended to be entirely on chain with minimal external services. We don't really plan to do stuff like this off-chain because our users prefer us to be entirely on-chain.

j75689 closed this as completed Jul 30, 2021

Release 1.1.0 stable cannot full sync on fast mode #329

Release 1.1.0 stable cannot full sync on fast mode #329

Comments

dracmic commented Jul 28, 2021 • edited

System information

Expected behaviour

Actual behaviour

Steps to reproduce the behaviour

Backtrace

dwjpeng commented Jul 28, 2021

maldiva commented Jul 28, 2021

Crypto2 commented Jul 28, 2021 • edited

cctanfujun commented Jul 28, 2021

flonussi commented Jul 29, 2021

dracmic commented Jul 29, 2021 • edited

cctanfujun commented Jul 29, 2021

cctanfujun commented Jul 29, 2021

lovelyrrg51 commented Jul 29, 2021

lovelyrrg51 commented Jul 29, 2021

dracmic commented Jul 29, 2021

lovelyrrg51 commented Jul 29, 2021 • edited

dracmic commented Jul 29, 2021 • edited

lovelyrrg51 commented Jul 29, 2021

Crypto2 commented Jul 29, 2021

lovelyrrg51 commented Jul 30, 2021

litebarb commented Jul 30, 2021

dracmic commented Jul 30, 2021

lovelyrrg51 commented Jul 30, 2021

litebarb commented Jul 30, 2021

lovelyrrg51 commented Jul 30, 2021

lovelyrrg51 commented Jul 30, 2021

litebarb commented Jul 30, 2021

lovelyrrg51 commented Jul 30, 2021

litebarb commented Jul 30, 2021 • edited

lovelyrrg51 commented Jul 30, 2021

litebarb commented Jul 30, 2021 • edited

litebarb commented Jul 30, 2021

lovelyrrg51 commented Jul 30, 2021

lovelyrrg51 commented Jul 30, 2021

litebarb commented Jul 30, 2021

lovelyrrg51 commented Jul 30, 2021

dracmic commented Jul 30, 2021 • edited

litebarb commented Jul 30, 2021 • edited

dracmic commented Jul 30, 2021

litebarb commented Jul 30, 2021

dracmic commented Jul 30, 2021

litebarb commented Jul 30, 2021

j75689 commented Jul 30, 2021

seiyria commented Aug 2, 2021

dracmic commented Jul 28, 2021 •

edited

Crypto2 commented Jul 28, 2021 •

edited

dracmic commented Jul 29, 2021 •

edited

lovelyrrg51 commented Jul 29, 2021 •

edited

dracmic commented Jul 29, 2021 •

edited

litebarb commented Jul 30, 2021 •

edited

litebarb commented Jul 30, 2021 •

edited

dracmic commented Jul 30, 2021 •

edited

litebarb commented Jul 30, 2021 •

edited