This repository has been archived by the owner on Nov 6, 2020. It is now read-only.
Bad block detected: TimestampOverflow #10688
Labels
F2-bug 🐞
The client fails to follow expected behavior.
M4-core ⛓
Core client code / Rust.
P5-sometimesoon 🌲
Issue is worth doing soon.
Milestone
Comments
jam10o-new
added
F2-bug 🐞
|
We have possible the same issue. 15 validators |
|
(cc @niklasad1 any ideas on what's going on here?) |
|
Hey, it looks like verify_timestamp failed. Can you run with |
|
Turn on |
|
Ok, I thought this would show |
|
Turn on engine=trace, but still no messages with Got one more fork, there was no errors, no fails of parity. All other nodes in sync. |
|
I don't seem to be able to use parity atm because of this too, I get 1 import in and it stops. This is from a freshly started archive node using 2.4.6. Possibly related to #10610 as I then just get this state: On further tests it seems I am importing fine until that TimeStamp block is hit and I drop to the 0.00 blk/s syncing state forever. |
|
FYI I reverted back to 2.2.9-stable and used the updated chain spec as mentioned by @joshua-mir in #10610 and this seems to unblock me. Just to note, the error changed using 2.2.9 to:
|
|
We have the same problem at https://trustlines.network/. For the 2.3.x line: for the 2.4.x line: To easy reproduce the behavior you can just run parity connected to an aura chain, and then manipulate the the time on you machine a little bit backwards, so that you will receive blocks from the future. The different behavior is then: 2.3.6, 2.4.1:Peers are not dropped, and after you fixed the system time everything is fine again. 2.3.7, 2.4.2:After that all peers get dropped and it will not recover from it. Only a restart helps. This makes the newer version unusable for us! |
|
My question here would be, how strict is the aura protocol with the timestamp. How much seconds from the future are allowed? And if you receive a block with a timestamp from the future from a certain peer, is parity suppose to drop that peer? |
|
@berndbohmeier to answer your question Aura is extremely strict - all steps must be broadcast and received within the stepduration you configure, and there is a strong assumption that peers have synchronized system clocks (I'd say within a half second). If things come "seconds from the future" they will almost certainly cause issues. Re: dropping peers, if a peer sends a bad block, parity may drop them. If the block is just on a fork with shared ancestry, it should not. I agree this might be an issue when applied to this particular error however. I should probably note #10720 should resolve this problem though |
|
Lets hope it does :) |
|
We have a pretty similar setup (just Ubuntu 18.04) and suddenly run into the same issue with one of the nodes. The node ran fine for about 1,5 month with Parity 2.4.5 and suddenly stopped syncing blocks. We had no luck to get the node synching new blocks until we deleted the storage folder (400 GB still free space). We upgraded to 2.4.6 and instead of the stopped synching we had the TimeOverFlow error mentioned in the issue description and the node suddenly had no peers anymore. Strange thing is, that we only have issues with virtual machines on Google Cloud and Azure so far. Dedicated hardware has no issues at all. |
|
We are observing the same issue in several nodes of our PoA-based network (Ubuntu 18.04 and Mac OS, Parity versions 2.4.6 and 2.4.5). The attached log is captured from a Ubuntu virtual machine running v2.4.5 and logging sync,network. The issue happens at line 483 and is being observed several times per day, causing downtimes to the infrastructure. For instance when ntp daemon detects a network configuration change like this: Unfortunately we can't roll back to v2.4.1 as suggested by @berndbohmeier and looks like #10720 has not been merged in v2.4.7. |
|
Hey, can anyone of you please try #10720 just to make sure that it actually fixes you problem? Thanks! |
|
Hi! We tested (@marcelorocha-e) your PR by using a VirtualBox machine (Ubuntu) running on MacOS. How do we test? We connect to our test network and pause the VM. After the resume, we observe the behaviour of Parity. There are three test cases:
The PR version #10720 still shows the error message when we resume the VM, but: |
|
Yes. After additional tests, we observe that with this fix the (there is a compilation warning: unused import) |
|
I've tested #10720 and |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I have configured a private network with PoA Aura and the follow chain spec:
{ "name": "Chain Name", "engine": { "authorityRound": { "params": { "blockReward": 0, "stepDuration": 5, "maximumEmptySteps": 24, "emptyStepsTransition": 0, "maximumUncleCountTransition": 0, "maximumUncleCount": 0, "validators": { "multi": { "0": { "list": [ "0x..." ] }, "5": { "safeContract": "0x0000000000000000000000000000000000000005" } } } } } }, "params": { "gasLimitBoundDivisor": "0x400", "maximumExtraDataSize": "0x20", "minGasLimit": "0x1388", "networkID": "0x1E61", "eip140Transition": 0, "eip211Transition": 0, "eip214Transition": 0, "eip658Transition": 0, "eip145Transition": 0, "eip1014Transition": 0, "eip1052Transition": 0 }, ...From block 6 onwards I have added 2 more validators and after a good working time for some reason 2 of those 3 validators started to show a warning message:
The 3 validators are in different servers and has the same clock time (UTC) and the same configuration:
The actual behavior is the validator that is not showing this message is not sealing block anymore.
The expected behavior is all the validators sealing blocks.
Thanks.
The text was updated successfully, but these errors were encountered: