Error: Header Rejected, too far in the future

[ImmaZoni]

Error Code

💔 Verification failed for block 0xf514f794c67003ed3ca47f8d3dbb9056b1aefc6de8b1a50a761048f39a5dd09d received from peer: 12D3KooWPEynXbQeuGwai76xheGav44cCs3n4cpMpCP94g7my1wT, "Header 0xf514f794c67003ed3ca47f8d3dbb9056b1aefc6de8b1a50a761048f39a5dd09d rejected: too far in the future"

Description

The user contacted me regarding their node crashing due to a power outage, when they reconnected there were no peers connected and their chain was behind the longest known chain.

As time progressed, in some cases the node would catch up; connect to 4 peers and work for about 1 min then fall back behind and drop all the peers again.

Expected Behavior

The node should catch up to the chain, and keep caught up.
furthermore, the node should not lose the peer connection.

Actual Behavior

The node would try to sync with other nodes and catch up for a moment, then fall far behind again.

Attempted Troubleshooting

We rebooted the machine, attempted to reconnect the node, and the issue persisted.
From here we purged the chain and attempted sync from Genysis. initially, it was syncing very effectively, but once it got to the longest chain the issue arose again.

General farmer usage

Steps to Reproduce

*This may be hard to replicate.

1. Run node & farmer as per normal workflow.
2. Unexpectedly power off the machine.
3. Boot machine & attempt to restart node & farmer
4. see if the issue arises.

Your Environment

• Version used: Spartan Testnet
• Operating System and version: Windows 10
• Desktop, Laptop, or Server: Lenovo Legion

[nazar-pc]

If after the purge issue still persists it might be an issue with system time. Is their time synced and up to date? If it slides significantly in either direction it might result in this.

[ImmaZoni]

If after the purge issue still persists it might be an issue with system time. Is their time up synced and to date? If it slides significantly in either direction it might result in this.

Gotcha, I will test with user and verify clock is properly synced

[nazar-pc]

I have seen this error about some blocks from a specific user yesterday. I guess it is inevitable in production network, we just need to make sure it is caused by time de-sync and not bug in our code.

[ImmaZoni]

Would it maybe be worth it to have the desktop farmer, and the CLI do a time sync check and just prompt the user if they are out of sync? Obviously, it's kind of out of scope to have the core protocol do this check, but we could integrate it with some supporting applications to make it easy for users to self resolve this issue, let me know your thoughts.

[nazar-pc]

I'm not aware of anyone else doing that, but I guess it might be useful 🤷‍♂️

Do you think we can close this for now?