client invalidateheader and resetchainhead

[antiochp]

Resolves #3607

---

Experimenting with a different approach here to resolving the various issues around sync and how it interacts with a chain fork.

This PR adds an endpoint and handler for resetchainhead via the owner api.

This will take a block hash and will do something roughly along the following lines -

• check the header being reset to is indeed present in the local db
• reset header_head to the existing head via "rewind and reapply" on the header extension
• then reset head (and header_head consistently) via "rewind and reapply" on txhashset extension (full blocks)

The should result in the ability to arbitrarily "rewind" or "reset" to an alternative block header.

Rather than trying to resolve all edge cases related to sync I'm proposing it makes more sense to have additional flexibility and allow a node operator to "manually" work around a fork via this api endpoint.

If we can get this working robustly then there is an additional api endpoint that could be added -

• invalidateheader

These could be used together to invalidate one chain fork and then reset the local chain to a header on an alternative fork.
I believe these would allow most fork scenarios to be recovered from reasonably gracefully albeit manually.

If this approach works then we may consider extending this and doing the "reset" in a more automated fashion without api involvement.

---

Non-archival node should be able to "reset" to prior to the horizon by -

• remove recent full blocks from db (effectively all blocks, horizon onwards)
• reset head (and tail)
• truncate the MMR structures
• force a state sync

This would potentially be a convenient way of retaining headers (up to reset point) locally while still resyncing state.

---

TODO

• error checking and basic validation (make sure its actually a hash)
• user feedback - success/failure messages (jsonrpc issues?)
• "can we rewind beyond horizon"
  • For a full archival node - need to confirm but technically possible?
  • For non-archival node see above (for now this is prevented and full resync is required)
• cleanup

[tromp]

check the header being reset to is indeed present in the local db

the header is specified by its (cycle)hash?

reset header_head to the existing head via "rewind and reapply" on the header extension

what is a header extension? what does reapplying a header mean? is this updating a constant amount of state?

then reset head (and header_head consistently) via "rewind and reapply" on txhashset extension (full blocks)

can this rewind beyond the horizon, if the node saw preceding blocks?

[antiochp]

No guarantees to this working 100% correctly and it may brick you local chain_data.

That said, the following works for me -

../target/release/grin client resetchainhead 0001df6e93e6462b754d4616a51a0b65917c9492a259175cccbf74af2a990ddf
Successfully reset chain head 0001df6e93e6462b754d4616a51a0b65917c9492a259175cccbf74af2a990ddf

The node then transitions into "header sync" when the syncer process next runs (a few seconds later).

[antiochp]

the header is specified by its (cycle)hash?

Yes by hash.

what is a header extension? what does reapplying a header mean?

We interact with the header MMR via a "header extension". This allows us to rewind/truncate and append new data to the header MMR.

can this rewind beyond the horizon, if the node saw preceding blocks?

No sure yet. I have only experimented with rewinding within the horizon.
In theory we should be able to get this to work for larger rewinds.

[antiochp]

curl -X POST \
     -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","id":1,"method":"reset_chain_head","params":["0002574861b6931d24533549418541a86ae4f8efadc6e2dd23016a518ddee32d"]}' \
         http://localhost:3413/v2/owner

{
  "id": 1,
  "jsonrpc": "2.0",
  "result": {
    "Ok": null
  }
}%

[antiochp]

../target/release/grin client resetchainhead 0002574861b6931d24533549418541a86ae4f8efadc6e2dd23016a518ddee32d

Successfully reset chain head 0002574861b6931d24533549418541a86ae4f8efadc6e2dd23016a518ddee32d

[antiochp]

We can now do the following -

• grin client invalidateheader 0002691b87125bb45aa76e54543d9375f2047b10ca1ee674dea22a890447e262
• grin client resetchainhead 0000b9e51d5733d778a0a13f2c85e843863db0ea4ea6a4b0b358873636dc5db3

We maintain a local set of invalid headers (denylist).
This does not persist across node restart to avoid getting a node into a persistent bad state.

We can simply invalidate a particular chain fork by identifying a header on the fork and invalidating it via invalidateheader.
This header(s) will be treated as invalid and will result in a peer ban.

By resetting chain state to an earlier header via resetchainhead we can then force the local node onto an alternative fork.
This assumes at least one of our peers is advertising the alternative fork that does not involve an invalid header.
Other peers that advertise a chain head on a fork with an invalid header will be banned over time.

A simple node restart will clear out the current "denylist". This should only ever be a temporary measure to help the local node navigate to the correct chain fork. Once on the correct "winning" fork the local "denylist" can be safely discarded.

[phyro]

is this one ready for assignees?

[antiochp]

Going to merge this shortly to master.