Discussion: Add canonical status for all blocks in archive node

[garethtdavies]

Anyone working with the archive node has to deal with recursive queries to determine the canonical chain. The archive node now processes blocks older than 290, and they are marked as canonical / orphaned but could this be implemented for all blocks up to the current chain tip?

Clearly, any blocks less than 290 from the current height are subject to a reorg, and so there would be overhead in implementing this - but this could potentially dramatically improve the usability of the archive node if blocks had a canonical state without invoking a recursive query. It would also make integrating it into services that expose the archive data easier to deliver.

[duccas]

I fully support the proposal. We also consider the implemented function of determining the canonical chain in the archive node to be absolutely inapplicable.
This was our first and most important task when creating the explorer, because we chose the option of using the archive node and not GraphQL.
And as Gareth said, this requires a rather complex recursive query which wastes resources. 290 blocks detection delay is too big and no one will use it.

[psteckler]

How much overhead do you see from performing a recursive query? Hard numbers, if you have them.

[garethtdavies]

I suppose marking as best_tip would also work if really want to keep the distinction of finalized blocks as a simple chain_status = 'canonical' OR chain_status = 'best_tip' would work and remove the need for a recursive query.

[jenr24G]

Hi! I've written a rust program that builds a graph of blocks within the transition frontier, and I'm working on applying various heuristics to the graph to determine what is most likely the canonical chain. I don't think it will be possible to say for 100% sure what the canonical chain is, but my program will be able to give a high level of certainty, and an eventual goal is to be able to give a probablility for a certain block either being canonical or not. I think this solution addresses the underlying problem of this discussion, and I'm wondering how people see this as connecting to the existing archive node solution.

[garethtdavies]

I might be misunderstanding this without looking at it, but we have an algorithm, so all block producers agree on the same current tip, roughly:

• Select the longest chain
• If tied, i.e., multiple blocks proposed at the same height, select the "best" VRF of the block winner
• If multiple blocks with the same VRF, i.e., multiple blocks from the same producer, compare the hash of the state hash and select the "best".

My suggestion would be to replicate this in the archive node, then once we know the current tip, we can simply walk back the chain using the previous state hash, updating any pending blocks. Nothing is finalized until 290 blocks have been built atop it, but the probability of a block being canonical will simply be based on its number of confirmations. As per this page https://docs.minaprotocol.com/node-operators/lifecycle-of-a-payment:

Finality (in blocks)	Average time for finality	Finality confidence (%)
8	33 mins	98.6709
15	60 mins	99.9231
23	1hr 32mins	99.9965
30	2hrs	99.9998
38	2hrs 32mins	100

[jenr24G]

My program replicates that functionality to the longest chain bullet, and I'm working on the VRF comparison now. Do you think it would be better to put this functionality in the current archive process, or would a separate dedicated process work as well? Also, what does the "best" hash of the state hash mean? This has been a very enlightening discussion, thanks a bunch for taking time out of your day to respond!

[garethtdavies]

I think ideally, in the archive process, labelling as such in the database such that it can be queried directly.

The comparing hashes of state hashes is for breaking the tie at the current tip when there is more than 1 block from the same producer (identical ones will simply be rejected). This is extremely common, with most producers running multiple block producing nodes and basically, this is so everyone can agree on the current tip. It was implemented here #6983, as not 100% sure how this comparison is made.

For the VRF comparison, you might be interested in https://github.com/zkvalidator/mina-vrf-rs as this is in Rust.