Problem: No BEP discussing the UTXO implementation #20
Conversation
Solution: Write an informational BEP with a brief discussion of current UTXO implementation.
utxo-info/README.md
Outdated
| @@ -0,0 +1,19 @@ | |||
| ``` | |||
| shortname: ?/UTXO-IMPL | |||
| name: A short discription of the current UTXO implementation | |||
utxo-info/README.md
Outdated
| ``` | ||
|
|
||
| ## Description | ||
| The UTXO (unspent transaction output) of each transaction is tracked using the a merkle tree. The merkle root of this tree is used as the app hash and retuned to Tendermint during `commit`. Below is a short summary of the UTXO is calculated, |
There was a problem hiding this comment.
It's not clear how all the outputs (which are like floating independent dots) are somehow connected to form a Merkle tree. Maybe draw a picture? They say pictures are worth 1000 words and in this case I think it's true.
utxo-info/README.md
Outdated
| ``` | ||
|
|
||
| ## Description | ||
| The UTXO (unspent transaction output) of each transaction is tracked using the a merkle tree. The merkle root of this tree is used as the app hash and retuned to Tendermint during `commit`. Below is a short summary of the UTXO is calculated, |
There was a problem hiding this comment.
Maybe link to some articles about Merkle trees.
https://en.wikipedia.org/wiki/Merkle_tree
What about AVL trees, IAVL trees, and IAVL+ trees. I gather that those are supposed to be useful somehow, as an improvement. Is it the rules for adding the next leaf UTXO, where to put it?
There was a problem hiding this comment.
@ttmc I have restrained the context of this BEP to just discussing the current implementation. Also, the leaves of the UTXO merkle tree are added to the list of the existing leaves and the list is sorted after which merkle root is calculated. As explained in the BEP, the leaves are added to the utxo collection and then all the leaves are read back and sorted, after which the root is calculated.
Solution: Add more details.
utxo-info/README.md
Outdated
| - This list of unspent outputs fetched from the `utxo` collection is considered as the leaves of the UTXO merkle tree. Refer the image below, | ||
| - The list of leaves is hashed and sorted after which the merkle root is calculated by pairing adjacent leaves. | ||
|
|
||
|  |
utxo-info/README.md
Outdated
|
|
||
|  | ||
|
|
||
|
|
There was a problem hiding this comment.
It's still not clear to me where one puts a new leaf node (UTXO). There are many possible spots to choose from. And it matters. The choice will affect the final hash. It seems that's where the AVL rules come into use. Is this right?
There was a problem hiding this comment.
Okay, lets take a step back. The unspent outputs or UTXO is stored in the utxo collection in MongoDB (there is no insert order here). When the merkle root is calculated all the leaves are fetched from the utxo collection, the leaves are hashed (lets call this H), then this hash list is sorted (Hs) and the merkle root is caculated by pairing the leaves. So if a new leaf is inserted we don't particularly insert it to specific position in the tree, its position among leaves will be determinted by the sort function which sorts the list of hashes H.
There was a problem hiding this comment.
Hmm, seems like we are relying on some mysterious sort function. It would be nice to know what it does and how it works. Can we reliably get a nice balanced tree doing this?
There was a problem hiding this comment.
Oh wow, rebuilding the whole tree from scratch every time will make it balanced, but that's a real waste of computing time and power. It's okay for now, but we need to do it smart in the future, using a proper IAVL+ library. That library should have a method like tree.insert_node(new_node) and then there will be other methods to compute the Merkle root of the tree (i.e. the app hash. That will also be fast, because most of those hash computations were done before and are stored somewhere handy).
There was a problem hiding this comment.
@ttmc You are right. We need to have some discussions around how to improve this implementation and make it crash recovery friendly.
Solution: Fix spelling
|
Should we move it to an issue and label it with BEP idea? |
|
I will change the label on this pull request from "new BEP" to "BEP idea" but I think it's fine to leave it as a PR, so as to preserve its history in one place. |
Solution: Write an informational BEP with a brief discussion of current UTXO implementation.