From 368c7bc0f66e508a23a316221fb12a746f64ff18 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Tue, 24 Sep 2019 14:39:53 +0200 Subject: [PATCH 01/29] Update storage.md --- docs/conceptual/core/storage.md | 9 --------- docs/conceptual/node/storage.md | 29 +++++++++++++++++++++++++++++ 2 files changed, 29 insertions(+), 9 deletions(-) delete mode 100644 docs/conceptual/core/storage.md create mode 100644 docs/conceptual/node/storage.md diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md deleted file mode 100644 index 6a108073ca..0000000000 --- a/docs/conceptual/core/storage.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -title: Storage ---- - -* Key Value Database, RocksDB - -* Trie abstraction - -* Runtime Storage abstraction diff --git a/docs/conceptual/node/storage.md b/docs/conceptual/node/storage.md new file mode 100644 index 0000000000..ae582c00be --- /dev/null +++ b/docs/conceptual/node/storage.md @@ -0,0 +1,29 @@ +--- +title: Storage +--- + +The Substrate uses a simple key-value storage abstraction to represent the client database. + +## Key-Value Store + +Substrate implements it's storage database with RocksDB, a persistent key-value store for fast storage environments. + +This is used for all the components of Substrate which require persistent storage such as: + +* Substrate clients +* Substrate light-clients +* Off-chain workers + +## Abstractions + +One advantage of using a simple key-value store is that you are able to easily abstract other storage structures on top. + +## Trie Abstraction + +Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") based on [`paritytech/trie`](https://github.com/paritytech/trie). + +TODO + +## Runtime Storage abstraction + + From ac6d78b72582fe9780614917d6d93f5f780b84cd Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Tue, 24 Sep 2019 16:24:28 +0200 Subject: [PATCH 02/29] skeleton of storage doc --- docs/conceptual/node/storage.md | 36 +++++++++++++++++++++++------- docs/development/module/storage.md | 2 +- 2 files changed, 29 insertions(+), 9 deletions(-) diff --git a/docs/conceptual/node/storage.md b/docs/conceptual/node/storage.md index ae582c00be..a5a9108ea9 100644 --- a/docs/conceptual/node/storage.md +++ b/docs/conceptual/node/storage.md @@ -2,11 +2,11 @@ title: Storage --- -The Substrate uses a simple key-value storage abstraction to represent the client database. +The Substrate uses a simple a key-value data store implemented as a database-backed modified Merkle tree. -## Key-Value Store +## Key-Value Database -Substrate implements it's storage database with RocksDB, a persistent key-value store for fast storage environments. +Substrate implements it's storage database with [RocksDB](https://rocksdb.org/)., a persistent key-value store for fast storage environments. This is used for all the components of Substrate which require persistent storage such as: @@ -14,16 +14,36 @@ This is used for all the components of Substrate which require persistent storag * Substrate light-clients * Off-chain workers -## Abstractions +## Trie Abstraction One advantage of using a simple key-value store is that you are able to easily abstract other storage structures on top. -## Trie Abstraction +Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") from [`paritytech/trie`](https://github.com/paritytech/trie) to provide a trie structure whose contents can be modified and whose root hash is recalculated efficiently. + +### Main Trie + +Substrate has a single main trie whose storage root hash is placed in the block header. This is used to easily verify the state of the blockchain and provide a basis for light clients to verify proofs. + +### Child Trie + +Substrate also provides an API to generate new child tries with their own root hash that can be used in the runtime. + +The root hash of these child tries are automatically stored in the main trie to maintain easy verification of the complete node state. + +## Runtime Storage API + +The Substrate runtime support library provides utilities which generates unique, deterministic keys for your runtime module storage items. These storage items are placed in the main trie and are accessible by querying the trie by key. + +## Next Steps + +### Learn More -Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") based on [`paritytech/trie`](https://github.com/paritytech/trie). +* Learn how to add [storage items](development/module/storage.md) into your Substrate runtime modules. -TODO +### Examples -## Runtime Storage abstraction +* View an example of creating [child tries]() in your Substrate runtime module. +### References +* Visit the reference docs for [`paritytech/trie`](https://substrate.dev/rustdocs/master/trie_db/trait.Trie.html). \ No newline at end of file diff --git a/docs/development/module/storage.md b/docs/development/module/storage.md index 032da5ac4c..d992a3e51f 100644 --- a/docs/development/module/storage.md +++ b/docs/development/module/storage.md @@ -67,7 +67,7 @@ TODO ### Examples -TODO +* View this example to see how you can use a `double_map` to act as a `killable` single-map. ### References From 09fb5ddcc3c2b15f9956a87b0b03db8a60b616b1 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Tue, 24 Sep 2019 16:27:53 +0200 Subject: [PATCH 03/29] typo --- docs/conceptual/node/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/node/storage.md b/docs/conceptual/node/storage.md index a5a9108ea9..6a91ff514a 100644 --- a/docs/conceptual/node/storage.md +++ b/docs/conceptual/node/storage.md @@ -6,7 +6,7 @@ The Substrate uses a simple a key-value data store implemented as a database-bac ## Key-Value Database -Substrate implements it's storage database with [RocksDB](https://rocksdb.org/)., a persistent key-value store for fast storage environments. +Substrate implements it's storage database with [RocksDB](https://rocksdb.org/), a persistent key-value store for fast storage environments. This is used for all the components of Substrate which require persistent storage such as: From 7b8b959937a8b7d080463de87bd066fdea1f2aad Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Tue, 24 Sep 2019 16:29:38 +0200 Subject: [PATCH 04/29] Update storage.md --- docs/conceptual/{node => core}/storage.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/conceptual/{node => core}/storage.md (100%) diff --git a/docs/conceptual/node/storage.md b/docs/conceptual/core/storage.md similarity index 100% rename from docs/conceptual/node/storage.md rename to docs/conceptual/core/storage.md From 97e85f55a3fc1c2fa186b6db9c23dc49b32cb301 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Fri, 27 Sep 2019 01:35:17 +0200 Subject: [PATCH 05/29] Integrate feedback --- docs/conceptual/core/storage.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 6a91ff514a..c3c8d05842 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -20,15 +20,21 @@ One advantage of using a simple key-value store is that you are able to easily a Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") from [`paritytech/trie`](https://github.com/paritytech/trie) to provide a trie structure whose contents can be modified and whose root hash is recalculated efficiently. -### Main Trie +Tries are important tool for blockchains because the allows for efficient storing and sharing of historical of block state. You don't have a state trie per block, but a trie hash that will point to the nodes from previous block states. However, accessing data to trie data is costly. Each read operation takes O(log N) time, where N is the number of elements stored in the trie. To mitigate this, we use a key value cache. -Substrate has a single main trie whose storage root hash is placed in the block header. This is used to easily verify the state of the blockchain and provide a basis for light clients to verify proofs. +All trie node are stored in RocksDB and part of the trie state can get pruned, i.e. a key-value pair can be deleted from the storage when it is out of pruning range for non archive nodes. Trie nodes are also encoded to allow for any key type, while still be able to avoid key collision. We do not use [reference counting](http://en.wikipedia.org/wiki/Reference_counting) for performance reasons. + +### State Trie + +Substrate has a single main trie, called the state trie, whose changing root hash is placed in each block header. This is used to easily verify the state of the blockchain and provide a basis for light clients to verify proofs. + +This trie only stores content for the canonical chain, not forks. There is a separate `state_db` layer that maintain the trie state with references counted in memory for all that is non canonical. See JournalDB. ### Child Trie Substrate also provides an API to generate new child tries with their own root hash that can be used in the runtime. -The root hash of these child tries are automatically stored in the main trie to maintain easy verification of the complete node state. +Child tries are identical to main state trie, except their root is stored and updated in the main trie instead of the block header. Since their headers are a part of the main state trie, it is still easy to verify of the complete node state when it includes child tries. ## Runtime Storage API From 402bba0cc28dc4b21b920108c2cb66711b6de9cd Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Fri, 27 Sep 2019 08:34:38 +0100 Subject: [PATCH 06/29] fixes --- docs/conceptual/core/storage.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index c3c8d05842..d5a2a29249 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -6,7 +6,7 @@ The Substrate uses a simple a key-value data store implemented as a database-bac ## Key-Value Database -Substrate implements it's storage database with [RocksDB](https://rocksdb.org/), a persistent key-value store for fast storage environments. +Substrate implements its storage database with [RocksDB](https://rocksdb.org/), a persistent key-value store for fast storage environments. This is used for all the components of Substrate which require persistent storage such as: @@ -20,9 +20,9 @@ One advantage of using a simple key-value store is that you are able to easily a Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") from [`paritytech/trie`](https://github.com/paritytech/trie) to provide a trie structure whose contents can be modified and whose root hash is recalculated efficiently. -Tries are important tool for blockchains because the allows for efficient storing and sharing of historical of block state. You don't have a state trie per block, but a trie hash that will point to the nodes from previous block states. However, accessing data to trie data is costly. Each read operation takes O(log N) time, where N is the number of elements stored in the trie. To mitigate this, we use a key value cache. +Tries are important tool for blockchains because they allow for efficient storing and sharing of historical of block state. You don't have a state trie per block, but a trie hash that will point to the nodes from previous block states. However, accessing data to trie data is costly. Each read operation takes O(log N) time, where N is the number of elements stored in the trie. To mitigate this, we use a key value cache. -All trie node are stored in RocksDB and part of the trie state can get pruned, i.e. a key-value pair can be deleted from the storage when it is out of pruning range for non archive nodes. Trie nodes are also encoded to allow for any key type, while still be able to avoid key collision. We do not use [reference counting](http://en.wikipedia.org/wiki/Reference_counting) for performance reasons. +All trie node are stored in RocksDB and part of the trie state can get pruned, i.e. a key-value pair can be deleted from the storage when it is out of pruning range for non archive nodes. The trie nodes partial path prefixes the hash of the encoded node for the RocksDB key to avoid key collision. We do not use [reference counting](http://en.wikipedia.org/wiki/Reference_counting) for performance reasons. ### State Trie From 89c720c2373456e8e6cea4a93e7b3e3eda0b6934 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Fri, 27 Sep 2019 08:38:47 +0100 Subject: [PATCH 07/29] clarify kind of node --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index d5a2a29249..daeafaeaed 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -20,7 +20,7 @@ One advantage of using a simple key-value store is that you are able to easily a Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") from [`paritytech/trie`](https://github.com/paritytech/trie) to provide a trie structure whose contents can be modified and whose root hash is recalculated efficiently. -Tries are important tool for blockchains because they allow for efficient storing and sharing of historical of block state. You don't have a state trie per block, but a trie hash that will point to the nodes from previous block states. However, accessing data to trie data is costly. Each read operation takes O(log N) time, where N is the number of elements stored in the trie. To mitigate this, we use a key value cache. +Tries are important tool for blockchains because they allow for efficient storing and sharing of historical of block state. You don't have a state trie per block, but a trie hash that will point to the trie nodes from previous block states. However, accessing data to trie data is costly. Each read operation takes O(log N) time, where N is the number of elements stored in the trie. To mitigate this, we use a key value cache. All trie node are stored in RocksDB and part of the trie state can get pruned, i.e. a key-value pair can be deleted from the storage when it is out of pruning range for non archive nodes. The trie nodes partial path prefixes the hash of the encoded node for the RocksDB key to avoid key collision. We do not use [reference counting](http://en.wikipedia.org/wiki/Reference_counting) for performance reasons. From d77a40b07b7be4c08e5f1a5f61e1faf0e643f1e6 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Fri, 27 Sep 2019 08:44:15 +0100 Subject: [PATCH 08/29] Clarify --- docs/conceptual/core/storage.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index daeafaeaed..0d489f65e5 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -20,7 +20,9 @@ One advantage of using a simple key-value store is that you are able to easily a Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") from [`paritytech/trie`](https://github.com/paritytech/trie) to provide a trie structure whose contents can be modified and whose root hash is recalculated efficiently. -Tries are important tool for blockchains because they allow for efficient storing and sharing of historical of block state. You don't have a state trie per block, but a trie hash that will point to the trie nodes from previous block states. However, accessing data to trie data is costly. Each read operation takes O(log N) time, where N is the number of elements stored in the trie. To mitigate this, we use a key value cache. +Tries are important tool for blockchains because they allow for efficient storing and sharing of historical of block state. Each block does not have it's own state trie, but a trie hash that will point to the trie nodes from previous block states. + +Accessing data to trie data is costly. Each read operation takes O(log N) time, where N is the number of elements stored in the trie. To mitigate this, we use a key value cache. All trie node are stored in RocksDB and part of the trie state can get pruned, i.e. a key-value pair can be deleted from the storage when it is out of pruning range for non archive nodes. The trie nodes partial path prefixes the hash of the encoded node for the RocksDB key to avoid key collision. We do not use [reference counting](http://en.wikipedia.org/wiki/Reference_counting) for performance reasons. From 999d39fc939c36b4a15604ff9d83eace14284bdd Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Fri, 27 Sep 2019 08:49:33 +0100 Subject: [PATCH 09/29] typo, fix line width --- docs/conceptual/core/storage.md | 67 +++++++++++++++++++++++---------- 1 file changed, 48 insertions(+), 19 deletions(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 0d489f65e5..70825690ac 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -2,56 +2,85 @@ title: Storage --- -The Substrate uses a simple a key-value data store implemented as a database-backed modified Merkle tree. +The Substrate uses a simple a key-value data store implemented as a +database-backed modified Merkle tree. ## Key-Value Database -Substrate implements its storage database with [RocksDB](https://rocksdb.org/), a persistent key-value store for fast storage environments. +Substrate implements its storage database with [RocksDB](https://rocksdb.org/), +a persistent key-value store for fast storage environments. -This is used for all the components of Substrate which require persistent storage such as: +This is used for all the components of Substrate which require persistent +storage such as: -* Substrate clients -* Substrate light-clients -* Off-chain workers +- Substrate clients +- Substrate light-clients +- Off-chain workers ## Trie Abstraction -One advantage of using a simple key-value store is that you are able to easily abstract other storage structures on top. +One advantage of using a simple key-value store is that you are able to easily +abstract other storage structures on top. -Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") from [`paritytech/trie`](https://github.com/paritytech/trie) to provide a trie structure whose contents can be modified and whose root hash is recalculated efficiently. +Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") from +[`paritytech/trie`](https://github.com/paritytech/trie) to provide a trie +structure whose contents can be modified and whose root hash is recalculated +efficiently. -Tries are important tool for blockchains because they allow for efficient storing and sharing of historical of block state. Each block does not have it's own state trie, but a trie hash that will point to the trie nodes from previous block states. +Tries are important tool for blockchains because they allow for efficient +storing and sharing of historical of block state. Each block does not have it's +own state trie, but a trie hash that will point to the trie nodes from previous +block states. -Accessing data to trie data is costly. Each read operation takes O(log N) time, where N is the number of elements stored in the trie. To mitigate this, we use a key value cache. +Accessing data to trie data is costly. Each read operation takes O(log N) time, +where N is the number of elements stored in the trie. To mitigate this, we use a +key value cache. -All trie node are stored in RocksDB and part of the trie state can get pruned, i.e. a key-value pair can be deleted from the storage when it is out of pruning range for non archive nodes. The trie nodes partial path prefixes the hash of the encoded node for the RocksDB key to avoid key collision. We do not use [reference counting](http://en.wikipedia.org/wiki/Reference_counting) for performance reasons. +All trie nodes are stored in RocksDB and part of the trie state can get pruned, +i.e. a key-value pair can be deleted from the storage when it is out of pruning +range for non archive nodes. The trie nodes partial path prefixes the hash of +the encoded node for the RocksDB key to avoid key collision. We do not use +[reference counting](http://en.wikipedia.org/wiki/Reference_counting) for +performance reasons. ### State Trie -Substrate has a single main trie, called the state trie, whose changing root hash is placed in each block header. This is used to easily verify the state of the blockchain and provide a basis for light clients to verify proofs. +Substrate has a single main trie, called the state trie, whose changing root +hash is placed in each block header. This is used to easily verify the state of +the blockchain and provide a basis for light clients to verify proofs. -This trie only stores content for the canonical chain, not forks. There is a separate `state_db` layer that maintain the trie state with references counted in memory for all that is non canonical. See JournalDB. +This trie only stores content for the canonical chain, not forks. There is a +separate `state_db` layer that maintain the trie state with references counted +in memory for all that is non canonical. See JournalDB. ### Child Trie -Substrate also provides an API to generate new child tries with their own root hash that can be used in the runtime. +Substrate also provides an API to generate new child tries with their own root +hash that can be used in the runtime. -Child tries are identical to main state trie, except their root is stored and updated in the main trie instead of the block header. Since their headers are a part of the main state trie, it is still easy to verify of the complete node state when it includes child tries. +Child tries are identical to main state trie, except their root is stored and +updated in the main trie instead of the block header. Since their headers are a +part of the main state trie, it is still easy to verify of the complete node +state when it includes child tries. ## Runtime Storage API -The Substrate runtime support library provides utilities which generates unique, deterministic keys for your runtime module storage items. These storage items are placed in the main trie and are accessible by querying the trie by key. +The Substrate runtime support library provides utilities which generates unique, +deterministic keys for your runtime module storage items. These storage items +are placed in the main trie and are accessible by querying the trie by key. ## Next Steps ### Learn More -* Learn how to add [storage items](development/module/storage.md) into your Substrate runtime modules. +- Learn how to add [storage items](development/module/storage.md) into your + Substrate runtime modules. ### Examples -* View an example of creating [child tries]() in your Substrate runtime module. +- View an example of creating [child tries]() in your Substrate runtime module. ### References -* Visit the reference docs for [`paritytech/trie`](https://substrate.dev/rustdocs/master/trie_db/trait.Trie.html). \ No newline at end of file +- Visit the reference docs for + [`paritytech/trie`](https://substrate.dev/rustdocs/master/trie_db/trait.Trie.html). From 68a8592de597477e6b0467b0aa16d4f3b15e0715 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Fri, 27 Sep 2019 09:15:12 +0100 Subject: [PATCH 10/29] fixes --- docs/conceptual/core/storage.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 70825690ac..b6f118659e 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -28,11 +28,11 @@ structure whose contents can be modified and whose root hash is recalculated efficiently. Tries are important tool for blockchains because they allow for efficient -storing and sharing of historical of block state. Each block does not have it's +storing and sharing of historical of block state. Each block does not have its own state trie, but a trie hash that will point to the trie nodes from previous block states. -Accessing data to trie data is costly. Each read operation takes O(log N) time, +Accessing trie data is costly. Each read operation takes O(log N) time, where N is the number of elements stored in the trie. To mitigate this, we use a key value cache. @@ -67,7 +67,7 @@ state when it includes child tries. The Substrate runtime support library provides utilities which generates unique, deterministic keys for your runtime module storage items. These storage items -are placed in the main trie and are accessible by querying the trie by key. +are placed in the state trie and are accessible by querying the trie by key. ## Next Steps From 5bbead7e6afe0ca12de834df68a29302400b109f Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Fri, 27 Sep 2019 09:23:42 +0100 Subject: [PATCH 11/29] replace misleading data --- docs/conceptual/core/storage.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index b6f118659e..a2e722b36d 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -28,13 +28,15 @@ structure whose contents can be modified and whose root hash is recalculated efficiently. Tries are important tool for blockchains because they allow for efficient -storing and sharing of historical of block state. Each block does not have its -own state trie, but a trie hash that will point to the trie nodes from previous -block states. - -Accessing trie data is costly. Each read operation takes O(log N) time, -where N is the number of elements stored in the trie. To mitigate this, we use a -key value cache. +storing and sharing of the historical block state. The root hash of the trie +provides a cryptographic fingerprint of the final resulting storage after +executing all state transitions by a blockchain. Thus, two blockchain nodes can +easily verify they have the same final state by simply comparing their trie +root. + +Accessing trie data is costly. Each read operation takes O(log N) time, where N +is the number of elements stored in the trie. To mitigate this, we use a key +value cache. All trie nodes are stored in RocksDB and part of the trie state can get pruned, i.e. a key-value pair can be deleted from the storage when it is out of pruning From ec323ac73efb303eb1efd52cb77a9ca9c8349c4f Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 02:09:12 +0200 Subject: [PATCH 12/29] Update docs/conceptual/core/storage.md Co-Authored-By: cheme --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index a2e722b36d..8ad1248221 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -20,7 +20,7 @@ storage such as: ## Trie Abstraction One advantage of using a simple key-value store is that you are able to easily -abstract other storage structures on top. +abstract storage structures on top of it. Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") from [`paritytech/trie`](https://github.com/paritytech/trie) to provide a trie From 5813b887b33234cc01afd20abf59a420f8572301 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 02:10:34 +0200 Subject: [PATCH 13/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 8ad1248221..b815668c1f 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -10,7 +10,7 @@ database-backed modified Merkle tree. Substrate implements its storage database with [RocksDB](https://rocksdb.org/), a persistent key-value store for fast storage environments. -This is used for all the components of Substrate which require persistent +This is used for all the components of Substrate that require persistent storage such as: - Substrate clients From d6d72922dc52bace3db11f054bb2d14aa41df202 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 02:10:42 +0200 Subject: [PATCH 14/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index b815668c1f..761411522c 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -27,7 +27,7 @@ Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") from structure whose contents can be modified and whose root hash is recalculated efficiently. -Tries are important tool for blockchains because they allow for efficient +Tries allow efficient storing and sharing of the historical block state. The root hash of the trie provides a cryptographic fingerprint of the final resulting storage after executing all state transitions by a blockchain. Thus, two blockchain nodes can From 1c5993e7a579a0768f2368dd76c8d8040cbf74dc Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 02:20:29 +0200 Subject: [PATCH 15/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 761411522c..7b4eca0ec9 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -67,7 +67,7 @@ state when it includes child tries. ## Runtime Storage API -The Substrate runtime support library provides utilities which generates unique, +The Substrate runtime support library provides utilities to generate unique, deterministic keys for your runtime module storage items. These storage items are placed in the state trie and are accessible by querying the trie by key. From ab8caada9ebc87fde56206e7f8885907dc9f8266 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 01:22:46 +0100 Subject: [PATCH 16/29] update from feedback --- docs/conceptual/core/storage.md | 34 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 18 deletions(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 761411522c..035f9898d4 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -2,16 +2,16 @@ title: Storage --- -The Substrate uses a simple a key-value data store implemented as a -database-backed modified Merkle tree. +Substrate uses a simple a key-value data store implemented as a database-backed +modified Merkle tree. ## Key-Value Database Substrate implements its storage database with [RocksDB](https://rocksdb.org/), a persistent key-value store for fast storage environments. -This is used for all the components of Substrate that require persistent -storage such as: +This is used for all the components of Substrate that require persistent storage +such as: - Substrate clients - Substrate light-clients @@ -27,11 +27,10 @@ Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") from structure whose contents can be modified and whose root hash is recalculated efficiently. -Tries allow efficient -storing and sharing of the historical block state. The root hash of the trie -provides a cryptographic fingerprint of the final resulting storage after -executing all state transitions by a blockchain. Thus, two blockchain nodes can -easily verify they have the same final state by simply comparing their trie +Tries allow efficient storing and sharing of the historical block state. The +trie root is a representation of the data within the trie; that is, two tries +with different data will always have different roots. Thus, two blockchain nodes +can easily verify they have the same final state by simply comparing their trie root. Accessing trie data is costly. Each read operation takes O(log N) time, where N @@ -40,16 +39,15 @@ value cache. All trie nodes are stored in RocksDB and part of the trie state can get pruned, i.e. a key-value pair can be deleted from the storage when it is out of pruning -range for non archive nodes. The trie nodes partial path prefixes the hash of -the encoded node for the RocksDB key to avoid key collision. We do not use -[reference counting](http://en.wikipedia.org/wiki/Reference_counting) for -performance reasons. +range for non archive nodes. We do not use [reference +counting](http://en.wikipedia.org/wiki/Reference_counting) for performance +reasons. ### State Trie -Substrate has a single main trie, called the state trie, whose changing root -hash is placed in each block header. This is used to easily verify the state of -the blockchain and provide a basis for light clients to verify proofs. +Substrate based chains have a single main trie, called the state trie, whose +root hash is placed in each block header. This is used to easily verify the +state of the blockchain and provide a basis for light clients to verify proofs. This trie only stores content for the canonical chain, not forks. There is a separate `state_db` layer that maintain the trie state with references counted @@ -62,8 +60,8 @@ hash that can be used in the runtime. Child tries are identical to main state trie, except their root is stored and updated in the main trie instead of the block header. Since their headers are a -part of the main state trie, it is still easy to verify of the complete node -state when it includes child tries. +part of the main state trie, it is still easy to verify the complete node state +when it includes child tries. ## Runtime Storage API From f4570ee4b50eb83dba096d0558d34e4f69ce3847 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 01:33:49 +0100 Subject: [PATCH 17/29] Add a section on why to use child tries --- docs/conceptual/core/storage.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 45de187420..11da4389c7 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -63,6 +63,11 @@ updated in the main trie instead of the block header. Since their headers are a part of the main state trie, it is still easy to verify the complete node state when it includes child tries. +Child tries are useful when you want your own independent trie with a separate +root hash that you can use to verify the specific content in that trie. +Subsections of a trie do not have a root-hash-like representation that satisfy +these needs automatically; thus a child trie is used instead. + ## Runtime Storage API The Substrate runtime support library provides utilities to generate unique, From 9db4f54ec41210ce3136dee7b4b9d7bdc037044d Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 23:27:09 +0100 Subject: [PATCH 18/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 11da4389c7..c5ae72a888 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -2,7 +2,7 @@ title: Storage --- -Substrate uses a simple a key-value data store implemented as a database-backed +Substrate uses a simple key-value data store implemented as a database-backed, modified Merkle tree. ## Key-Value Database From 23f725b106b10fb4fdd0d6dfdcfddedfc7ec3eb4 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 23:27:24 +0100 Subject: [PATCH 19/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index c5ae72a888..0fff5831ec 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -10,7 +10,7 @@ modified Merkle tree. Substrate implements its storage database with [RocksDB](https://rocksdb.org/), a persistent key-value store for fast storage environments. -This is used for all the components of Substrate that require persistent storage +This is used for all the components of Substrate that require persistent storage, such as: - Substrate clients From 59f529a227dfb2a4db4717267ebc6066032eab68 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 23:28:02 +0100 Subject: [PATCH 20/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 0fff5831ec..37e918b977 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -30,7 +30,7 @@ efficiently. Tries allow efficient storing and sharing of the historical block state. The trie root is a representation of the data within the trie; that is, two tries with different data will always have different roots. Thus, two blockchain nodes -can easily verify they have the same final state by simply comparing their trie +can easily verify that they have the same state by simply comparing their trie root. Accessing trie data is costly. Each read operation takes O(log N) time, where N From 07026a36a6489f21b3802afba07468e5fcbcead5 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 23:28:18 +0100 Subject: [PATCH 21/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 37e918b977..8358e5be51 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -31,7 +31,7 @@ Tries allow efficient storing and sharing of the historical block state. The trie root is a representation of the data within the trie; that is, two tries with different data will always have different roots. Thus, two blockchain nodes can easily verify that they have the same state by simply comparing their trie -root. +roots. Accessing trie data is costly. Each read operation takes O(log N) time, where N is the number of elements stored in the trie. To mitigate this, we use a key From 9240c4865531318274b25dd44a4bd8e134da9e82 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 23:28:56 +0100 Subject: [PATCH 22/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 8358e5be51..c583e24595 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -35,7 +35,8 @@ roots. Accessing trie data is costly. Each read operation takes O(log N) time, where N is the number of elements stored in the trie. To mitigate this, we use a key -value cache. +is the number of elements stored in the trie. To mitigate this, we use a +key-value cache. All trie nodes are stored in RocksDB and part of the trie state can get pruned, i.e. a key-value pair can be deleted from the storage when it is out of pruning From 2cc0216bb1a3e52aef45df70224480f2d344da34 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 23:29:08 +0100 Subject: [PATCH 23/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index c583e24595..5d13be666f 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -57,7 +57,7 @@ in memory for all that is non canonical. See JournalDB. ### Child Trie Substrate also provides an API to generate new child tries with their own root -hash that can be used in the runtime. +hashes that can be used in the runtime. Child tries are identical to main state trie, except their root is stored and updated in the main trie instead of the block header. Since their headers are a From 9cb7da5e1c1ea5dc690e0e94bc3cc6c3abcf2d00 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 23:30:31 +0100 Subject: [PATCH 24/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 5d13be666f..8b60f82952 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -51,7 +51,7 @@ root hash is placed in each block header. This is used to easily verify the state of the blockchain and provide a basis for light clients to verify proofs. This trie only stores content for the canonical chain, not forks. There is a -separate `state_db` layer that maintain the trie state with references counted +separate `state_db` layer that maintains the trie state with references counted in memory for all that is non canonical. See JournalDB. ### Child Trie From df675414b96253194e6b5ba3ee45997bf5d18219 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 23:30:48 +0100 Subject: [PATCH 25/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 8b60f82952..f92f6d2d1d 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -40,7 +40,7 @@ key-value cache. All trie nodes are stored in RocksDB and part of the trie state can get pruned, i.e. a key-value pair can be deleted from the storage when it is out of pruning -range for non archive nodes. We do not use [reference +range for non-archive nodes. We do not use [reference counting](http://en.wikipedia.org/wiki/Reference_counting) for performance reasons. From 96fee72c7e1ed8d32f8154fc09013fe05d31e357 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 23:30:58 +0100 Subject: [PATCH 26/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index f92f6d2d1d..e8f033c7aa 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -52,7 +52,7 @@ state of the blockchain and provide a basis for light clients to verify proofs. This trie only stores content for the canonical chain, not forks. There is a separate `state_db` layer that maintains the trie state with references counted -in memory for all that is non canonical. See JournalDB. +in memory for all that is non-canonical. See JournalDB. ### Child Trie From 330d8a193a28def70bf6bd8bebf5ba092307ef31 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 23:31:15 +0100 Subject: [PATCH 27/29] Update docs/conceptual/core/storage.md Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index e8f033c7aa..ac00aa35d7 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -59,7 +59,7 @@ in memory for all that is non-canonical. See JournalDB. Substrate also provides an API to generate new child tries with their own root hashes that can be used in the runtime. -Child tries are identical to main state trie, except their root is stored and +Child tries are identical to the main state trie, except that a child trie's root is stored and updated in the main trie instead of the block header. Since their headers are a part of the main state trie, it is still easy to verify the complete node state when it includes child tries. From 72550f59f87ecd96277c19fcb6a3262cb807ae45 Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 23:41:36 +0100 Subject: [PATCH 28/29] Final fixes --- docs/conceptual/core/storage.md | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index ac00aa35d7..660acad285 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -10,8 +10,8 @@ modified Merkle tree. Substrate implements its storage database with [RocksDB](https://rocksdb.org/), a persistent key-value store for fast storage environments. -This is used for all the components of Substrate that require persistent storage, -such as: +This is used for all the components of Substrate that require persistent +storage, such as: - Substrate clients - Substrate light-clients @@ -34,7 +34,6 @@ can easily verify that they have the same state by simply comparing their trie roots. Accessing trie data is costly. Each read operation takes O(log N) time, where N -is the number of elements stored in the trie. To mitigate this, we use a key is the number of elements stored in the trie. To mitigate this, we use a key-value cache. @@ -51,18 +50,20 @@ root hash is placed in each block header. This is used to easily verify the state of the blockchain and provide a basis for light clients to verify proofs. This trie only stores content for the canonical chain, not forks. There is a -separate `state_db` layer that maintains the trie state with references counted -in memory for all that is non-canonical. See JournalDB. +separate [`state_db` +layer](https://substrate.dev/rustdocs/master/substrate_state_db/index.html) that +maintains the trie state with references counted in memory for all that is +non-canonical. ### Child Trie Substrate also provides an API to generate new child tries with their own root hashes that can be used in the runtime. -Child tries are identical to the main state trie, except that a child trie's root is stored and -updated in the main trie instead of the block header. Since their headers are a -part of the main state trie, it is still easy to verify the complete node state -when it includes child tries. +Child tries are identical to the main state trie, except that a child trie's +root is stored and updated as a node in the main trie instead of the block +header. Since their headers are a part of the main state trie, it is still easy +to verify the complete node state when it includes child tries. Child tries are useful when you want your own independent trie with a separate root hash that you can use to verify the specific content in that trie. @@ -71,9 +72,11 @@ these needs automatically; thus a child trie is used instead. ## Runtime Storage API -The Substrate runtime support library provides utilities to generate unique, -deterministic keys for your runtime module storage items. These storage items -are placed in the state trie and are accessible by querying the trie by key. +The Substrate's [Support +module](https://substrate.dev/rustdocs/master/srml_support/index.html) provides +utilities to generate unique, deterministic keys for your runtime module storage +items. These storage items are placed in the state trie and are accessible by +querying the trie by key. ## Next Steps From 2018486bf4acfaeebc619e2867ba7d690982858d Mon Sep 17 00:00:00 2001 From: Shawn Tabrizi Date: Sat, 28 Sep 2019 23:41:49 +0100 Subject: [PATCH 29/29] add todo --- docs/conceptual/core/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conceptual/core/storage.md b/docs/conceptual/core/storage.md index 660acad285..f811eddb3e 100644 --- a/docs/conceptual/core/storage.md +++ b/docs/conceptual/core/storage.md @@ -87,7 +87,7 @@ querying the trie by key. ### Examples -- View an example of creating [child tries]() in your Substrate runtime module. +- View an example of creating [child tries](TODO) in your Substrate runtime module. ### References