From bfb0d2dbcb600d6e3d0a687d2ed938f0da745f11 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bastian=20K=C3=B6cher?= Date: Tue, 28 Feb 2023 23:41:00 +0100 Subject: [PATCH 1/3] More fine grained control of `clear_prefix` and `storage_kill` from the runtime --- ...vironment-clear-prefix-and-storage-kill.md | 178 ++++++++++++++++++ 1 file changed, 178 insertions(+) create mode 100644 proposals/0000-runtime-environment-clear-prefix-and-storage-kill.md diff --git a/proposals/0000-runtime-environment-clear-prefix-and-storage-kill.md b/proposals/0000-runtime-environment-clear-prefix-and-storage-kill.md new file mode 100644 index 0000000..27179bd --- /dev/null +++ b/proposals/0000-runtime-environment-clear-prefix-and-storage-kill.md @@ -0,0 +1,178 @@ +--- +Title: More fine grained control of `clear_prefix` and `storage_kill` from the runtime +Number: 0 +Status: Proposed +Authors: + - Bastian Köcher +Created: 2023-02-27 +Category: Runtime Environment +Requires: +Replaces: +--- + +## Summary + +Give the runtime more fine grained control over the internals of `clear_prefix` and `storage_kill`. +We are proposing to introduce `ext_Storage_clear_prefix_version_3`, `ext_DefaultChildStorage_clear_prefix_version_3` +and `ext_DefaultChildStorage_storage_kill_version_3`. Notable changes are the introduction of a new +third parameter `maybe_cursor` and the change of the return value to `MultiRemovalResults`. +The `maybe_cursor` enables a runtime developer to start removing storage items from +a defined position. The new return value will give more details on how many items were +removed from the state and how many from the overlay etc. + +## Motivation + +Currently when a runtime developer calls `clear_prefix`/`kill_storage` with the +same set of parameters in the same context it leads to the same keys being deleted. +This can be quite confusing for runtime developers who are for example calling `clear_prefix` +with some `limit` multiple times and expecting `limit * number of calls` keys to be deleted. +While the actual result is that all calls remove exactly the same keys. They also don't get +any information on how many keys got deleted in the state and how got deleted in the overlay. + +## Detailed Solution design + +Introduce the following new host functions: + +```rust +fn ext_Storage_clear_prefix_version_3(prefix: u64, maybe_limit: u32, maybe_cursor: u64) -> u64; +fn ext_DefaultChildStorage_clear_prefix_version_3(storage_key: u64, prefix: u64, maybe_limit: u32, maybe_cursor: u64) -> u64; +fn ext_DefaultChildStorage_storage_kill_version_4(storage_key: u64, maybe_limit: u32, maybe_cursor: u64) -> u64; +``` + +The following list explains the parameters and the return value of all functions above: + +- `prefix` is a packed `(ptr | length)` pointing to a memory area that must +be interpreted as a list of bytes. All keys that are going to be deleted must +start with `prefix`. +- `maybe_limit` must be interpreted as `unsigned 32 bits`. This +limit is either `2^32 - 1` which corresponds to no limit at all or to some other +number which should be taken as maximum number of keys to delete. +- `maybe_cursor` is a packed `(ptr | length)` pointing to a memory +area that must be interpreted as a list of bytes. Must be used as `cursor` if +length is greater than `0`. The `cursor` will be the first key to delete. +- `storage_key` is a packed `(ptr | length)` pointing to a memory +area that must be interpreted as a list of bytes. Defines the child trie to use +and must be of non-zero length. +- Return value is a packed `(ptr | length)` pointing to a memory area +that must be interpreted as SCALE-encoded `MultiRemovalResults`. + +`MultiRemovalResults` is declared in Rust as: + +```rust +struct MultiRemovalResults { + /// A continuation cursor which, if `Some` must be provided to the subsequent removal call. + /// If `None` then all removals are complete and no further calls are needed. + /// Must be the key in the state after the last deleted key. + pub maybe_cursor: Option>, + /// The number of items removed from the state. + /// + /// Keys are are already in the overlay do not count towards keys being removed from state. E.g. the overlay already has + /// key `AB`, the state also has `AB` and you are deleting with `prefix` `A`. `AB` would not be counted for `state`. + pub state: u32, + /// The number of unique keys removed, taking into account both the state and the overlay. + pub unique: u32, + /// The number of processed keys from the state. Should in maximum be `limit`. + pub loops: u32, +} +``` + +### Implementation of `ext_Storage_clear_prefix_version_3` + +Clear keys from the state and the overlay of the currently modified keys that +are starting with `prefix`.`maybe_limit` defines the maximum number of keys to +delete from the state. The keys that are deleted in the overlay aren't counted +towards `maybe_limit`. If `maybe_cursor` is of non-zero length, the +implementation needs to seek the given `cursor` and start the iteration with +`cursor` (or the first key in lexicographical order after `cursor` if not present). +Returns `MultiRemovalResults` to inform the caller about the operation. + +#### Examples + +Let's assume our state looks like this (only showing keys, as values are not important): + +| Key | +| ------------- | +| A | +| B | +| C | +| CA | +| CB | +| CC | +| CD | +| D | +| E | + +The overlay of currently modified keys looks like this (again only keys): + +| Key | +| ------------- | +| A | +| CB | +| CD | +| CE | +| F | + +In the following we use `clear_prefix` as Rust wrapper around +`ext_Storage_clear_prefix_version_3`. For simplicity we represent the +`prefix` and `maybe_cursor` as string like literal. + +1. Call `clear_prefix("A", u32::max_value(), "")`: + +Expected result: +`MultiRemovalResults { maybe_cursors: None, state: 0, unique: 1, loops: 1 }` + +Keys deleted: `[ A ]` + + +2. Call `clear_prefix("C", u32::max_value(), "")`: + +Expected result: +`MultiRemovalResults { maybe_cursors: None, state: 2, unique: 6, loops: 5 }` + +Keys deleted: `[ C, CA, CB, CC, CD, CE ]` + + +3. Call `clear_prefix("C", 2, "")`: + +Expected result: +`MultiRemovalResults { maybe_cursors: Some("CB"), state: 2, unique: 5, loops: 2 }` + +Keys deleted: `[ C, CA, CB, CD, CE ]` + + +4. Call `clear_prefix("C", 2, "CB")`: + +Expected result: +`MultiRemovalResults { maybe_cursors: Some("CD"), state: 1, unique: 4, loops: 2 }` + +Keys deleted: `[ CB, CC, CD, CE ]` + + +5. Call `clear_prefix("F", u32::max_value(), "")`: + +Expected result: +`MultiRemovalResults { maybe_cursors: None, state: 0, unique: 1, loops: 0 }` + +Keys deleted: `[ F ]` + +### Implementation of `ext_DefaultChildStorage_clear_prefix_version_3` + +Follows the exact same implementation as `ext_Storage_clear_prefix_version_3`, +but does the operation on the child trie defined by `storage_key`. + +### Implementation of `ext_DefaultChildStorage_storage_kill_version_4` + +Follows the exact same implementation as `ext_Storage_clear_prefix_version_3`, +but does the operation on the child trie defined by `storage_key`. As purpose of +this function is to delete the entire child trie, it doesn't take any `prefix`. + +## Security considerations + +The runtime is seen as a trusted code blob that don't require any special security considerations. + +## Alternatives + +Change `clear_prefix` and `kill_storage` to take into account the deleted keys +in the overlay to skip them while iterating the `state`. This approach let's +the node guess on the assumed operation of the runtime which isn't a great approach. + From c20471fa10552f8022af8443e2c10d64ffb3fb07 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bastian=20K=C3=B6cher?= Date: Wed, 1 Mar 2023 12:46:35 +0100 Subject: [PATCH 2/3] Use the correct snake case namings --- ...vironment-clear-prefix-and-storage-kill.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/proposals/0000-runtime-environment-clear-prefix-and-storage-kill.md b/proposals/0000-runtime-environment-clear-prefix-and-storage-kill.md index 27179bd..5374a39 100644 --- a/proposals/0000-runtime-environment-clear-prefix-and-storage-kill.md +++ b/proposals/0000-runtime-environment-clear-prefix-and-storage-kill.md @@ -13,8 +13,8 @@ Replaces: ## Summary Give the runtime more fine grained control over the internals of `clear_prefix` and `storage_kill`. -We are proposing to introduce `ext_Storage_clear_prefix_version_3`, `ext_DefaultChildStorage_clear_prefix_version_3` -and `ext_DefaultChildStorage_storage_kill_version_3`. Notable changes are the introduction of a new +We are proposing to introduce `ext_storage_clear_prefix_version_3`, `ext_default_child_storage_clear_prefix_version_3` +and `ext_default_child_storage_storage_kill_version_3`. Notable changes are the introduction of a new third parameter `maybe_cursor` and the change of the return value to `MultiRemovalResults`. The `maybe_cursor` enables a runtime developer to start removing storage items from a defined position. The new return value will give more details on how many items were @@ -34,9 +34,9 @@ any information on how many keys got deleted in the state and how got deleted in Introduce the following new host functions: ```rust -fn ext_Storage_clear_prefix_version_3(prefix: u64, maybe_limit: u32, maybe_cursor: u64) -> u64; -fn ext_DefaultChildStorage_clear_prefix_version_3(storage_key: u64, prefix: u64, maybe_limit: u32, maybe_cursor: u64) -> u64; -fn ext_DefaultChildStorage_storage_kill_version_4(storage_key: u64, maybe_limit: u32, maybe_cursor: u64) -> u64; +fn ext_storage_clear_prefix_version_3(prefix: u64, maybe_limit: u32, maybe_cursor: u64) -> u64; +fn ext_default_child_storage_clear_prefix_version_3(storage_key: u64, prefix: u64, maybe_limit: u32, maybe_cursor: u64) -> u64; +fn ext_default_child_storage_storage_kill_version_4(storage_key: u64, maybe_limit: u32, maybe_cursor: u64) -> u64; ``` The following list explains the parameters and the return value of all functions above: @@ -76,7 +76,7 @@ struct MultiRemovalResults { } ``` -### Implementation of `ext_Storage_clear_prefix_version_3` +### Implementation of `ext_storage_clear_prefix_version_3` Clear keys from the state and the overlay of the currently modified keys that are starting with `prefix`.`maybe_limit` defines the maximum number of keys to @@ -113,7 +113,7 @@ The overlay of currently modified keys looks like this (again only keys): | F | In the following we use `clear_prefix` as Rust wrapper around -`ext_Storage_clear_prefix_version_3`. For simplicity we represent the +`ext_storage_clear_prefix_version_3`. For simplicity we represent the `prefix` and `maybe_cursor` as string like literal. 1. Call `clear_prefix("A", u32::max_value(), "")`: @@ -155,14 +155,14 @@ Expected result: Keys deleted: `[ F ]` -### Implementation of `ext_DefaultChildStorage_clear_prefix_version_3` +### Implementation of `ext_default_child_storage_clear_prefix_version_3` -Follows the exact same implementation as `ext_Storage_clear_prefix_version_3`, +Follows the exact same implementation as `ext_storage_clear_prefix_version_3`, but does the operation on the child trie defined by `storage_key`. -### Implementation of `ext_DefaultChildStorage_storage_kill_version_4` +### Implementation of `ext_default_child_storage_storage_kill_version_4` -Follows the exact same implementation as `ext_Storage_clear_prefix_version_3`, +Follows the exact same implementation as `ext_storage_clear_prefix_version_3`, but does the operation on the child trie defined by `storage_key`. As purpose of this function is to delete the entire child trie, it doesn't take any `prefix`. From 6491aacda256c56a7aed3ee96ab8882699936cbd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bastian=20K=C3=B6cher?= Date: Wed, 1 Mar 2023 13:36:06 +0100 Subject: [PATCH 3/3] Give some example --- ...nvironment-clear-prefix-and-storage-kill.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/proposals/0000-runtime-environment-clear-prefix-and-storage-kill.md b/proposals/0000-runtime-environment-clear-prefix-and-storage-kill.md index 5374a39..922f014 100644 --- a/proposals/0000-runtime-environment-clear-prefix-and-storage-kill.md +++ b/proposals/0000-runtime-environment-clear-prefix-and-storage-kill.md @@ -29,6 +29,22 @@ with some `limit` multiple times and expecting `limit * number of calls` keys to While the actual result is that all calls remove exactly the same keys. They also don't get any information on how many keys got deleted in the state and how got deleted in the overlay. +Let's say you have some lazy clean up functionality that can be triggered by +sending a transaction and then reward the sender. With the current implementation of +`clear_prefix` you are only informed about the number of elements deleted in the state +and if all elements with the given `prefix` are deleted. This means that if you +include two transactions from different senders in one block, the second sender will +be rewarded while actually not having helped to delete any keys. For sure you could +prevent this from happening ensuring that only one transaction is included per block. +However, it would be a much better experience if the runtime developer is able to +control `clear_prefix` more fine grained by providing the `cursor`. This would allow +the runtime developer to include multiple of these transactions in one block, because +each of them would be able to delete unique data in the state. The new `clear_prefix` +would also give information on the total number of deleted keys, which includes +deletions from the state and the overlay, and total number of unique deletions +in the state. It also provides the next `cursor` as return value or `None` if +all keys with the `prefix` are deleted. + ## Detailed Solution design Introduce the following new host functions: @@ -66,7 +82,7 @@ struct MultiRemovalResults { pub maybe_cursor: Option>, /// The number of items removed from the state. /// - /// Keys are are already in the overlay do not count towards keys being removed from state. E.g. the overlay already has + /// Keys that are already in the overlay do not count towards keys being removed from state. E.g. the overlay already has /// key `AB`, the state also has `AB` and you are deleting with `prefix` `A`. `AB` would not be counted for `state`. pub state: u32, /// The number of unique keys removed, taking into account both the state and the overlay.