Implement generational storage #87

joshuajbouw · 2021-05-14T11:59:13Z

This implements the backwards compatible generational storage, which passes the tests.

How it works is that we have generational addresses, which start at 0, which increment by 1 every time a self destruct action, or more specifically, every time we Apply::Delete.

This feature is backwards compatible with the current storage structure and shouldn't cause conflicting issues as our keys are either a length of 54 or 58. Prior, our account storage kept a length of 54; however, to give us access to new storage, we must provide a new key. That is where the key length of 58 comes into play as the extra 4 bytes will allow us to tack on the generation value u32.

NOTE that the tests are -expectedly- failing. They were addressed in #85.

* Base precompile code between connectors * Handle errors and validate input * Use proper result

mfornet · 2021-05-14T12:06:50Z

src/engine.rs

@@ -222,16 +222,16 @@ impl Engine {
            .unwrap_or_else(U256::zero)
    }

-    pub fn remove_storage(address: &Address, key: &H256) {
-        sdk::remove_storage(&storage_to_key(address, key));
+    pub fn remove_storage(address: &Address, key: &H256, generation: u32) {


What do you think about creating a separate struct:

AddressWithGeneration { address: Address, generation: u64, }

You can use a different (and better) name for it.

This way interface for all this method will be a little bit cleaner.

Typically I do stuff like that if we get over 5 arguments. Here, I didn't see much of a point but, I get that grouping it together would be fine.

mfornet · 2021-05-14T12:11:54Z

src/engine.rs

@@ -503,7 +522,8 @@ impl evm::backend::Backend for Engine {

    /// Get storage value of address at index.
    fn storage(&self, address: Address, index: H256) -> H256 {


Not from this PR, but still

It is weird to use index for this variable name here and key elsewhere (see get_storage/set_storage), when they are referring to the same thing (have same semantics). I'd keep key which I believe is what ethereum is calling this value.

Yes, SSTORE and SLOAD use the nomenclature key.

src/storage.rs

mfornet · 2021-05-14T12:24:23Z

src/engine.rs

+        let generation = Self::get_generation(&address);
+        Engine::get_storage(&address, &index, generation)


This doesn't feel as efficient as possible. We are making two read access per read in the evm.

What about we have some cache created on the fly on each call. Since the most common scenarios is contracts reading from the same address this should be a noticeable improve. Generation is only read once per call per address.

We can discuss how to do this.

Yeah, it was kind of frustrating to really find a good way to do this and I was quite hesitant. I wanted to look into if we can do some early returns on some other parts. But, I'm not sure if that would be possible. The idea is to only read it once the entire time though.

I agree that caching would likely be a good idea. It's a little unfortunate that the type signature of fn storage is &self instead of &mut self.

Maybe it's worth merging a version without caching and make an issue to do some optimization work in the future. This way we can get the core fix in faster and also spend more dedicated time on optimization. I think optimization should usually be a dedicated task so that measurements can be made as part of the process. E.g. we will know how much gas is saved by adding a caching layer, or whatever other clever solution we might come up with given more time to design something.

Agree with Michael, let's delay this optimisation for future PR

In principle, we could defer optimizations. However, given that we are already at (or over) any reasonable gas use with the current contract, the impact of everything like this ought to at least be quantified. Adding a few gas-guzzling tweaks here and there may impact our bottom line soon enough.

I made some changes to this which stores the value in state upon creation. That way if there are multiple storage reads, then it'll only do it once. I don't know if the storage read is going to be always required at least once.

src/engine.rs

* Validate register length in `read_input_arr20()` * Only read register length in `Engine::get_code_size` * Add `read_input_borsh()` * Ensure `method.args.len() == args_decoded.len()` * Ensure register size is 8 in `read_u64` * Use constant to specify the register ID used in `read_input()`

birchmd

LGTM
I filed #105 to follow up on the performance question since it will not be addressed in this PR

birchmd · 2021-05-21T14:52:37Z

src/lib.rs

@@ -1,3 +1,4 @@
+#![feature(array_methods)]


What is this for?

That could have been a Jetbrains Rust automatic addition.

birchmd · 2021-05-21T14:53:33Z

src/engine.rs

                    }
                }
-                Apply::Delete { address } => Engine::remove_account(&address),
+                Apply::Delete { address } => {
+                    let generation = Self::get_generation(&address);


Nit: if both match arms need generation then we could call get_generation before the match

We need the address from somewhere though that needs to be modified is the issue with that.

mfornet · 2021-05-28T16:16:11Z

src/engine.rs

@@ -542,17 +563,20 @@ impl ApplyBackend for Engine {



This won't work since remove_all_storage is empty right now.

Not shown by github since this code was not touched in the PR (copy-pasting here):

if reset_storage { Engine::remove_all_storage(&address) }

Out of curiosity, when is the flat reset_storage set to true?

Its a possible flag that can be passed back by the EVM. Technically, we don't wipe it, but I just have it in there for prosperity sake. Maybe there will eventually be a use for it. Lots of notes and discussion on that already. See engine.rs L#287

src/engine.rs

joshuajbouw · 2021-06-01T15:40:32Z

This should be good to go once tests pass.

mfornet · 2021-06-01T22:06:12Z

Let's merge #84 against this PR and make sure the test for self-destruct is passing after this test.

artob · 2021-06-02T08:11:22Z

@joshuajbouw Tests still failing

joshuajbouw · 2021-06-02T08:47:04Z

@joshuajbouw Tests still failing

As far as I was aware this was expected as it was fixed elsewhere? I'll double check if that was merged to the target branch.

birchmd · 2021-06-02T13:40:33Z

@artob @mfornet this PR is targeting the branch with the new test (test_self_destruct) which #84 is based on. I suggest we should merge this PR into #84 (instead of the other way around) since it is easier and then we can debug any test failures there.

mfornet

LGTM, Approving to merge into #84

joshuajbouw · 2021-06-04T04:16:25Z

Can someone merge this please?

mfornet · 2021-06-04T12:05:35Z

Merging into #84 and will fix tests there

* Add tests for state check after selfdestruct * Aurora runner tracks storage usage to avoid underflow when storage is released in future transactions (#85) * Implement generational storage (#87) * Base precompile code between connectors (#73) * Base precompile code between connectors * Handle errors and validate input * Use proper result * Document `aurora encode-address` usage. * Cache cargo artifacts between CI runs. (#92) * Address comments from audit. (#86) * Validate register length in `read_input_arr20()` * Only read register length in `Engine::get_code_size` * Add `read_input_borsh()` * Ensure `method.args.len() == args_decoded.len()` * Ensure register size is 8 in `read_u64` * Use constant to specify the register ID used in `read_input()` * Reduce size of `cargo cache` in CI. (#95) * Define a `Wei` newtype for balances. (#96) * Fix evm-bully builds after recent refactoring. (#100) * Refactor `Engine::get_state` to return a `Result`. (#99) * Ensure that `Cargo.lock` in the repo is valid. (#101) * Remove unneeded nightly feature. (#102) * Implement BC generational storage. * fix address input * remove note * put key on the end of the storage key * remove pub from methods * Dispatch precompiles on the full address. (#107) * Support state migration on upgrade. (#103) * Implement the ETH connector. (#59) * Move when to call `set_generation` * Fix arg Co-authored-by: Marcelo Fornet <mfornet94@gmail.com> Co-authored-by: Arto Bendiken <arto@aurora.dev> Co-authored-by: Michael Birch <michael@near.org> Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com> Co-authored-by: Evgeny Ukhanov <mrlsd@ya.ru> * Fix layout of the key * Fix all tests (don't wipe the storage all the time) * Use correct generation in writing storage * Remove unnecessary references Co-authored-by: Michael Birch <michael@near.org> Co-authored-by: Joshua J. Bouw <dev@joshuajbouw.com> Co-authored-by: Arto Bendiken <arto@aurora.dev> Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com> Co-authored-by: Evgeny Ukhanov <mrlsd@ya.ru> Co-authored-by: Michael Birch <michael.birch@aurora.dev>

Base precompile code between connectors (#73)

37c8334

* Base precompile code between connectors * Handle errors and validate input * Use proper result

joshuajbouw added the C-enhancement Category: New feature or request label May 14, 2021

joshuajbouw requested a review from artob as a code owner May 14, 2021 11:59

joshuajbouw requested a review from mfornet May 14, 2021 11:59

mfornet suggested changes May 14, 2021

View reviewed changes

artob added 2 commits May 14, 2021 17:11

Document aurora encode-address usage.

02dbfd8

Merge branch 'master' into develop

53f4cc6

artob changed the title ~~Implement BC generational storage.~~ Implement generational storage May 15, 2021

artob added this to In progress in EVM on MainNet via automation May 15, 2021

birchmd and others added 2 commits May 17, 2021 23:04

Cache cargo artifacts between CI runs. (#92)

996f9fc

Merge branch 'master' into develop

2474b5d

joshuajbouw force-pushed the feat/gen_storage branch 2 times, most recently from eb569aa to 9f11b69 Compare May 18, 2021 06:25

EVM on MainNet automation moved this from In progress to Review in progress May 18, 2021

birchmd requested changes May 18, 2021

View reviewed changes

src/engine.rs Outdated Show resolved Hide resolved

birchmd and others added 12 commits May 19, 2021 01:33

Reduce size of cargo cache in CI. (#95)

ca41305

Define a Wei newtype for balances. (#96)

bfaa2c2

Fix evm-bully builds after recent refactoring. (#100)

7d4990f

Refactor Engine::get_state to return a Result. (#99)

7b352b3

Ensure that Cargo.lock in the repo is valid. (#101)

2689e3b

Remove unneeded nightly feature. (#102)

060f310

Implement BC generational storage.

7cdd7c3

fix address input

1682bbd

remove note

3a5bac6

put key on the end of the storage key

fb3ca77

remove pub from methods

77ba897

joshuajbouw force-pushed the feat/gen_storage branch 2 times, most recently from 9437b33 to 77ba897 Compare May 21, 2021 14:47

birchmd mentioned this pull request May 21, 2021

Profile and optimize generational storage #105

Open

3 tasks

birchmd approved these changes May 21, 2021

View reviewed changes

joshuajbouw requested a review from mfornet May 27, 2021 04:49

birchmd added 2 commits May 28, 2021 17:26

Dispatch precompiles on the full address. (#107)

046a560

Support state migration on upgrade. (#103)

b09b530

mfornet suggested changes May 28, 2021

View reviewed changes

Implement the ETH connector. (#59)

5144281

artob assigned joshuajbouw May 31, 2021

joshuajbouw added 2 commits June 1, 2021 20:44

Merge branch 'master' into feat/gen_storage

6b02195

Move when to call set_generation

bff62d4

joshuajbouw requested a review from mfornet June 1, 2021 13:48

joshuajbouw assigned artob and mfornet and unassigned joshuajbouw Jun 1, 2021

Fix arg

2ae28fb

EVM on MainNet automation moved this from Review in progress to Reviewer approved Jun 3, 2021

mfornet approved these changes Jun 3, 2021

View reviewed changes

mfornet merged commit d67b59e into test_self_destruct Jun 4, 2021

EVM on MainNet automation moved this from Reviewer approved to Done Jun 4, 2021

mfornet deleted the feat/gen_storage branch June 4, 2021 12:05

artob mentioned this pull request Jun 4, 2021

Implement generational storage with SELFDESTRUCT tests #84

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement generational storage #87

Implement generational storage #87

joshuajbouw commented May 14, 2021 •

edited

mfornet May 14, 2021

joshuajbouw May 14, 2021

mfornet May 14, 2021

artob May 15, 2021

mfornet May 14, 2021

mfornet May 14, 2021

joshuajbouw May 14, 2021

birchmd May 14, 2021

mfornet May 14, 2021

artob May 15, 2021

joshuajbouw May 18, 2021 •

edited

birchmd left a comment

birchmd May 21, 2021

joshuajbouw May 21, 2021

birchmd May 21, 2021

joshuajbouw Jun 1, 2021

mfornet May 28, 2021 •

edited

joshuajbouw Jun 1, 2021

joshuajbouw commented Jun 1, 2021 •

edited

mfornet commented Jun 1, 2021

artob commented Jun 2, 2021

joshuajbouw commented Jun 2, 2021 •

edited

birchmd commented Jun 2, 2021

mfornet left a comment

joshuajbouw commented Jun 4, 2021

mfornet commented Jun 4, 2021

		@@ -503,7 +522,8 @@ impl evm::backend::Backend for Engine {

		/// Get storage value of address at index.
		fn storage(&self, address: Address, index: H256) -> H256 {

		let generation = Self::get_generation(&address);
		Engine::get_storage(&address, &index, generation)

Implement generational storage #87

Implement generational storage #87

Conversation

joshuajbouw commented May 14, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshuajbouw May 18, 2021 • edited

Choose a reason for hiding this comment

birchmd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfornet May 28, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshuajbouw commented Jun 1, 2021 • edited

mfornet commented Jun 1, 2021

artob commented Jun 2, 2021

joshuajbouw commented Jun 2, 2021 • edited

birchmd commented Jun 2, 2021

mfornet left a comment

Choose a reason for hiding this comment

joshuajbouw commented Jun 4, 2021

mfornet commented Jun 4, 2021

joshuajbouw commented May 14, 2021 •

edited

joshuajbouw May 18, 2021 •

edited

mfornet May 28, 2021 •

edited

joshuajbouw commented Jun 1, 2021 •

edited

joshuajbouw commented Jun 2, 2021 •

edited