Implement the block execution pipeline #17

jsign · 2024-01-11T11:33:17Z

This PR rebuilds our block execution pipeline to something rather complete. The implementation is based on the execution-spec and the EVMC docs to integrate with EVMOne.

Laterally, this also fixed existing bugs, added more signature support, RLP decoding for types, and many things needed to make this work.

Instead of writing a long PR description, I'll do some heavy PR commenting to help review and explain these other things.

…ions Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

…ich requied fixes Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

…ing snapshoting more seameless Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign · 2024-01-11T11:34:37Z

build.zig

-        .root_source_file = .{ .path = "src/main.zig" },
+        .root_source_file = .{ .path = "src/lib.zig" },


It's usually recommended to put library tests in their own file, to separate the unit_test executable from the "real" executable.

that's good, but maybe give it a more specific name than lib ?

Yeah, maybe. I'll think about it.

jsign · 2024-01-11T11:38:04Z

src/blockchain/blockchain.zig

The rockstars in this PR are two new files:

src/blockchain/blockchain.zig

src/blockchain/vm.zig

The former (this file) contains most of the execution specs code, which includes:

Block validation

Tx validation

Gas fee calcs (base fee, refunds, intrinsic costs, etc)

Access lists initialization

Tx -> Message creation (i.e: the first message to exec)

The latter (i.e: .../vm.zig) has the message execution pipeline, which mostly translates to EVMOne implementation.

jsign · 2024-01-11T11:38:48Z

src/blockchain/blockchain.zig

+const VM = vm.VM;
+const Keccak256 = std.crypto.hash.sha3.Keccak256;
+
+pub const Blockchain = struct {


This is the top level entity that manages the execution pipeline.

jsign · 2024-01-11T11:39:27Z

src/blockchain/blockchain.zig

+    state: *StateDB,
+    prev_block: BlockHeader,
+    last_256_blocks_hashes: [256]Hash32, // ordered in asc order


These are the three main fields that it tracks:

The current state.

The previous block, which will be used when asked to run the next block to do some validations.

The last 256 block hashes since this state is needed for BLOCKHASH.

The txn signer for this chain id (to be used in getting sender from signature).

jsign · 2024-01-11T11:40:32Z

src/blockchain/blockchain.zig

+        };
+    }
+
+    pub fn runBlock(self: *Blockchain, block: Block) !void {


On every new block, the client calls this fn to move forward.

jsign · 2024-01-11T13:11:06Z

src/types/transaction.zig

@@ -24,24 +24,38 @@ pub const Txn = union(TxnTypes) {
    }

    // decode decodes a transaction from bytes. The provided bytes are referenced in the returned transaction.
-    pub fn decode(bytes: []const u8) !Txn {
+    pub fn decode(arena: Allocator, bytes: []const u8) !Txn {


Now we receive an allocator, since now we fixed zig-rlp to allow decoding the data field which is a slice that requires an allocation.

Note something important. The rlp.deserialize(...) receives an allocator and might potentially do a lot of allocations (in theory, depending how nested the struct is and how many slices might have).

But... there's no deinit(). So actually the RLP library is using the allocator in a way that is not easy to do a "deinit" unless the caller knows all the fields that might have been allocated, etc.

This is clearly a case for an arena allocator, which how I use the library. (SImilar to how std.json works). The client should create an arena, send that as the allocator, and for "deinit()" simply destroy the arena.

jsign · 2024-01-11T13:16:35Z

src/types/transaction.zig


        return error.UnsupportedTxnType;
    }

+    pub fn decodeFromRLP(self: *Txn, arena: Allocator, serialized: []const u8) !usize {


This is a custom decoder I had to do, since the Txn type is an union and obviously that type isn't (and can't) be supported in zig-rlp.

Moreover, I need a logic to see if the encoded tx is a LegacyTx or is envelop-based one.

This is done by checking if the first byte:

If it's a RLP-struct, then the serialized is a LegacyTx directly.

If it isn't, we have to first decode it as a string and then we can delegate to Txn.decode(..) so it can decode it appropriately depending on the envelope tx type.

Probably the niciest example of a custom RLP decoder for zig-rlp.

actually we could probably implement rlp unions, assuming that they work exactly like transactions: you start with an index in the union and then the payload of the subtype. But let's see if that shows up anywhere else than here. I'll make a pr when this one is merged, so that the logic is abstracted to comptime stuff.

jsign · 2024-01-11T13:19:57Z

src/types/transaction.zig

+            var ltx: LegacyTxn = undefined;
+            const size = rlp.deserialize(LegacyTxn, arena, serialized, &ltx);
+            self.* = .{ .LegacyTxn = ltx };


Note that (unfortunately) I cant do self.* = try Txn.decode(arena, str); here (as done in L54), since I need the usize of rlp.deserialize to return in the custom deserializer.

In the case of envelope based tx (L52-L54), I can use Txn.decode since I had to first deserialize serialized as a string with rlp.deserialized which gave me the size I needed. Pretty funny!

jsign · 2024-01-11T13:20:29Z

src/types/transaction.zig

@@ -1,8 +1,8 @@
 const std = @import("std");


Note: I was tempted to do the Txn->Tx renaming in this PR but I prefer to do that in a separate one to avoid even more changed lines. I'll do that soon after.

jsign · 2024-01-11T18:36:29Z

src/blockchain/blockchain.zig

+        const prev_block_hash = try common.decodeRLPAndHash(BlockHeader, allocator, prev_block, null);
+        if (!std.mem.eql(u8, &curr_block.parent_hash, &prev_block_hash))
+            return error.InvalidParentHash;
+    }


Doing this decodeRLPAndHash(...) is that I found this (fixed now) bug.

The usual thing of "Ah, doing this check is easy" and you get in a rabbit hole. :P

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign · 2024-01-31T11:58:49Z

src/blockchain/vm.zig

+        return evmc.EVMC_ACCESS_COLD;
+    }
+
+    fn call(ctx: ?*evmc.struct_evmc_host_context, msg: [*c]const evmc.struct_evmc_message) callconv(.C) evmc.struct_evmc_result {


Days after comment: this call is missing some things for contract creation transactions. I'm working on that now and it will be in another PR. Just mentioning.

gballet

Wow, that was intense! I left a lot of questions and comments, but they can all be ignored for now, except the one about the max code size. But feel free to merge whenever you feel it's ready.

gballet · 2024-01-15T17:23:11Z

build.zig

-        .root_source_file = .{ .path = "src/main.zig" },
+        .root_source_file = .{ .path = "src/lib.zig" },


that's good, but maybe give it a more specific name than lib ?

gballet · 2024-01-16T09:52:58Z

src/blockchain/blockchain.zig

+pub const Blockchain = struct {
+    allocator: Allocator,
+    chain_id: config.ChainId,
+    state: *StateDB,


meh, that name strikes terror in the hearts of men. Let's name it something more contributor-friendly like just State. It's not meant to be a db, is it?

Do you refer to the field name or the type name?
StateDB is a database, yes (in memory one).
state refers to the state of the blockchain.

Are you suggesting State instead of StateDB?
I don't have a strong opinion, just checking that's your suggestion.

gballet · 2024-01-16T09:59:55Z

src/blockchain/blockchain.zig

+        var arena = std.heap.ArenaAllocator.init(self.allocator);
+        defer arena.deinit();
+        const allocator = arena.allocator();


couldn't we make the top-level allocator an area allocator, and just pass self.allocator() when we call runBlock ? and then let the caller deinit() it? This way, we can use other types of allocators for "unit" tests without having to worry about it, and also so that some data can be extracted by the client call after runBlock ? I fear that, otherwise, it might force us to allocate+copy data that we then need to pass to the called, and then have to make the runBlock function signature impossibly long.

gballet · 2024-02-06T09:49:04Z

src/blockchain/vm.zig

+    pub fn processMessageCall(self: *VM, msg: Message) !evmc.struct_evmc_result {
+        const evmc_message: evmc.struct_evmc_message = .{
+            .kind = if (msg.target != null) evmc.EVMC_CALL else evmc.EVMC_CREATE,
+            .flags = 0,
+            .depth = 0,
+            .gas = @intCast(msg.gas),
+            .recipient = toEVMCAddress(msg.current_target),
+            .sender = toEVMCAddress(msg.caller),
+            .input_data = msg.data.ptr,
+            .input_size = msg.data.len,
+            .value = blk: {
+                var txn_value: [32]u8 = undefined;
+                std.mem.writeIntSliceBig(u256, &txn_value, msg.value);
+                break :blk .{ .bytes = txn_value };
+            },
+            .create2_salt = undefined, // EVMC docs: field only mandatory for CREATE2 kind which doesn't apply at depth 0.
+            .code_address = toEVMCAddress(msg.code_address),
+        };


ok if that's what you need, I was wondering if we could instead just abstract the struct_evmc_message as a Message and not have to worry about the internals/conversion, but I guess it's not necessary right now since we don't have any alternative interpreter.

gballet · 2024-02-06T09:52:23Z

src/blockchain/vm.zig

+
+// EVMOneHost contains the implementation of the EVMC host interface.
+// https://evmc.ethereum.org/structevmc__host__interface.html
+const EVMOneHost = struct {


I would move this to its own emvc.zig file for cleanliness

gballet · 2024-02-06T15:12:35Z

src/blockchain/blockchain.zig

+    allocator: Allocator,
+    chain_id: config.ChainId,
+    state: *StateDB,
+    prev_block: BlockHeader,


how about making this a ?BlockHeader since we execute in stateless and might not have that previous block?

Huh? A stateless client must always have the previous block.
Even if it's the first one that it validates. If that isn't the case, how would verify the witness proof?

gballet · 2024-02-06T15:18:10Z

src/blockchain/blockchain.zig

+        const prev_block_hash = try common.decodeRLPAndHash(BlockHeader, allocator, prev_block, null);
+        if (!std.mem.eql(u8, &curr_block.parent_hash, &prev_block_hash))
+            return error.InvalidParentHash;
+    }


gballet · 2024-02-06T15:21:47Z

src/blockchain/blockchain.zig

+
+    fn applyBody(allocator: Allocator, chain: *Blockchain, state: *StateDB, block: Block) !BlockExecutionResult {
+        var gas_available = block.header.gas_limit;
+        for (block.transactions) |tx| {


actually it should be called applyTransactions or something of the sort, because the body contains more information than just the txs, like uncles (not implemented, but still) and withdrawals. If you want to process witndrawals in here was well, then maybe applyStateTransition ?

gballet · 2024-02-06T15:31:53Z

src/blockchain/blockchain.zig

+        // TODO: self destruct processing
+        // for address in output.accounts_to_delete:
+        // destroy_account(env.state, address)


yeah I agree, but in spite of what I said in other comments about the tooling, we should start by supporting our basic use case. I said otherwise before I got to this massive file 😉

gballet · 2024-02-06T15:34:55Z

src/blockchain/blockchain.zig

+            return false;
+        if (tx.getNonce() >= (2 << 64) - 1)
+            return false;
+        if (tx.getTo() == null and tx.getData().len > 2 * params.max_code_size)


why 2* here?

Yeah, it was surprising to me too. Turns out that the inint code size seems to be double the max code size. This is done this way in the spec, and also double checked in geth.

jsign added 30 commits January 2, 2024 08:26

blockchain: add Blockchain abstraction and start making block validat…

862b1db

…ions Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: finish block header validations

3234577

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

types: nits

867d1fa

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

types/block: refactor

98b8c31

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

types/block: add non-header fields

14d6b6d

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: add top-level run block logic

54dbc14

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

types: create type for LogsBloom

b5e1367

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

signer: remove old comments

1e50008

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: add tx validations

91ac17c

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: apply tx body code

74377b1

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: process transaction progress

1b31d57

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: process transaction more progress

3b9bdd0

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: more impl progress

301a00c

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: almost integrating EVMOne to new blockchain abstraction

f4dd7b0

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: cleanup

f17cc74

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

util: move package to common

d76935c

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

vm & blockchain cleanups

c0d9b3e

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: refactor

c2c0484

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: fixes

6922d6d

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: implement EVMOne block_hash

3ec6d47

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: implement EVMOne account_exists

d56ebb3

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

general fixes

edbc371

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: implement EVMOne get_storage & fixes

bf61d3f

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: implement EVMOne set_storage

fa1da46

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: implement EVMOne get_balance

7458ac6

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: implement EVMOne get_code_size

267efaa

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: implement EVMOne get_code_hash

ec36452

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: implement EVMOne copy_code

0061ed2

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: implement access_account and access_storage & refactors

97bb412

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: more progress

a4e1cc1

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign added 21 commits January 8, 2024 20:57

blockchain/vm: more progress

9fa5295

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

general: compilation fixes

a802461

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

general fixes

3175eec

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

more compilation fixes and refactors

90d168f

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix last compilation errors

57b9512

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

mod: update zig-rlp and fix codebase

320a6a3

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

types/transaction: add customized RLP decoder for Txn

3783027

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

signer: implement pubkey recovery pre EIP-155

b1b0414

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain/vm: fixes

6dd3c95

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

general fixes

b9e1fc0

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: refactor

5b1ec07

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: include validations with parent block & update zig-rlp wh…

9f7c506

…ich requied fixes Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

mod: update to tentative zig-rlp and avoid workaround

558bcd0

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: manage prev_block reference and last 256 hashes

8d59b33

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

common: add rlp helpers

87d2054

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: refactor

1fccb1b

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: make accessed addresses and storage keys into statedb mak…

c2e3e47

…ing snapshoting more seameless Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: cleanups

bc5f106

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

reorganize packages and unit tests

c27d93a

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

execspectests: compare post-state entry by entry

6534007

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

blockchain: create unique txn signer

57400eb

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign commented Jan 11, 2024

View reviewed changes

jsign requested a review from gballet January 11, 2024 18:37

jsign changed the title ~~[draft] Implement the block execution pipeline~~ Implement the block execution pipeline Jan 11, 2024

mod: use zig-rlp v0.1.0

555ea0f

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign mentioned this pull request Jan 23, 2024

Implement EIP-158 and EIP-2200 which allow to pass 7 extra spec tests #18

Merged

jsign commented Jan 31, 2024

View reviewed changes

gballet reviewed Feb 6, 2024

View reviewed changes

jsign merged commit e6be1bc into main Feb 6, 2024
4 checks passed

jsign deleted the jsign-vm2 branch February 6, 2024 23:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement the block execution pipeline #17

Implement the block execution pipeline #17

jsign commented Jan 11, 2024 •

edited

Loading

jsign Jan 11, 2024

gballet Jan 15, 2024

jsign Feb 6, 2024

jsign Jan 11, 2024

jsign Jan 11, 2024

jsign Jan 11, 2024

jsign Jan 11, 2024

jsign Jan 11, 2024

jsign Jan 11, 2024

gballet Feb 6, 2024

jsign Jan 11, 2024

jsign Jan 11, 2024 •

edited

Loading

jsign Jan 11, 2024

gballet Feb 6, 2024

jsign Jan 31, 2024

gballet left a comment

gballet Jan 15, 2024

gballet Jan 16, 2024

jsign Feb 6, 2024

gballet Jan 16, 2024

gballet Feb 6, 2024

gballet Feb 6, 2024

gballet Feb 6, 2024

jsign Feb 6, 2024

gballet Feb 6, 2024

gballet Feb 6, 2024

gballet Feb 6, 2024

gballet Feb 6, 2024

jsign Feb 6, 2024

		.root_source_file = .{ .path = "src/main.zig" },
		.root_source_file = .{ .path = "src/lib.zig" },

Implement the block execution pipeline #17

Implement the block execution pipeline #17

Conversation

jsign commented Jan 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsign Jan 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gballet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsign commented Jan 11, 2024 •

edited

Loading

jsign Jan 11, 2024 •

edited

Loading