Buffer to Uint8Array conversion (#2566)

* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>

packages/block/src/block.ts

@@ -5,10 +5,9 @@ import { BlobEIP4844Transaction, Capability, TransactionFactory } from '@ethereu
 import {
   KECCAK256_RLP,
   Withdrawal,
-  arrToBufArr,
   bigIntToHex,
-  bufArrToArr,
-  bufferToHex,
+  bytesToHex,
+  equalsBytes,
   intToHex,
   isHexPrefixed,
   ssz,
@@ -20,15 +19,15 @@ import { blockFromRpc } from './from-rpc'
 import { BlockHeader } from './header'
 import { getDataGasPrice } from './helpers'
 
-import type { BlockBuffer, BlockData, BlockOptions, JsonBlock, JsonRpcBlock } from './types'
+import type { BlockBytes, BlockData, BlockOptions, JsonBlock, JsonRpcBlock } from './types'
 import type { Common } from '@ethereumjs/common'
 import type {
   FeeMarketEIP1559Transaction,
   Transaction,
   TxOptions,
   TypedTransaction,
 } from '@ethereumjs/tx'
-import type { WithdrawalBuffer } from '@ethereumjs/util'
+import type { WithdrawalBytes } from '@ethereumjs/util'
 
 /**
  * An object that represents the block.
@@ -49,7 +48,7 @@ export class Block {
   public static async genWithdrawalsTrieRoot(wts: Withdrawal[], emptyTrie?: Trie) {
     const trie = emptyTrie ?? new Trie()
     for (const [i, wt] of wts.entries()) {
-      await trie.put(Buffer.from(RLP.encode(i)), arrToBufArr(RLP.encode(wt.raw())))
+      await trie.put(RLP.encode(i), RLP.encode(wt.raw()))
     }
     return trie.root()
   }
@@ -70,7 +69,7 @@ export class Block {
   public static async genTransactionsTrieRoot(txs: TypedTransaction[], emptyTrie?: Trie) {
     const trie = emptyTrie ?? new Trie()
     for (const [i, tx] of txs.entries()) {
-      await trie.put(Buffer.from(RLP.encode(i)), tx.serialize())
+      await trie.put(RLP.encode(i), tx.serialize())
     }
     return trie.root()
   }
@@ -133,8 +132,8 @@ export class Block {
    * @param serialized
    * @param opts
    */
-  public static fromRLPSerializedBlock(serialized: Buffer, opts?: BlockOptions) {
-    const values = arrToBufArr(RLP.decode(Uint8Array.from(serialized))) as BlockBuffer
+  public static fromRLPSerializedBlock(serialized: Uint8Array, opts?: BlockOptions) {
+    const values = RLP.decode(Uint8Array.from(serialized)) as BlockBytes
 
     if (!Array.isArray(values)) {
       throw new Error('Invalid serialized block input. Must be array')
@@ -144,12 +143,12 @@ export class Block {
   }
 
   /**
-   * Static constructor to create a block from an array of Buffer values
+   * Static constructor to create a block from an array of Bytes values
    *
    * @param values
    * @param opts
    */
-  public static fromValuesArray(values: BlockBuffer, opts?: BlockOptions) {
+  public static fromValuesArray(values: BlockBytes, opts?: BlockOptions) {
     if (values.length > 4) {
       throw new Error('invalid block. More values than expected were received')
     }
@@ -163,7 +162,7 @@ export class Block {
       )
     }
 
-    const [headerData, txsData, uhsData, withdrawalsBuffer] = values
+    const [headerData, txsData, uhsData, withdrawalBytes] = values
 
     const header = BlockHeader.fromValuesArray(headerData, opts)
 
@@ -199,7 +198,7 @@ export class Block {
       uncleHeaders.push(BlockHeader.fromValuesArray(uncleHeaderData, uncleOpts))
     }
 
-    const withdrawals = (withdrawalsBuffer as WithdrawalBuffer[])
+    const withdrawals = (withdrawalBytes as WithdrawalBytes[])
       ?.map(([index, validatorIndex, address, amount]) => ({
         index,
         validatorIndex,
@@ -312,27 +311,27 @@ export class Block {
   }
 
   /**
-   * Returns a Buffer Array of the raw Buffers of this block, in order.
+   * Returns a Array of the raw Bytes Arays of this block, in order.
    */
-  raw(): BlockBuffer {
-    const bufferArray = <BlockBuffer>[
+  raw(): BlockBytes {
+    const bytesArray = <BlockBytes>[
       this.header.raw(),
       this.transactions.map((tx) =>
         tx.supports(Capability.EIP2718TypedTransaction) ? tx.serialize() : tx.raw()
-      ) as Buffer[],
+      ) as Uint8Array[],
       this.uncleHeaders.map((uh) => uh.raw()),
     ]
     const withdrawalsRaw = this.withdrawals?.map((wt) => wt.raw())
     if (withdrawalsRaw) {
-      bufferArray.push(withdrawalsRaw)
+      bytesArray.push(withdrawalsRaw)
     }
-    return bufferArray
+    return bytesArray
   }
 
   /**
    * Returns the hash of the block.
    */
-  hash(): Buffer {
+  hash(): Uint8Array {
     return this.header.hash()
   }
 
@@ -346,8 +345,8 @@ export class Block {
   /**
    * Returns the rlp encoding of the block.
    */
-  serialize(): Buffer {
-    return Buffer.from(RLP.encode(bufArrToArr(this.raw())))
+  serialize(): Uint8Array {
+    return RLP.encode(this.raw())
   }
 
   /**
@@ -365,14 +364,14 @@ export class Block {
   async validateTransactionsTrie(): Promise<boolean> {
     let result
     if (this.transactions.length === 0) {
-      result = this.header.transactionsTrie.equals(KECCAK256_RLP)
+      result = equalsBytes(this.header.transactionsTrie, KECCAK256_RLP)
       return result
     }
 
-    if (this.txTrie.root().equals(KECCAK256_RLP)) {
+    if (equalsBytes(this.txTrie.root(), KECCAK256_RLP) === true) {
       await this.genTxTrie()
     }
-    result = this.txTrie.root().equals(this.header.transactionsTrie)
+    result = equalsBytes(this.txTrie.root(), this.header.transactionsTrie)
     return result
   }
 
@@ -487,8 +486,8 @@ export class Block {
    */
   validateUnclesHash(): boolean {
     const uncles = this.uncleHeaders.map((uh) => uh.raw())
-    const raw = RLP.encode(bufArrToArr(uncles))
-    return Buffer.from(keccak256(raw)).equals(this.header.uncleHash)
+    const raw = RLP.encode(uncles)
+    return equalsBytes(keccak256(raw), this.header.uncleHash)
   }
 
   /**
@@ -499,7 +498,7 @@ export class Block {
       throw new Error('EIP 4895 is not activated')
     }
     const withdrawalsRoot = await Block.genWithdrawalsTrieRoot(this.withdrawals!)
-    return withdrawalsRoot.equals(this.header.withdrawalsRoot!)
+    return equalsBytes(withdrawalsRoot, this.header.withdrawalsRoot!)
   }
 
   /**
@@ -523,7 +522,7 @@ export class Block {
     }
 
     // Header does not count an uncle twice.
-    const uncleHashes = this.uncleHeaders.map((header) => header.hash().toString('hex'))
+    const uncleHashes = this.uncleHeaders.map((header) => bytesToHex(header.hash()))
     if (!(new Set(uncleHashes).size === uncleHashes.length)) {
       const msg = this._errorMsg('duplicate uncles')
       throw new Error(msg)
@@ -572,7 +571,7 @@ export class Block {
   public errorStr() {
     let hash = ''
     try {
-      hash = bufferToHex(this.hash())
+      hash = bytesToHex(this.hash())
     } catch (e: any) {
       hash = 'error'
     }

packages/block/src/from-rpc.ts

@@ -1,5 +1,5 @@
 import { TransactionFactory } from '@ethereumjs/tx'
-import { TypeOutput, setLengthLeft, toBuffer, toType } from '@ethereumjs/util'
+import { TypeOutput, setLengthLeft, toBytes, toType } from '@ethereumjs/util'
 
 import { blockHeaderFromRpc } from './header-from-rpc'
 
@@ -21,7 +21,7 @@ function normalizeTxParams(_txParams: any) {
   // strict byte length checking
   txParams.to =
     txParams.to !== null && txParams.to !== undefined
-      ? setLengthLeft(toBuffer(txParams.to), 20)
+      ? setLengthLeft(toBytes(txParams.to), 20)
       : null
 
   txParams.v = toType(txParams.v, TypeOutput.BigInt)