Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EVM: bnadd/bnmul Precompile Performance Optimization #2568

Merged
merged 1 commit into from Mar 6, 2023

Conversation

holgerd77
Copy link
Member

Replaces #1570

(second commit optimizations from the above PR already done along some subsequent jumpdest analysis/optimization work)

…r the necessary 128 bytes as input for the WASM call
@acolytec3
Copy link
Contributor

Looks great! I tested this on my Intel Silicon based laptop and it definitely improves performance on these precompiles.

@acolytec3 acolytec3 merged commit e1a2cca into master Mar 6, 2023
@holgerd77 holgerd77 deleted the vm-performance-test-bnadd-fix branch March 6, 2023 19:59
acolytec3 added a commit that referenced this pull request Mar 20, 2023
* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
jochem-brouwer added a commit that referenced this pull request Mar 26, 2023
* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
acolytec3 added a commit that referenced this pull request Mar 27, 2023
* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
acolytec3 added a commit that referenced this pull request Mar 27, 2023
* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
acolytec3 added a commit that referenced this pull request Apr 3, 2023
* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
acolytec3 added a commit that referenced this pull request Apr 3, 2023
* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
acolytec3 added a commit that referenced this pull request Apr 3, 2023
* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
acolytec3 added a commit that referenced this pull request Apr 4, 2023
* `Buffer` to `Uint8Array` conversion (#2566)

* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>

* Devp2p status fix

* Remove buffer detritus

* Switch db to view

* buffer cleanup

* Correctly parse heads from DB

* Fix encodings

* Cast db values to uint8array

* Fix db bug in ethash

* Remove unused peerId check

* Add pow miner test

* Finish test

* Move pow test to integration tests

* client: lint

* rename test helper

* Add timeout to PoW test

* Fix test runner

* remove console log

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
acolytec3 added a commit that referenced this pull request Apr 11, 2023
* `Buffer` to `Uint8Array` conversion (#2566)

* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>

* Devp2p status fix

* Remove buffer detritus

* Switch db to view

* buffer cleanup

* Correctly parse heads from DB

* Fix encodings

* Cast db values to uint8array

* Fix db bug in ethash

* Remove unused peerId check

* Add pow miner test

* Finish test

* Move pow test to integration tests

* client: lint

* rename test helper

* Add timeout to PoW test

* Update eip4844 txs to decoupled blobs spec

* fix sharding spec and blobtx

* fix the sharding muli client run

* fix args

* fix compatibility with latest lodestar branch

* update test help

* Fix fee market test

* lint

* Move all kzg stuff to util

* Various cleanup

* Update to latest c-kzg

* Fix utils

* Remove outdated kzg references

* Fix client tests

* Update c-kzg dep

* Fix karma, remove Buffer references

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>
Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
g11tech added a commit that referenced this pull request Apr 13, 2023
* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
g11tech added a commit that referenced this pull request Apr 13, 2023
* `Buffer` to `Uint8Array` conversion (#2566)

* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>

* Devp2p status fix

* Remove buffer detritus

* Switch db to view

* buffer cleanup

* Correctly parse heads from DB

* Fix encodings

* Cast db values to uint8array

* Fix db bug in ethash

* Remove unused peerId check

* Add pow miner test

* Finish test

* Move pow test to integration tests

* client: lint

* rename test helper

* Add timeout to PoW test

* Fix test runner

* remove console log

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
g11tech added a commit that referenced this pull request Apr 13, 2023
* `Buffer` to `Uint8Array` conversion (#2566)

* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>

* Devp2p status fix

* Remove buffer detritus

* Switch db to view

* buffer cleanup

* Correctly parse heads from DB

* Fix encodings

* Cast db values to uint8array

* Fix db bug in ethash

* Remove unused peerId check

* Add pow miner test

* Finish test

* Move pow test to integration tests

* client: lint

* rename test helper

* Add timeout to PoW test

* Update eip4844 txs to decoupled blobs spec

* fix sharding spec and blobtx

* fix the sharding muli client run

* fix args

* fix compatibility with latest lodestar branch

* update test help

* Fix fee market test

* lint

* Move all kzg stuff to util

* Various cleanup

* Update to latest c-kzg

* Fix utils

* Remove outdated kzg references

* Fix client tests

* Update c-kzg dep

* Fix karma, remove Buffer references

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>
Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
@holgerd77 holgerd77 changed the title VM: bnadd/bnmul Precompile Performance Optimization EVM: bnadd/bnmul Precompile Performance Optimization Apr 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants