Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EIP3860: only check max init code size on create contract tx #2575

Merged
merged 1 commit into from
Mar 9, 2023

Conversation

jochem-brouwer
Copy link
Member

See: #2569

We limit tx data size to MAX_INITCODE_SIZE (of EIP 3860) previously, but this should only be checked in a CREATE-transaction (to to field is empty, i.e. undefined or null). This PR fixes this.

I would like two people to verify / review this before merging.

Note: apparently we have either not tested this, or we have not joined testnets, or this is not covered by the EIP tests, which should be added/tested.

})

const maxInitCodeSize = common.param('vm', 'maxInitCodeSize')
const txTypes = [0, 1, 2, 5]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side note: why is 4844 tx type 5 and not 3?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No idea but that's what the EIP specifies.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the end you can't make an order for tx type numbers just out of EIPs, since there is no guarantee in which HF things are applied (if this is the goal to keep tx type numbers somewhat "chronologically").

So I would cautiously assume that there might just be other EIPs "in between" which claiming tx type numbers 3 and 4 (but totally just a guess).

@codecov
Copy link

codecov bot commented Mar 8, 2023

Codecov Report

Merging #2575 (edab5db) into master (1d65ea0) will decrease coverage by 0.32%.
The diff coverage is 100.00%.

Additional details and impacted files

Impacted file tree graph

Flag Coverage Δ
block 90.13% <ø> (ø)
blockchain 90.40% <ø> (ø)
client 87.68% <ø> (ø)
common 95.82% <ø> (ø)
devp2p ?
ethash ∅ <ø> (∅)
evm 79.27% <ø> (ø)
rlp ?
statemanager ?
trie 90.36% <ø> (-0.18%) ⬇️
tx 94.16% <100.00%> (+0.44%) ⬆️
util 84.50% <ø> (ø)
vm ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Copy link
Contributor

@acolytec3 acolytec3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I'm surprised the createContract && this.... didn't trigger the linter but I guess it's happy so good by me.

@jochem-brouwer
Copy link
Member Author

I initially committed a file txData.to == null (this is both true if txData.to is either undefined or null), however the linter did not like this. (explicit check if it is either null or undefined, reflected in current changes)

Why should the linter not approve the new statement, though?

@acolytec3
Copy link
Contributor

I initially committed a file txData.to == null (this is both true if txData.to is either undefined or null), however the linter did not like this. (explicit check if it is either null or undefined, reflected in current changes)

Why should the linter not approve the new statement, though?

I thought the new linter rules explicitly said you always have to explicitly check in conditional statements (even if the value in question is boolean). I guess not though for boolean values.

@davidmurdoch
Copy link
Contributor

Should the allowUnlimitedContractSize flag be considered here?

@jochem-brouwer
Copy link
Member Author

Good point @davidmurdoch!

Please note: MAX_CODE_SIZE (EIP 170) is the max code size of deployed code. However, with EIP 3860, we now have max initcode size (the init code of the contract which we want to deploy - which is thus different than runtime code)

If this is wished to be included, then we need a new flag: allowUnlimitedInitCodeSize (or something similar) since this situation is different. (runtime vs. initcode)

@davidmurdoch
Copy link
Contributor

davidmurdoch commented Mar 8, 2023

I think it's plausible that users that are already making use of allowUnlimitedContractSize to deploy "initcode" (tx.data) larger than this new MAX_INITCODE_SIZE limitation, so they'd be stuck without a way of also bailing out of this init code size check (Ganache's own tests actually fall victim to this limitation). I like the idea of using allowUnlimitedInitCodeSize as the option name and will update Ganache options to match.

@holgerd77
Copy link
Member

I guess not though for boolean values

No, it's not for boolean values, only for null and undefined. Just stumbled upon this lately as well, so if you have got a boolean option: myOption: boolean linter is fine with just a non-explicit check (which give me some relief), if you have got myOption?: boolean you need to do myOption === true.

Have the impression this is a very good balance.

Copy link
Member

@holgerd77 holgerd77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, also read up on the discussion and the issue here. LGTM.

@holgerd77 holgerd77 changed the title EIP3860 fixes EIP3860: only check max init code size on create contract tx Mar 9, 2023
@holgerd77
Copy link
Member

Short nit: please give such PRs a somewhat more expressive title since this will become the commit message (and also for the PR itself this is helpful).

(have done here now)

@holgerd77 holgerd77 merged commit 26a8dcd into master Mar 9, 2023
@holgerd77 holgerd77 deleted the eip3860fix branch March 9, 2023 13:07
acolytec3 added a commit that referenced this pull request Mar 20, 2023
* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
jochem-brouwer added a commit that referenced this pull request Mar 26, 2023
* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
acolytec3 added a commit that referenced this pull request Mar 27, 2023
* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
acolytec3 added a commit that referenced this pull request Mar 27, 2023
* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
acolytec3 added a commit that referenced this pull request Apr 3, 2023
* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
acolytec3 added a commit that referenced this pull request Apr 3, 2023
* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
acolytec3 added a commit that referenced this pull request Apr 3, 2023
* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
acolytec3 added a commit that referenced this pull request Apr 4, 2023
* `Buffer` to `Uint8Array` conversion (#2566)

* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>

* Devp2p status fix

* Remove buffer detritus

* Switch db to view

* buffer cleanup

* Correctly parse heads from DB

* Fix encodings

* Cast db values to uint8array

* Fix db bug in ethash

* Remove unused peerId check

* Add pow miner test

* Finish test

* Move pow test to integration tests

* client: lint

* rename test helper

* Add timeout to PoW test

* Fix test runner

* remove console log

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
acolytec3 added a commit that referenced this pull request Apr 11, 2023
* `Buffer` to `Uint8Array` conversion (#2566)

* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>

* Devp2p status fix

* Remove buffer detritus

* Switch db to view

* buffer cleanup

* Correctly parse heads from DB

* Fix encodings

* Cast db values to uint8array

* Fix db bug in ethash

* Remove unused peerId check

* Add pow miner test

* Finish test

* Move pow test to integration tests

* client: lint

* rename test helper

* Add timeout to PoW test

* Update eip4844 txs to decoupled blobs spec

* fix sharding spec and blobtx

* fix the sharding muli client run

* fix args

* fix compatibility with latest lodestar branch

* update test help

* Fix fee market test

* lint

* Move all kzg stuff to util

* Various cleanup

* Update to latest c-kzg

* Fix utils

* Remove outdated kzg references

* Fix client tests

* Update c-kzg dep

* Fix karma, remove Buffer references

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>
Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
g11tech added a commit that referenced this pull request Apr 13, 2023
* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
g11tech added a commit that referenced this pull request Apr 13, 2023
* `Buffer` to `Uint8Array` conversion (#2566)

* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>

* Devp2p status fix

* Remove buffer detritus

* Switch db to view

* buffer cleanup

* Correctly parse heads from DB

* Fix encodings

* Cast db values to uint8array

* Fix db bug in ethash

* Remove unused peerId check

* Add pow miner test

* Finish test

* Move pow test to integration tests

* client: lint

* rename test helper

* Add timeout to PoW test

* Fix test runner

* remove console log

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
g11tech added a commit that referenced this pull request Apr 13, 2023
* `Buffer` to `Uint8Array` conversion (#2566)

* V7 update to master 1 (#2593)

* Added v7 release reference in main README table (#2562)

* common: Schedule shanghai on goerli (#2563)

* common: Schedule shanghai on goerli

* update timestamp

* util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564)

* util/tx: Shift ssz back to case dependency free ES2019 compatible version

* update package lock

* update karma ecma version

* VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568)

* EVM: Memory Fix & Other Optimizations (#2570)

* EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:*

* VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging)

* EVM: CALL/CREATE debug exit msg differentiation

* EVM: avoid buffer copy in memory read (performance)

* EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification

* EVM: Added EIP check for transient storage checkpointing

* EVM: Precompile Debug Logger Improvements (#2572)

* EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging

* EVM: Added remaining precompile debug loggers

* EVM: added error cases to BLS precompile debug log

* EVM: Added missing precompile return value debug logs

* Small fixes

* tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575)

* EVM: Avoid memory.read() Memory Copy (#2573)

* EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode

* EVM: Add direct memory read to all calling opcodes

* EVM: Copy over memory on IDENTITY precompile

* EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow)

* Some optimizations

* blockchain: fix merge->clique transition (#2571)

* Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577)

* client/engine: ensure finalized/safe blocks are in canonical chain

* client: engine-api: fix finalized block check

* client/tests: fix forkchoice updated test

* client: add fcu tests to check if blocks are part of canonical chain

* client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579)

* client/engine: ensure invalid blockhash response matches spec (#2583)

* client/engine: delete invalid skeleton blocks (#2584)

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Setup to dev/test snapsync with sim architecture (#2574)

* Setup to dev/test snapsync with sim architecture

* modfiy single-run to setup a lodestar<>geth node to snapsync from

* setup an ethereumjs inline client and get it to peer with geth

* cleanup setup a bit

* snapsync run spec

* get the snap testdev sim working

* finalize the test infra and update usage doc

* enhance coverage

* Use geth RPC to connect to ethJS

* refac wait for snap sync completion

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* client: Add safe and finalized blockoptions to the chain (#2585)

* client: Add safe and finalized blockoptions to the chain

* fix tests

* fix more tests

* fix remaining

* cleanup

* enhance coverage

* unset scheduled goerli timestamp based hfs colliding with test

* Client: Small Debug Helpers and CLI Improvements (#2586)

* Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks

* Client -> Execution: NumBlocksPerIteration (default: 50) as an option

* Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0

* Apply suggestions from code review

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* common: Schedule Shanghai on mainnet! (#2591)

* common: Schedule Shanghai on mainnet!

* clear hf timestamp for test

* VM: Diff-based Touched Accounts Checkpointing (#2581)

* VM: Switched to a more efficient diff-based way of touched account checkpointing

* VM: move accessed storage inefficient checkpointing problem to berlin, haha

* EVM: avoid memory copy in MLOAD opcode function

* Remove console.log() in EVM

* vmState: ensure touched accounts delete stack gets properly updated on commit

* vm/eei: save touched height

* vm/vmState: new possible fix for touched accounts

* vm/vmState: another attempt to fix touched accounts journaling

* vm: add journaling

* Check correct journal height on revert

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* First pass - most util changes

* Fix most account stuff

* Fix account test

* Many byte fixes

* util: fix constants tests

* remaining fixes

* Turn off ci jobs

* monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes

* util: update description of bytes exporT

* util: remove unused import

* common: use bytesToHex helper instead of toString('hex')

* trie: refactor non-test files to Uint8Array

* util: add binary string utils

* util: remove extra Uint8Array.from

* util: remove arrToBufArray util

* trie: adjust tests and fix outstanding issues

* util: remove binarystring utils and add compareBytes and randomBytes util

* common: refactor common with Uint8Array

* util: accept 0x-prefixed and non-prefixed hex strings in toBytes

* tx: refactor Buffer -> Uint8Array

* tx: remove unused import

* util: revert toBytes update

* block: refactor Buffer -> uint8array

* block: adjust import

* trie: refactor remaining buffer instances

* move devp2p to uint8Array

* statemanager: refactor buffer -> uint8array

* util: simplify zeros

* util: add concatBytesUnsafe

* ethash: partial migration

* ethash: update examples

* ethash: wip fixes

* ethash: more WIP

* ethash: ensure fnv input is read from mix, not mixView

* blockchain: migrate to uint8array

* ethash: renable all tests

* ethash: fix bytesReverse

* Fix miner tests

* many hexToBytes moves

* most of evm/vm moves

* evm: Fix all tests

* vm: more fixes

* More fixes

* vm: fix receipts encoding

* vm: fix tester

* client: refactor buffer to uint8array

* client: additional uint8 adjustments

* client: fix most tests

* fix remaining client unit tests

* reactivate most CI

* client: fix les test/protocol

* turn client CI on

* util: fix name typo

* lint

* Fix withdrawals

* remove buffarraytoarr

* Remove bufArrtoArr references

* Lint

* fix examples

* Fix difficulty test

* lint

* block: update randomBytes import

* replace randombytes import

* client: fix sim test util

* vm: fix example

* devp2p: update snappy typing and fix tests

* Fix tests

* Remove additional buffer references

* rustbn fixes

* add 0x prefix to precompile address

* Remove `node-ip` dependency and buffer references in devp2p

* Switch slice to subarray

* evm: fix blake2f

* Merge fixes

* more merge commit fixes

* more test fixes

* Address all the feedback

* fix dns test

* Update packages/util/src/bytes.ts

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>

* Fix return type for baToJSON

* util: instantiate hexByByte array

* Remove baToJson

* rebase fixes

* Fix event typing

* Revert outdated initcode changes

* lint

---------

Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: g11tech <gajinder@g11.in>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>

* Devp2p status fix

* Remove buffer detritus

* Switch db to view

* buffer cleanup

* Correctly parse heads from DB

* Fix encodings

* Cast db values to uint8array

* Fix db bug in ethash

* Remove unused peerId check

* Add pow miner test

* Finish test

* Move pow test to integration tests

* client: lint

* rename test helper

* Add timeout to PoW test

* Update eip4844 txs to decoupled blobs spec

* fix sharding spec and blobtx

* fix the sharding muli client run

* fix args

* fix compatibility with latest lodestar branch

* update test help

* Fix fee market test

* lint

* Move all kzg stuff to util

* Various cleanup

* Update to latest c-kzg

* Fix utils

* Remove outdated kzg references

* Fix client tests

* Update c-kzg dep

* Fix karma, remove Buffer references

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>
Co-authored-by: Holger Drewes <Holger.Drewes@gmail.com>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Gabriel Rocheleau <contact@rockwaterweb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants