Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add assertoor to additional toolings #419

Merged
merged 31 commits into from
Jan 9, 2024
Merged

feat: add assertoor to additional toolings #419

merged 31 commits into from
Jan 9, 2024

Conversation

pk910
Copy link
Member

@pk910 pk910 commented Dec 19, 2023

Added assertoor testnet testing tool.

There are currently 6 tests included:

  • Check for chain stability (finalizing, high voting consensus, no reorgs)
  • Check if all clients propose a block
  • Validator Lifecycle Test
    Test for the whole validator lifecycle, includes Deposits, BLSChanges, Exits & Slashings
    This Test takes up to 2 days to run through due to deposit delays & queues.
  • Normal Transaction Test
    Test if normal EOA transactions (type 0/2) are processed by all client pairs (inclusion & submission)
  • Blob Transaction Test
    Test if blob transactions (type 3) are processed by all client pairs (inclusion & submission)
  • All-Opcodes Transaction Test
    Generates & submits a transaction that triggers every EVM opcode once.
    Checks if the transaction succeeds and no client forks off

@pk910 pk910 changed the title Added assertoor to additional toolings feat: add assertoor to additional toolings Dec 19, 2023
@pk910
Copy link
Member Author

pk910 commented Jan 3, 2024

Ready for first review :)

I'm using this config to run through the tests faster:

participants:
  - el_client_type: geth
    el_client_image: ethpandaops/geth:master-5b57727
    cl_client_type: lighthouse
    cl_client_image: ethpandaops/lighthouse:sidecar-inclusion-proof-ee3767c
    count: 1
  - el_client_type: geth
    el_client_image: ethpandaops/geth:master-5b57727
    cl_client_type: teku
    cl_client_image: ethpandaops/teku:master-992b224
    count: 1
network_params:
  deneb_fork_epoch: 1
  genesis_delay: 100
  seconds_per_slot: 3
  num_validator_keys_per_node: 250
launch_additional_services: true
additional_services:
  - dora
  - assertoor
snooper_enabled: true
assertoor_params:
  run_stability_check: true
  run_block_proposal_check: true
  run_lifecycle_test: true
  run_transaction_test: true
  run_blob_transaction_test: true
  run_opcodes_transaction_test: true

run_lifecycle_test will take quite long to complete (up to 48h).
With default network params, the deposits cannot be voted in before voting period 2, which starts at epoch 128.
If everything goes well from there, the votes reach consensus at about epoch 160.

All other tests are much faster.

@pk910 pk910 marked this pull request as ready for review January 3, 2024 23:06
@barnabasbusa
Copy link
Contributor

Could you please add a new config as a github test, so it is ran overnight?

How long should one of these tests last? (assuming run_lifecycle_test: false )

@pk910
Copy link
Member Author

pk910 commented Jan 4, 2024

Could you please add a new config as a github test, so it is ran overnight?

I don't get that. Yea, I think that's actually the plan to run this test later on :)
But it's out of scope for this PR, as it'll be in a separate repository targeting our own runner for it?

How long should one of these tests last? (assuming run_lifecycle_test: false )

The other tests (except the validator lifecycle one) shouldn't take too long.
It should run through in a few epochs.

The first two "check"-tests are not doing anything, just monitoring the network for about 1-2 epochs.

The transaction related tests shouldn't take longer than a few mins.
It depends on when all clients got a chance to propose a block with a relevant TX.

ETA with default network params:

  • Stability check: 30mins
  • Every client pair proposed a block: 1min
  • Transaction Test: 5min
  • Blob Transaction Test: 3min
  • All-Opcodes Transaction Test: 5min
  • Validator Lifecycle Test: ~29h
    (It takes 9h30min for a testnet with 4sec slot time)

@pk910 pk910 marked this pull request as draft January 5, 2024 02:40
@pk910 pk910 marked this pull request as ready for review January 8, 2024 15:27
@pk910 pk910 merged commit 76dde3e into main Jan 9, 2024
17 checks passed
@pk910 pk910 deleted the pk910/assertoor branch January 9, 2024 17:29
h4ck3rk3y pushed a commit that referenced this pull request Jan 23, 2024
🤖 I have created a release *beep* *boop*
---


##
[1.3.0](1.2.0...1.3.0)
(2024-01-22)


### Features

* add assertoor to additional toolings
([#419](#419))
([76dde3e](76dde3e))
* add devnets support
([#384](#384))
([2bae099](2bae099))
* add pitfalls for persistent storage as a warning
([#441](#441))
([69da8f0](69da8f0))
* add support for testnets
([#437](#437))
([5584cc8](5584cc8))
* Add Xatu Sentry
([#466](#466))
([b9523cb](b9523cb))
* enable checkpoint sync for devnets
([#448](#448))
([b367cfe](b367cfe))
* enable persistence
([#422](#422))
([8d40056](8d40056))
* enable syncing ephemery
([#459](#459))
([f8289cb](f8289cb))
* enable syncing shadowforks
([#457](#457))
([313a586](313a586))


### Bug Fixes

* add CL genesis delay to final genesis time
([#469](#469))
([e36027b](e36027b))
* add prysm-multiarch upstream image
([#451](#451))
([6feba23](6feba23))
* added supprot for boot enr file
([#456](#456))
([fd26e5c](fd26e5c))
* bump max mem limit for nimbus on holesky
([#439](#439))
([fb84787](fb84787))
* dora template fix
([#452](#452))
([f9243ea](f9243ea))
* enable ws for geth
([#446](#446))
([d5bf451](d5bf451))
* erigon chain should be set to dev
([#447](#447))
([1f40d84](1f40d84))
* erigon command arg
([#454](#454))
([5ae56a1](5ae56a1))
* fix typo
([#440](#440))
([933a313](933a313))
* guid fix for besu/teku/erigon/nimbus
([#443](#443))
([2283464](2283464))
* increase db size for geth
([#453](#453))
([0c67998](0c67998))
* logging bug
([#462](#462))
([f6098a1](f6098a1))
* parallel key generation
([#423](#423))
([060fd8f](060fd8f))
* re-add networkid
([#464](#464))
([4d96409](4d96409))
* typo
([#445](#445))
([e61c58a](e61c58a))
* Update nethermind to expose host on 0.0.0.0
([#467](#467))
([0bd29dd](0bd29dd))
* use all enrs for nimbus via bootstrap file
([#450](#450))
([bb5a0c1](bb5a0c1))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
bharath-123 pushed a commit to bharath-123/eth2-package that referenced this pull request Feb 18, 2024
🤖 I have created a release *beep* *boop*
---


##
[1.3.0](ethpandaops/ethereum-package@1.2.0...1.3.0)
(2024-01-22)


### Features

* add assertoor to additional toolings
([ethpandaops#419](ethpandaops#419))
([76dde3e](ethpandaops@76dde3e))
* add devnets support
([ethpandaops#384](ethpandaops#384))
([2bae099](ethpandaops@2bae099))
* add pitfalls for persistent storage as a warning
([ethpandaops#441](ethpandaops#441))
([69da8f0](ethpandaops@69da8f0))
* add support for testnets
([ethpandaops#437](ethpandaops#437))
([5584cc8](ethpandaops@5584cc8))
* Add Xatu Sentry
([ethpandaops#466](ethpandaops#466))
([b9523cb](ethpandaops@b9523cb))
* enable checkpoint sync for devnets
([ethpandaops#448](ethpandaops#448))
([b367cfe](ethpandaops@b367cfe))
* enable persistence
([ethpandaops#422](ethpandaops#422))
([8d40056](ethpandaops@8d40056))
* enable syncing ephemery
([ethpandaops#459](ethpandaops#459))
([f8289cb](ethpandaops@f8289cb))
* enable syncing shadowforks
([ethpandaops#457](ethpandaops#457))
([313a586](ethpandaops@313a586))


### Bug Fixes

* add CL genesis delay to final genesis time
([ethpandaops#469](ethpandaops#469))
([e36027b](ethpandaops@e36027b))
* add prysm-multiarch upstream image
([ethpandaops#451](ethpandaops#451))
([6feba23](ethpandaops@6feba23))
* added supprot for boot enr file
([ethpandaops#456](ethpandaops#456))
([fd26e5c](ethpandaops@fd26e5c))
* bump max mem limit for nimbus on holesky
([ethpandaops#439](ethpandaops#439))
([fb84787](ethpandaops@fb84787))
* dora template fix
([ethpandaops#452](ethpandaops#452))
([f9243ea](ethpandaops@f9243ea))
* enable ws for geth
([ethpandaops#446](ethpandaops#446))
([d5bf451](ethpandaops@d5bf451))
* erigon chain should be set to dev
([ethpandaops#447](ethpandaops#447))
([1f40d84](ethpandaops@1f40d84))
* erigon command arg
([ethpandaops#454](ethpandaops#454))
([5ae56a1](ethpandaops@5ae56a1))
* fix typo
([ethpandaops#440](ethpandaops#440))
([933a313](ethpandaops@933a313))
* guid fix for besu/teku/erigon/nimbus
([ethpandaops#443](ethpandaops#443))
([2283464](ethpandaops@2283464))
* increase db size for geth
([ethpandaops#453](ethpandaops#453))
([0c67998](ethpandaops@0c67998))
* logging bug
([ethpandaops#462](ethpandaops#462))
([f6098a1](ethpandaops@f6098a1))
* parallel key generation
([ethpandaops#423](ethpandaops#423))
([060fd8f](ethpandaops@060fd8f))
* re-add networkid
([ethpandaops#464](ethpandaops#464))
([4d96409](ethpandaops@4d96409))
* typo
([ethpandaops#445](ethpandaops#445))
([e61c58a](ethpandaops@e61c58a))
* Update nethermind to expose host on 0.0.0.0
([ethpandaops#467](ethpandaops#467))
([0bd29dd](ethpandaops@0bd29dd))
* use all enrs for nimbus via bootstrap file
([ethpandaops#450](ethpandaops#450))
([bb5a0c1](ethpandaops@bb5a0c1))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants