Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add assertions to sim tests #4276

Closed
6 of 15 tasks
dapplion opened this issue Jul 10, 2022 · 0 comments · Fixed by #4510, #4573, #4582, #4583 or #4584
Closed
6 of 15 tasks

Add assertions to sim tests #4276

dapplion opened this issue Jul 10, 2022 · 0 comments · Fixed by #4510, #4573, #4582, #4583 or #4584
Assignees
Labels
prio-high Resolve issues as soon as possible. scope-testing Issues for adding test coverage, fixing existing tests or testing strategies.

Comments

@dapplion
Copy link
Contributor

dapplion commented Jul 10, 2022

Sim tests are a crucial tool for our CI pipeline, but currently lack of assertions to detect bugs. Currently tests only assert that:

  • Finality / Justification is reached within N epochs

Proposed assertions

Assertion on epoch:

  • Attester participation rates a minimum threshold: Assert with debug states route
  • Sync committee participation rates a minimum threshold: Assert with following beacon blocks
  • All blocks in epoch are proposed: Assert with debug state route
  • Blocks received by all nodes with max threshold: ?? How to assert?
  • Gossip scores of all peers at a minimum value: Prysm, Lighthouse, and Lodestar expose peer scores via custom API routes
  • Correct voting of a majority vote of eth1_data

Run modes / actions to perform

  • Run to finalization + N epochs
  • Validator exit: Assert inclusion with blocks, effect with validator status route
  • Validator proposer slashing: Assert inclusion with blocks, effect with validator status route
  • Validator attester slashing: Assert inclusion with blocks, effect with validator status route
  • Process deposits from eth1 contract: Assert inclusion with blocks, effect with validator status route
  • Go through forks: Other assertions cover healthy transition
  • Finalized sync: Assert reaching head within some time limit
  • Unknown block sync: ?? Not sure how to trigger
  • Lightclient sync + follow head: Light-client should be managed programatically against a node run in another process. Assert syncs to head + follows the head in time

Background

To get started, Prysm has done an excellent job on this field so we should base our work on theirs.

@dapplion dapplion added the scope-testing Issues for adding test coverage, fixing existing tests or testing strategies. label Jul 10, 2022
@philknows philknows added the prio-high Resolve issues as soon as possible. label Jul 13, 2022
@nazarhussain nazarhussain linked a pull request Sep 21, 2022 that will close this issue
@nazarhussain nazarhussain linked a pull request Sep 21, 2022 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
prio-high Resolve issues as soon as possible. scope-testing Issues for adding test coverage, fixing existing tests or testing strategies.
Projects
Archived in project
3 participants