From ad2dff688b05b495e68f3c5ac2fe45f914e02020 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 27 Jan 2023 16:20:19 +0100 Subject: [PATCH 01/18] Add docs --- docs/beacon-chain-template.md | 84 +++++++++++++++++++++++++++++++++++ docs/new-feature.md | 17 +++++++ 2 files changed, 101 insertions(+) create mode 100644 docs/beacon-chain-template.md create mode 100644 docs/new-feature.md diff --git a/docs/beacon-chain-template.md b/docs/beacon-chain-template.md new file mode 100644 index 0000000000..4d22d3908e --- /dev/null +++ b/docs/beacon-chain-template.md @@ -0,0 +1,84 @@ +# `beacon-chain.md` Template + +# -- The Beacon Chain + +## Table of contents + + + + + + + + + +## Introduction + +## Notation + +## Custom types + +## Constants + +### [CATEGORY OF CONSTANTS] + +| Name | Value | +| - | - | +| `` | ``` | + +## Preset + + +### [CATEGORY OF PRESETS] + +| Name | Value | +| - | - | +| `` | `` | + +## Configuration + +### [CATEGORY OF CONFIGURATIONS] + +| Name | Value | +| - | - | +| `` | `` | + +## Containers + +### [CATEGORY OF CONTAINERS] + +#### `CONTAINER_NAME` + +```python +class CONTAINER_NAME(Container): + FILED_NAME: SSZ_TYPE +``` + +## Helper functions + +### [CATEGORY OF HELPERS] + +```python + +``` + +### Epoch processing + + +### Block processing + + + + +## Testing + +*Note*: The function `initialize_beacon_state_from_eth1` is modified for pure testing only. + +```python +def initialize_beacon_state_from_eth1(eth1_block_hash: Hash32, + eth1_timestamp: uint64, + deposits: Sequence[Deposit], + execution_payload_header: ExecutionPayloadHeader=ExecutionPayloadHeader() + ) -> BeaconState: + ... +``` diff --git a/docs/new-feature.md b/docs/new-feature.md new file mode 100644 index 0000000000..fa8f80ae89 --- /dev/null +++ b/docs/new-feature.md @@ -0,0 +1,17 @@ +# How to add a new feature proposal in consensus-specs + +1. Create a folder under `./specs/features`. +2. Choose the "previous fork" to extend: usually, use the scheduled or the lasted mainnet fork version. +3. Write down your proposed `beacon-chain.md` change: + - Template: https://notes.ethereum.org/@hww/beacon-chain-template + - Reference to the previous fork content +4. Add `fork.md`: reference to the previous `fork.md` file. +5. Make it executable: + - Update [`constants.py`](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/helpers/constants.py) with the new feature name. + - Update [`setup.py`](https://github.com/ethereum/consensus-specs/blob/dev/setup.py): + - Add a new `SpecBuilder` with the new feature name constant + - Add the new `SpecBuilder` to `spec_builders` + - Add the path of the new markdown files in `finalize_options` +6. Bonus: + - Add `validator.md` if honest validator behavior changes with your change. +7. Need help? Tag spec elves to clean up your PR. From 69fbe7fa3a4f2f8984911dccc0f768ea35b75756 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 30 Jan 2023 14:35:32 +0100 Subject: [PATCH 02/18] update link to template --- docs/new-feature.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/new-feature.md b/docs/new-feature.md index fa8f80ae89..655a4e03f6 100644 --- a/docs/new-feature.md +++ b/docs/new-feature.md @@ -3,7 +3,7 @@ 1. Create a folder under `./specs/features`. 2. Choose the "previous fork" to extend: usually, use the scheduled or the lasted mainnet fork version. 3. Write down your proposed `beacon-chain.md` change: - - Template: https://notes.ethereum.org/@hww/beacon-chain-template + - [Beacon Chain Spec Template](./beacon-chain-template.md) - Reference to the previous fork content 4. Add `fork.md`: reference to the previous `fork.md` file. 5. Make it executable: From 4cadc61403b57a80a9178db1036fdc8bb7a3cadc Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 28 Feb 2023 21:56:43 +0800 Subject: [PATCH 03/18] Add more info --- docs/new-feature.md | 53 ++++++++++++++++++++++++++++++++------------- 1 file changed, 38 insertions(+), 15 deletions(-) diff --git a/docs/new-feature.md b/docs/new-feature.md index 655a4e03f6..c089c5e64e 100644 --- a/docs/new-feature.md +++ b/docs/new-feature.md @@ -1,17 +1,40 @@ # How to add a new feature proposal in consensus-specs -1. Create a folder under `./specs/features`. -2. Choose the "previous fork" to extend: usually, use the scheduled or the lasted mainnet fork version. -3. Write down your proposed `beacon-chain.md` change: - - [Beacon Chain Spec Template](./beacon-chain-template.md) - - Reference to the previous fork content -4. Add `fork.md`: reference to the previous `fork.md` file. -5. Make it executable: - - Update [`constants.py`](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/helpers/constants.py) with the new feature name. - - Update [`setup.py`](https://github.com/ethereum/consensus-specs/blob/dev/setup.py): - - Add a new `SpecBuilder` with the new feature name constant - - Add the new `SpecBuilder` to `spec_builders` - - Add the path of the new markdown files in `finalize_options` -6. Bonus: - - Add `validator.md` if honest validator behavior changes with your change. -7. Need help? Tag spec elves to clean up your PR. +### 1. Create a folder under `./specs/_features` + +For example, if it's an `EIP-9999` CL spec, you can create a `./specs/_features/eip9999` folder. + +### 2. Choose the "previous fork" to extend: usually, use the scheduled or the latest mainnet fork version. + +For example, if the latest fork is Capella, use `./specs/capella` content as your "previous fork". + +### 3. Write down your proposed `beacon-chain.md` change +- You can either use [Beacon Chain Spec Template](./beacon-chain-template.md), or make a copy of the latest fork content and then edit it. +- Tips: + - We use [`doctoc`](https://www.npmjs.com/package/doctoc) tool to generate the table of content. + ``` + cd consensus-specs + doctoc specs + ``` + - The differences between "Constants", "Configurations", and "Presets": + - Constants: The constant that should never be changed. + - Configurations: The settings that we may change for different networks. + - Presets: The settings that we may change for testing. + - Readability and simplicity are more important than efficiency and optimization. + - Use simple Python rather than the fancy Python dark magic. + +### 4. Add `fork.md` +You can refer to the previous fork's `fork.md` file. + +### 5. Make it executable +- Update [`constants.py`](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/helpers/constants.py) with the new feature name. +- Update [`setup.py`](https://github.com/ethereum/consensus-specs/blob/dev/setup.py): + - Add a new `SpecBuilder` with the new feature name constant. e.g., `EIP9999SpecBuilder` + - Add the new `SpecBuilder` to `spec_builders` list. + - Add the path of the new markdown files in `finalize_options` function. + +### Bonus +- Add `validator.md` if honest validator behavior changes with your change. + +### Need help? +You can tag spec elves for cleaning up your PR. 🧚 From 99c6a48257ee450ef1c4d43919a95b36b5c93089 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 31 Mar 2023 18:02:07 +0800 Subject: [PATCH 04/18] Try mkdocs --- .gitignore | 7 +++ Makefile | 23 ++++++++ docs/README.md | 70 +++++++++++++++++++++++ docs/{ => docs}/beacon-chain-template.md | 0 docs/{ => docs}/new-feature.md | 0 mkdocs.yml | 3 + setup.py | 1 + specs/_features/eip6110/beacon-chain.md | 2 +- specs/_features/sharding/p2p-interface.md | 2 +- 9 files changed, 106 insertions(+), 2 deletions(-) create mode 100644 docs/README.md rename docs/{ => docs}/beacon-chain-template.md (100%) rename docs/{ => docs}/new-feature.md (100%) create mode 100644 mkdocs.yml diff --git a/.gitignore b/.gitignore index c56a658ce2..1f4c8818a2 100644 --- a/.gitignore +++ b/.gitignore @@ -35,3 +35,10 @@ tests/core/pyspec/eth2spec/test_results.xml # TOC tool outputs temporary files *.tmp + +# docs reader build +docs/specs +docs/sync +docs/ssz +docs/fork_choice +site diff --git a/Makefile b/Makefile index 1ec399e3a8..892b70661c 100644 --- a/Makefile +++ b/Makefile @@ -19,6 +19,11 @@ GENERATORS = $(sort $(dir $(wildcard $(GENERATOR_DIR)/*/.))) # Map this list of generator paths to "gen_{generator name}" entries GENERATOR_TARGETS = $(patsubst $(GENERATOR_DIR)/%/, gen_%, $(GENERATORS)) GENERATOR_VENVS = $(patsubst $(GENERATOR_DIR)/%, $(GENERATOR_DIR)/%venv, $(GENERATORS)) +# Documents +DOCS_DIR = ./docs +SSZ_DIR = ./ssz +SYNC_DIR = ./sync +FORK_CHOICE_DIR = ./fork_choice # To check generator matching: #$(info $$GENERATOR_TARGETS is [${GENERATOR_TARGETS}]) @@ -214,3 +219,21 @@ detect_generator_incomplete: $(TEST_VECTOR_DIR) detect_generator_error_log: $(TEST_VECTOR_DIR) [ -f $(GENERATOR_ERROR_LOG_FILE) ] && echo "[ERROR] $(GENERATOR_ERROR_LOG_FILE) file exists" || echo "[PASSED] error log file does not exist" + + +# For docs reader +install_docs: + python3 -m venv venv; . venv/bin/activate; python3 -m pip install -e .[docs]; + +build_docs: + . venv/bin/activate; + cp -r $(SPEC_DIR) $(DOCS_DIR); + cp -r $(SYNC_DIR) $(DOCS_DIR); + cp -r $(SSZ_DIR) $(DOCS_DIR); + cp -r $(FORK_CHOICE_DIR) $(DOCS_DIR); + cp $(CURRENT_DIR)/README.md $(DOCS_DIR)/README.md + mkdocs build + +serve_docs: + . venv/bin/activate; + mkdocs serve diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000000..9f2528263e --- /dev/null +++ b/docs/README.md @@ -0,0 +1,70 @@ +# Ethereum Proof-of-Stake Consensus Specifications + +[![Join the chat at https://discord.gg/qGpsxSA](https://img.shields.io/badge/chat-on%20discord-blue.svg)](https://discord.gg/qGpsxSA) [![Join the chat at https://gitter.im/ethereum/sharding](https://badges.gitter.im/ethereum/sharding.svg)](https://gitter.im/ethereum/sharding?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) + +To learn more about proof-of-stake and sharding, see the [PoS documentation](https://ethereum.org/en/developers/docs/consensus-mechanisms/pos/), [sharding documentation](https://ethereum.org/en/upgrades/sharding/) and the [research compendium](https://notes.ethereum.org/s/H1PGqDhpm). + +This repository hosts the current Ethereum proof-of-stake specifications. Discussions about design rationale and proposed changes can be brought up and discussed as issues. Solidified, agreed-upon changes to the spec can be made through pull requests. + +## Specs + +[![GitHub release](https://img.shields.io/github/v/release/ethereum/eth2.0-specs)](https://github.com/ethereum/eth2.0-specs/releases/) [![PyPI version](https://badge.fury.io/py/eth2spec.svg)](https://badge.fury.io/py/eth2spec) + +Core specifications for Ethereum proof-of-stake clients can be found in [specs](specs/). These are divided into features. +Features are researched and developed in parallel, and then consolidated into sequential upgrades when ready. + +### Stable Specifications + +| Seq. | Code Name | Fork Epoch | Specs | +| - | - | - | - | +| 0 | **Phase0** |`0` |
  • Core
    • [The beacon chain](specs/phase0/beacon-chain.md)
    • [Deposit contract](specs/phase0/deposit-contract.md)
    • [Beacon chain fork choice](specs/phase0/fork-choice.md)
  • Additions
    • [Honest validator guide](specs/phase0/validator.md)
    • [P2P networking](specs/phase0/p2p-interface.md)
    • [Weak subjectivity](specs/phase0/weak-subjectivity.md)
| +| 1 | **Altair** | `74240` |
  • Core
    • [Beacon chain changes](specs/altair/beacon-chain.md)
    • [Altair fork](specs/altair/fork.md)
  • Additions
    • [Light client sync protocol](specs/altair/light-client/sync-protocol.md) ([full node](specs/altair/light-client/full-node.md), [light client](specs/altair/light-client/light-client.md), [networking](specs/altair/light-client/p2p-interface.md))
    • [Honest validator guide changes](specs/altair/validator.md)
    • [P2P networking](specs/altair/p2p-interface.md)
| +| 2 | **Bellatrix**
(["The Merge"](https://ethereum.org/en/upgrades/merge/)) | `144896` |
  • Core
    • [Beacon Chain changes](specs/bellatrix/beacon-chain.md)
    • [Bellatrix fork](specs/bellatrix/fork.md)
    • [Fork choice changes](specs/bellatrix/fork-choice.md)
  • Additions
    • [Honest validator guide changes](specs/bellatrix/validator.md)
    • [P2P networking](specs/bellatrix/p2p-interface.md)
| +| 3 | **Capella** | `194048` |
  • Core
    • [Beacon chain changes](specs/capella/beacon-chain.md)
    • [Capella fork](specs/capella/fork.md)
  • Additions
    • [Light client sync protocol changes](specs/capella/light-client/sync-protocol.md) ([fork](specs/capella/light-client/fork.md), [full node](specs/capella/light-client/full-node.md), [networking](specs/capella/light-client/p2p-interface.md))
    • [Validator additions](specs/capella/validator.md)
    • [P2P networking](specs/capella/p2p-interface.md)
| + +### In-development Specifications +| Code Name or Topic | Specs | Notes | +| - | - | - | +| Deneb (tentative) |
  • Core
    • [Beacon Chain changes](specs/deneb/beacon-chain.md)
    • [Deneb fork](specs/deneb/fork.md)
    • [Polynomial commitments](specs/deneb/polynomial-commitments.md)
    • [Fork choice changes](specs/deneb/fork-choice.md)
  • Additions
    • [Light client sync protocol changes](specs/deneb/light-client/sync-protocol.md) ([fork](specs/deneb/light-client/fork.md), [full node](specs/deneb/light-client/full-node.md), [networking](specs/deneb/light-client/p2p-interface.md))
    • [Honest validator guide changes](specs/deneb/validator.md)
    • [P2P networking](specs/deneb/p2p-interface.md)
| +| Sharding (outdated) |
  • Core
    • [Beacon Chain changes](specs/_features/sharding/beacon-chain.md)
  • Additions
    • [P2P networking](specs/_features/sharding/p2p-interface.md)
| +| Custody Game (outdated) |
  • Core
    • [Beacon Chain changes](specs/_features/custody_game/beacon-chain.md)
  • Additions
    • [Honest validator guide changes](specs/_features/custody_game/validator.md)
| Dependent on sharding | +| Data Availability Sampling (outdated) |
  • Core
    • [Core types and functions](specs/_features/das/das-core.md)
    • [Fork choice changes](specs/_features/das/fork-choice.md)
  • Additions
    • [P2P Networking](specs/_features/das/p2p-interface.md)
    • [Sampling process](specs/_features/das/sampling.md)
|
  • Dependent on sharding
  • [Technical explainer](https://hackmd.io/@HWeNw8hNRimMm2m2GH56Cw/B1YJPGkpD)
| +| EIP-6110 |
  • Core
    • [Beacon Chain changes](specs/_features/eip6110//beacon-chain.md)
    • [EIP-6110 fork](specs/_features/eip6110/fork.md)
  • Additions
    • [Honest validator guide changes](specs/_features/eip6110/validator.md)
| + +### Accompanying documents can be found in [specs](specs) and include: + +* [SimpleSerialize (SSZ) spec](ssz/simple-serialize.md) +* [Merkle proof formats](ssz/merkle-proofs.md) +* [General test format](tests/formats/README.md) + +## Additional specifications for client implementers + +Additional specifications and standards outside of requisite client functionality can be found in the following repos: + +* [Beacon APIs](https://github.com/ethereum/beacon-apis) +* [Beacon Metrics](https://github.com/ethereum/beacon-metrics/) + +## Design goals + +The following are the broad design goals for the Ethereum proof-of-stake consensus specifications: +* to minimize complexity, even at the cost of some losses in efficiency +* to remain live through major network partitions and when very large portions of nodes go offline +* to select all components such that they are either quantum secure or can be easily swapped out for quantum secure counterparts when available +* to utilize crypto and design techniques that allow for a large participation of validators in total and per unit time +* to allow for a typical consumer laptop with `O(C)` resources to process/validate `O(1)` shards (including any system level validation such as the beacon chain) + +## Useful external resources + +* [Design Rationale](https://notes.ethereum.org/s/rkhCgQteN#) +* [Phase 0 Onboarding Document](https://notes.ethereum.org/s/Bkn3zpwxB) +* [Combining GHOST and Casper paper](https://arxiv.org/abs/2003.03052) + +## For spec contributors + +Documentation on the different components used during spec writing can be found here: +* [YAML Test Generators](tests/generators/README.md) +* [Executable Python Spec, with Py-tests](tests/core/pyspec/README.md) + +## Consensus spec tests + +Conformance tests built from the executable python spec are available in the [Ethereum Proof-of-Stake Consensus Spec Tests](https://github.com/ethereum/consensus-spec-tests) repo. Compressed tarballs are available in [releases](https://github.com/ethereum/consensus-spec-tests/releases). diff --git a/docs/beacon-chain-template.md b/docs/docs/beacon-chain-template.md similarity index 100% rename from docs/beacon-chain-template.md rename to docs/docs/beacon-chain-template.md diff --git a/docs/new-feature.md b/docs/docs/new-feature.md similarity index 100% rename from docs/new-feature.md rename to docs/docs/new-feature.md diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000000..1d104819e0 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,3 @@ +site_name: Consensus Specs +site_url: https://example.com/ +theme: readthedocs diff --git a/setup.py b/setup.py index 52bad2b71b..2af391db54 100644 --- a/setup.py +++ b/setup.py @@ -1181,6 +1181,7 @@ def run(self): "test": ["pytest>=4.4", "pytest-cov", "pytest-xdist"], "lint": ["flake8==5.0.4", "mypy==0.981", "pylint==2.15.3"], "generator": ["python-snappy==0.6.1", "filelock"], + "docs": ["mkdocs==1.4.2"] }, install_requires=[ "eth-utils>=2.0.0,<3", diff --git a/specs/_features/eip6110/beacon-chain.md b/specs/_features/eip6110/beacon-chain.md index 70a72a5f45..01999a929e 100644 --- a/specs/_features/eip6110/beacon-chain.md +++ b/specs/_features/eip6110/beacon-chain.md @@ -33,7 +33,7 @@ This is the beacon chain specification of in-protocol deposits processing mechanism. This mechanism relies on the changes proposed by [EIP-6110](http://eips.ethereum.org/EIPS/eip-6110). -*Note:* This specification is built upon [Capella](../../capella/beacon_chain.md) and is under active development. +*Note:* This specification is built upon [Capella](../../capella/beacon-chain.md) and is under active development. ## Constants diff --git a/specs/_features/sharding/p2p-interface.md b/specs/_features/sharding/p2p-interface.md index c29146fe9d..553444eff1 100644 --- a/specs/_features/sharding/p2p-interface.md +++ b/specs/_features/sharding/p2p-interface.md @@ -47,7 +47,7 @@ Following the same scheme as the [Phase0 gossip topics](../../phase0/p2p-interfa | `shard_column_{subnet_id}` | `SignedShardSample` | | `builder_block_bid` | `BuilderBlockBid` | -The [DAS network specification](./das-p2p.md) defines additional topics. +The [DAS network specification](../das/das-core.md) defines additional topics. #### Builder block bid From 104509656e1901d73b36d6ef0c76e54a6674080e Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 31 Mar 2023 18:18:53 +0800 Subject: [PATCH 05/18] Create docs.yml --- .github/workflows/docs.yml | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 .github/workflows/docs.yml diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 0000000000..1f6f8ae428 --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,28 @@ + +name: Publish docs via GitHub Pages +on: + push: + branches: + - master + +jobs: + build: + name: Deploy docs + runs-on: ubuntu-latest + steps: + - name: Build docs + run: | + make install_docs + make build_docs + + - name: Checkout main + uses: actions/checkout@v2 + + - name: Deploy docs + uses: mhausenblas/mkdocs-deploy-gh-pages@master + # Or use mhausenblas/mkdocs-deploy-gh-pages@nomaterial to build without the mkdocs-material theme + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + CONFIG_FILE: mkdocs.yml + EXTRA_PACKAGES: build-base + GITHUB_DOMAIN: hwwhww.github.io From 860e6f997a7335732c3da92781011f013775056a Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 31 Mar 2023 18:23:44 +0800 Subject: [PATCH 06/18] Update docs.yml --- .github/workflows/docs.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 1f6f8ae428..84440425fc 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -10,14 +10,14 @@ jobs: name: Deploy docs runs-on: ubuntu-latest steps: + - name: Checkout main + uses: actions/checkout@v2 + - name: Build docs run: | make install_docs make build_docs - - name: Checkout main - uses: actions/checkout@v2 - - name: Deploy docs uses: mhausenblas/mkdocs-deploy-gh-pages@master # Or use mhausenblas/mkdocs-deploy-gh-pages@nomaterial to build without the mkdocs-material theme From 45b11c08309f5d0eb9d4e700b53adfd174973e5c Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 31 Mar 2023 18:28:11 +0800 Subject: [PATCH 07/18] update --- .github/workflows/docs.yml | 4 +--- Makefile | 6 ++++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 84440425fc..c0780f406d 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -14,9 +14,7 @@ jobs: uses: actions/checkout@v2 - name: Build docs - run: | - make install_docs - make build_docs + run: make copy_docs - name: Deploy docs uses: mhausenblas/mkdocs-deploy-gh-pages@master diff --git a/Makefile b/Makefile index 892b70661c..ab5521663a 100644 --- a/Makefile +++ b/Makefile @@ -225,13 +225,15 @@ detect_generator_error_log: $(TEST_VECTOR_DIR) install_docs: python3 -m venv venv; . venv/bin/activate; python3 -m pip install -e .[docs]; -build_docs: - . venv/bin/activate; +copy_docs: cp -r $(SPEC_DIR) $(DOCS_DIR); cp -r $(SYNC_DIR) $(DOCS_DIR); cp -r $(SSZ_DIR) $(DOCS_DIR); cp -r $(FORK_CHOICE_DIR) $(DOCS_DIR); cp $(CURRENT_DIR)/README.md $(DOCS_DIR)/README.md + +build_docs: copy_docs + . venv/bin/activate; mkdocs build serve_docs: From 679334eb1d8e0516d6da3f6792e805b0073f7935 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 31 Mar 2023 18:29:47 +0800 Subject: [PATCH 08/18] update --- .github/workflows/docs.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index c0780f406d..b6d14c3f9d 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -23,4 +23,3 @@ jobs: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} CONFIG_FILE: mkdocs.yml EXTRA_PACKAGES: build-base - GITHUB_DOMAIN: hwwhww.github.io From d2eafdea9964410c26c1d1fce05c8d69bfc8ff39 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 31 Mar 2023 18:41:05 +0800 Subject: [PATCH 09/18] update --- .github/workflows/docs.yml | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index b6d14c3f9d..2b66100f39 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -1,25 +1,24 @@ -name: Publish docs via GitHub Pages +name: Publish docs on: push: branches: - - master - + - master +permissions: + contents: write jobs: - build: - name: Deploy docs + deploy: runs-on: ubuntu-latest steps: - - name: Checkout main - uses: actions/checkout@v2 - + - uses: actions/checkout@v3 - name: Build docs run: make copy_docs - - - name: Deploy docs - uses: mhausenblas/mkdocs-deploy-gh-pages@master - # Or use mhausenblas/mkdocs-deploy-gh-pages@nomaterial to build without the mkdocs-material theme - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - CONFIG_FILE: mkdocs.yml - EXTRA_PACKAGES: build-base + - uses: actions/setup-python@v4 + with: + python-version: 3.x + - uses: actions/cache@v2 + with: + key: ${{ github.ref }} + path: .cache + - run: pip install mkdocs-material + - run: mkdocs gh-deploy --force From 6c9f5d165b1765d8a19fc1b3f427fccbadb23ac9 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 31 Mar 2023 18:02:07 +0800 Subject: [PATCH 10/18] Try mkdocs --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 1f4c8818a2..82026c27bd 100644 --- a/.gitignore +++ b/.gitignore @@ -41,4 +41,5 @@ docs/specs docs/sync docs/ssz docs/fork_choice +docs/README.md site From 2c6a41b3d3b740c9486e02030d92896dc976f492 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 6 Apr 2023 13:04:00 +0800 Subject: [PATCH 11/18] Add "B: Make it executable for pytest and test generator" section --- docs/docs/new-feature.md | 129 +++++++++++++++++- .../{ => templates}/beacon-chain-template.md | 0 2 files changed, 126 insertions(+), 3 deletions(-) rename docs/docs/{ => templates}/beacon-chain-template.md (100%) diff --git a/docs/docs/new-feature.md b/docs/docs/new-feature.md index c089c5e64e..5e6180329f 100644 --- a/docs/docs/new-feature.md +++ b/docs/docs/new-feature.md @@ -1,5 +1,32 @@ # How to add a new feature proposal in consensus-specs + + +## Table of Contents + +- [A. Make it executable for linter checks](#a-make-it-executable-for-linter-checks) + - [1. Create a folder under `./specs/_features`](#1-create-a-folder-under-specs_features) + - [2. Choose the "previous fork" to extend: usually, use the scheduled or the latest mainnet fork version.](#2-choose-the-previous-fork-to-extend-usually-use-the-scheduled-or-the-latest-mainnet-fork-version) + - [3. Write down your proposed `beacon-chain.md` change](#3-write-down-your-proposed-beacon-chainmd-change) + - [4. Add `fork.md`](#4-add-forkmd) + - [5. Make it executable](#5-make-it-executable) +- [B: Make it executable for pytest and test generator](#b-make-it-executable-for-pytest-and-test-generator) + - [1. Add `light-client/*` docs if you updated the content of `BeaconBlock`](#1-add-light-client-docs-if-you-updated-the-content-of-beaconblock) + - [2. Add the mainnet and minimal presets and update the configs](#2-add-the-mainnet-and-minimal-presets-and-update-the-configs) + - [3. Update `context.py`](#3-update-contextpy) + - [4. Update `constants.py`](#4-update-constantspy) + - [5. Update `genesis.py`:](#5-update-genesispy) + - [6. To add fork transition tests, update fork_transition.py](#6-to-add-fork-transition-tests-update-fork_transitionpy) + - [7. Update CI configurations](#7-update-ci-configurations) +- [Others](#others) + - [Bonus](#bonus) + - [Need help?](#need-help) + + + + +## A. Make it executable for linter checks + ### 1. Create a folder under `./specs/_features` For example, if it's an `EIP-9999` CL spec, you can create a `./specs/_features/eip9999` folder. @@ -9,7 +36,7 @@ For example, if it's an `EIP-9999` CL spec, you can create a `./specs/_features/ For example, if the latest fork is Capella, use `./specs/capella` content as your "previous fork". ### 3. Write down your proposed `beacon-chain.md` change -- You can either use [Beacon Chain Spec Template](./beacon-chain-template.md), or make a copy of the latest fork content and then edit it. +- You can either use [Beacon Chain Spec Template](./templates/beacon-chain-template.md), or make a copy of the latest fork content and then edit it. - Tips: - We use [`doctoc`](https://www.npmjs.com/package/doctoc) tool to generate the table of content. ``` @@ -25,7 +52,6 @@ For example, if the latest fork is Capella, use `./specs/capella` content as you ### 4. Add `fork.md` You can refer to the previous fork's `fork.md` file. - ### 5. Make it executable - Update [`constants.py`](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/helpers/constants.py) with the new feature name. - Update [`setup.py`](https://github.com/ethereum/consensus-specs/blob/dev/setup.py): @@ -33,8 +59,105 @@ You can refer to the previous fork's `fork.md` file. - Add the new `SpecBuilder` to `spec_builders` list. - Add the path of the new markdown files in `finalize_options` function. +## B: Make it executable for pytest and test generator + +### 1. Add `light-client/*` docs if you updated the content of `BeaconBlock` +- You can refer to the previous fork's `light-client/*` file. +- Add the path of the new markdown files in `setup.py`'s `finalize_options` function. + +### 2. Add the mainnet and minimal presets and update the configs +- Add presets: `presets/mainnet/.yaml` and `presets/minimal/.yaml` +- Update configs: `configs/mainnet.yaml` and `configs/minimal.yaml` + +### 3. Update [`context.py`](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/context.py) +- Update `spec_targets` by adding `` + +```python +from eth2spec.eip9999 import mainnet as spec_eip9999_mainnet, minimal as spec_eip9999_minimal + +... + +spec_targets: Dict[PresetBaseName, Dict[SpecForkName, Spec]] = { + MINIMAL: { + ... + EIP9999: spec_eip9999_minimal, + }, + MAINNET: { + ... + EIP9999: spec_eip9999_mainnet + }, +} +``` + +### 4. Update [`constants.py`](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/helpers/constants.py) +- Add `` to `ALL_PHASES` and `TESTGEN_FORKS` + +### 5. Update [`genesis.py`](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/helpers/genesis.py): + +We use `create_genesis_state` to create the default `state` in tests. + +- Update `create_genesis_state` by adding `fork_version` setting: + +```python +def create_genesis_state(spec, validator_balances, activation_threshold): + ... + if spec.fork == ALTAIR: + current_version = spec.config.ALTAIR_FORK_VERSION + ... + elif spec.fork == EIP9999: + # Add the previous fork version of given fork + previous_version = spec.config. + current_version = spec.config.EIP9999_FORK_VERSION +``` + +- If the given feature changes `BeaconState` fields, you have to set the initial values by adding: + +```python +def create_genesis_state(spec, validator_balances, activation_threshold): + ... + if is_post_eip9999(spec): + state. = + + return state +``` + +- If the given feature changes `ExecutionPayload` fields, you have to set the initial values by updating `get_sample_genesis_execution_payload_header` helper. + +### 6. To add fork transition tests, update [fork_transition.py](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/helpers/fork_transition.py) + +```python +def do_fork(state, spec, post_spec, fork_epoch, with_block=True, sync_aggregate=None, operation_dict=None): + ... + + if post_spec.fork == ALTAIR: + state = post_spec.upgrade_to_altair(state) + ... + elif post_spec.fork == EIP9999: + state = post_spec.upgrade_to_eip9999(state) + + ... + + if post_spec.fork == ALTAIR: + assert state.fork.previous_version == post_spec.config.GENESIS_FORK_VERSION + assert state.fork.current_version == post_spec.config.ALTAIR_FORK_VERSION + ... + elif post_spec.fork == EIP9999: + assert state.fork.previous_version == post_spec.config. + assert state.fork.current_version == post_spec.config.EIP9999_FORK_VERSION + + ... +``` + +### 7. Update CI configurations +- Update [GitHub Actions config](https://github.com/ethereum/consensus-specs/blob/dev/.github/workflows/run-tests.yml) + - Update `pyspec-tests.strategy.matrix.version` list by adding new feature to it +- Update [CircleCI config](https://github.com/ethereum/consensus-specs/blob/dev/.circleci/config.yml) + - Add new job to the `workflows.test_spec.jobs` + +## Others + ### Bonus -- Add `validator.md` if honest validator behavior changes with your change. +- Add `validator.md` if honest validator behavior changes with the new feature. ### Need help? You can tag spec elves for cleaning up your PR. 🧚 diff --git a/docs/docs/beacon-chain-template.md b/docs/docs/templates/beacon-chain-template.md similarity index 100% rename from docs/docs/beacon-chain-template.md rename to docs/docs/templates/beacon-chain-template.md From 517913a84038f22286d5f991a64b1a2775bcb93d Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 6 Apr 2023 19:41:58 +0800 Subject: [PATCH 12/18] Use mkdocs-material --- mkdocs.yml | 12 +++++++++++- setup.py | 2 +- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/mkdocs.yml b/mkdocs.yml index 1d104819e0..3ec34d6dc6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,3 +1,13 @@ site_name: Consensus Specs site_url: https://example.com/ -theme: readthedocs +theme: + name: material +markdown_extensions: + - pymdownx.highlight: + anchor_linenums: true + use_pygments: true + pygments_lang_class: true + linenums: true + anchor_linenums: true + pygments_lang_class: true + - pymdownx.superfences diff --git a/setup.py b/setup.py index 2af391db54..96acd65fb2 100644 --- a/setup.py +++ b/setup.py @@ -1181,7 +1181,7 @@ def run(self): "test": ["pytest>=4.4", "pytest-cov", "pytest-xdist"], "lint": ["flake8==5.0.4", "mypy==0.981", "pylint==2.15.3"], "generator": ["python-snappy==0.6.1", "filelock"], - "docs": ["mkdocs==1.4.2"] + "docs": ["mkdocs==1.4.2", "mkdocs-material==9.1.5"] }, install_requires=[ "eth-utils>=2.0.0,<3", From ca7b32fee00976e11feb6c3389a41ffe27b13ea0 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 6 Apr 2023 21:34:36 +0800 Subject: [PATCH 13/18] Use `mkdocs-awesome-pages-plugin` to create custom specs order --- .github/workflows/docs.yml | 2 +- docs/.pages | 5 +++++ mkdocs.yml | 9 +++++++-- setup.py | 2 +- specs/.pages | 4 ++++ 5 files changed, 18 insertions(+), 4 deletions(-) create mode 100644 docs/.pages create mode 100644 specs/.pages diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 2b66100f39..eab3bba173 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -20,5 +20,5 @@ jobs: with: key: ${{ github.ref }} path: .cache - - run: pip install mkdocs-material + - run: pip install -e .[docs] - run: mkdocs gh-deploy --force diff --git a/docs/.pages b/docs/.pages new file mode 100644 index 0000000000..d9e382ede5 --- /dev/null +++ b/docs/.pages @@ -0,0 +1,5 @@ +nav: + - Home: + - README.md + - specs + - ... diff --git a/mkdocs.yml b/mkdocs.yml index 3ec34d6dc6..280911308e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,13 +1,18 @@ site_name: Consensus Specs site_url: https://example.com/ +repo_url: https://github.com/ethereum/consensus-specs theme: name: material + features: + - navigation.tabs markdown_extensions: - pymdownx.highlight: - anchor_linenums: true use_pygments: true pygments_lang_class: true linenums: true anchor_linenums: true - pygments_lang_class: true - pymdownx.superfences + - mdx_truly_sane_lists + +plugins: + - awesome-pages diff --git a/setup.py b/setup.py index 96acd65fb2..fc3acb8062 100644 --- a/setup.py +++ b/setup.py @@ -1181,7 +1181,7 @@ def run(self): "test": ["pytest>=4.4", "pytest-cov", "pytest-xdist"], "lint": ["flake8==5.0.4", "mypy==0.981", "pylint==2.15.3"], "generator": ["python-snappy==0.6.1", "filelock"], - "docs": ["mkdocs==1.4.2", "mkdocs-material==9.1.5"] + "docs": ["mkdocs==1.4.2", "mkdocs-material==9.1.5", "mdx-truly-sane-lists==1.3", "mkdocs-awesome-pages-plugin==2.8.0"] }, install_requires=[ "eth-utils>=2.0.0,<3", diff --git a/specs/.pages b/specs/.pages new file mode 100644 index 0000000000..7e47dc5f75 --- /dev/null +++ b/specs/.pages @@ -0,0 +1,4 @@ +nav: + - phase0 + - ... + - _features From e2116ecab7a7626c2bce7146c08a6903ad22214a Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 6 Apr 2023 21:44:42 +0800 Subject: [PATCH 14/18] Add toc permalink --- mkdocs.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index 280911308e..a8ebed5049 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -6,6 +6,8 @@ theme: features: - navigation.tabs markdown_extensions: + - toc: + permalink: true - pymdownx.highlight: use_pygments: true pygments_lang_class: true From 81fa6ac3be292d72836a73dfa6e0ca2fe28a825e Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 6 Apr 2023 21:45:23 +0800 Subject: [PATCH 15/18] Update site_url --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index a8ebed5049..f4bd2e1f9f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,5 +1,5 @@ site_name: Consensus Specs -site_url: https://example.com/ +site_url: https://ethereum.github.io/consensus-specs/ repo_url: https://github.com/ethereum/consensus-specs theme: name: material From 8903da5eb692a8e70a78a82d9bba0748ff5f47e6 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 6 Apr 2023 23:18:47 +0800 Subject: [PATCH 16/18] Add .pages files for navigations. Update highlight style --- docs/light-client/.pages | 5 +++++ docs/light-client/index.md | 1 + fork_choice/.pages | 7 +++++++ mkdocs.yml | 3 ++- 4 files changed, 15 insertions(+), 1 deletion(-) create mode 100644 docs/light-client/.pages create mode 100644 docs/light-client/index.md create mode 100644 fork_choice/.pages diff --git a/docs/light-client/.pages b/docs/light-client/.pages new file mode 100644 index 0000000000..a372a5d2e5 --- /dev/null +++ b/docs/light-client/.pages @@ -0,0 +1,5 @@ +nav: + - 'Index': index.md + - 'Altair': specs/altair/light-client/sync-protocol + - 'Capella': specs/capella/light-client/sync-protocol + - 'Deneb': specs/deneb/light-client/sync-protocol diff --git a/docs/light-client/index.md b/docs/light-client/index.md new file mode 100644 index 0000000000..32155b1852 --- /dev/null +++ b/docs/light-client/index.md @@ -0,0 +1 @@ +# Light client specifications diff --git a/fork_choice/.pages b/fork_choice/.pages new file mode 100644 index 0000000000..a5e6ccc904 --- /dev/null +++ b/fork_choice/.pages @@ -0,0 +1,7 @@ +nav: + - ... + - Fork Choice -- Core: + - phase0: specs/phase0/fork-choice + - bellatrix: specs/bellatrix/fork-choice + - capella: specs/capella/fork-choice + - deneb: specs/deneb/fork-choice diff --git a/mkdocs.yml b/mkdocs.yml index f4bd2e1f9f..b8af5c899e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -10,7 +10,8 @@ markdown_extensions: permalink: true - pymdownx.highlight: use_pygments: true - pygments_lang_class: true + noclasses: true + pygments_style: 'colorful' linenums: true anchor_linenums: true - pymdownx.superfences From f7569dfe239fb82fc73a05d4dd370be6b4ff5bd3 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 7 Apr 2023 00:21:59 +0800 Subject: [PATCH 17/18] Dark theme --- docs/stylesheets/extra.css | 34 ++++++++++++++++++++++++++++++++++ mkdocs.yml | 21 ++++++++++++++++++--- 2 files changed, 52 insertions(+), 3 deletions(-) create mode 100644 docs/stylesheets/extra.css diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css new file mode 100644 index 0000000000..3849762488 --- /dev/null +++ b/docs/stylesheets/extra.css @@ -0,0 +1,34 @@ +/* Reference: https://zenn.dev/mebiusbox/articles/81d977a72cee01 */ + +[data-md-color-scheme=default] { + --md-default-fg-color--light: #222 !important; +} +[data-md-color-scheme=slate] { + --md-default-fg-color--light: #fefefe !important; + --md-typeset-a-color: #fc0 !important; +} + +.md-typeset pre { + color: #f8f8f2; +} +.md-typeset .highlighttable { + margin-left:-20px; + margin-right: -20px; + border-radius: 0; +} +.md-typeset .highlighttable > * { + --md-code-bg-color: #222 !important; + --md-code-fg-color: #fefefe !important; +} +.md-typeset .highlighttable .linenos .linenodiv pre span { + background-color: #222 !important; + color: #fefefe !important; +} +.md-typeset .highlighttable .md-clipboard:before, +.md-typeset .highlighttable .md-clipboard:after { + color: rgba(240,240,240,.8); +} +.md-typeset .highlighttable .md-clipboard:hover:before, +.md-typeset .highlighttable .md-clipboard:hover:after { + color: rgba(102,217,224,1); +} diff --git a/mkdocs.yml b/mkdocs.yml index b8af5c899e..0e407bff43 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,21 +1,36 @@ site_name: Consensus Specs site_url: https://ethereum.github.io/consensus-specs/ +repo_name: ethereum/consensus-specs repo_url: https://github.com/ethereum/consensus-specs theme: name: material + palette: + - scheme: default + primary: black + toggle: + icon: material/brightness-7 + name: Switch to dark mode + - scheme: slate + primary: black + toggle: + icon: material/brightness-4 + name: Switch to light mode features: - navigation.tabs + - search markdown_extensions: - toc: permalink: true - pymdownx.highlight: use_pygments: true noclasses: true - pygments_style: 'colorful' + pygments_style: monokai linenums: true anchor_linenums: true - pymdownx.superfences - mdx_truly_sane_lists - plugins: - - awesome-pages + - search + - awesome-pages +extra_css: + - stylesheets/extra.css From 6089d7676311e1717739189e002e1a4b06fa23f5 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 18 Apr 2023 18:14:39 +0800 Subject: [PATCH 18/18] Fix list indent --- README.md | 4 ++++ mkdocs.yml | 12 ++++++++---- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 9f2528263e..a655d886d7 100644 --- a/README.md +++ b/README.md @@ -65,6 +65,10 @@ Documentation on the different components used during spec writing can be found * [YAML Test Generators](tests/generators/README.md) * [Executable Python Spec, with Py-tests](tests/core/pyspec/README.md) +## Online viewer of the latest release (latest `master` branch) + +[Ethereum Consensus Specs](https://ethereum.github.io/consensus-specs/) + ## Consensus spec tests Conformance tests built from the executable python spec are available in the [Ethereum Proof-of-Stake Consensus Spec Tests](https://github.com/ethereum/consensus-spec-tests) repo. Compressed tarballs are available in [releases](https://github.com/ethereum/consensus-spec-tests/releases). diff --git a/mkdocs.yml b/mkdocs.yml index 0e407bff43..dc6b352baa 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,7 +1,6 @@ -site_name: Consensus Specs +site_name: Ethereum Consensus Specs site_url: https://ethereum.github.io/consensus-specs/ repo_name: ethereum/consensus-specs -repo_url: https://github.com/ethereum/consensus-specs theme: name: material palette: @@ -21,16 +20,21 @@ theme: markdown_extensions: - toc: permalink: true + - pymdownx.superfences - pymdownx.highlight: use_pygments: true noclasses: true pygments_style: monokai linenums: true anchor_linenums: true - - pymdownx.superfences - - mdx_truly_sane_lists + - mdx_truly_sane_lists: + nested_indent: 4 plugins: - search - awesome-pages extra_css: - stylesheets/extra.css +extra: + social: + - icon: fontawesome/brands/github + link: https://github.com/ethereum/consensus-specs