Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Staged Test Network Plan #116

Closed
pipermerriam opened this issue Dec 7, 2021 · 11 comments
Closed

Staged Test Network Plan #116

pipermerriam opened this issue Dec 7, 2021 · 11 comments

Comments

@pipermerriam
Copy link
Member

pipermerriam commented Dec 7, 2021

I would like to outline some proposed structure for how we roll out our test network.

The goal of this test network is to iterate towards the full functionality necessary for the Chain History network.

Phases

Phase Zero: Overlay Only

During this phase we will simply establish our ability to establish a stable overlay network using the Portal Wire Protocol messages.

  • A: Clients support can be externally configured in some way to specify bootnodes.
  • B: Clients can use use the Portal Wire Protocol using the protocol identifier: 0x500B (History Network)
  • C: Clients can send and receive PING/PONG messages using the overlay protocol
    • "fake" radius information in the payload
  • D: Clients can send and respond to FINDNODE/FOUNDNODES messages using the overlay protocol.
  • E: Clients periodically check the liveliness of nodes in their routing table and evict unreachable nodes
  • F: Clients actively populate their routing table by exploring the DHT (typically via random exploration using the recursive find nodes algorithm)
  • G: Client has published instructions on how to build, configure, and launch a node for the test network.

In this phase, each team will be responsible for deploying however many nodes they wish. Each team may supply the ENR records for any stable "bootnodes" they will be operating.

What we are testing:

  1. Client interoperability and stability
    • Message encoding and decoding for PING/PONG/FINDNODE/FOUNDNODES
  2. Healthy overlay network
    • Do clients populate and maintain their routing tables as expected.
    • Can we navigate the network as expected to find nodes the closest nodes to a location.
Client A B C D E F G
Fluffy ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Trin ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Ultralight ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️

Phase One: Simple Content Retrieval

During this phase we will be testing the ability to find and retrieve content stored in the network by other nodes.

  • A. Clients have a basic implementation of a content database
    • The database can be manually populated with data
    • support for looking up a piece of content by it's content-id
  • B. Clients have support for the content-key format for headers referenced by their block hash.
  • C. Clients support a subset of the functionality for serving FINDCONTENT requests
    • Clients only need to support responding when the payload can be returned within the CONTENT message response.
    • Clients do not need to support uTP based transmission at this stage
  • D. A client can be launched with a pre-populated database of content
    • In this phase we will populate our test network with nodes which have had their content databases pre-populated.
    • We will likely do something like pre-loading the first 1 million block headers

What we are testing

  1. The ability for nodes to traverse the DHT and find the nodes which have the content they need
    • Any block header in the first million blocks should be retrievable by its block hash.
  2. The basic transmission of content without uTP
Client A B C D
Fluffy ✔️ ✔️ ✔️ ✔️
Trin ✔️ ✔️ ✔️ ✔️
Ultralight ✔️ ✔️ ✔️ ✔️

Phase Two: Gossip

During this phase we will build out the mechanisms needed for gossip, including use of the uTP protocol.

  • A. Clients support uTP
    • Listening for inbound connections
    • Establishing outbound connections
    • Receiving data sent over the uTP stream
  • B. Client support for gossip
    • Responding to OFFER messages if the content is of interest
    • Establishing outbound connection upon receiving an ACCEPT message and sending the data over uTP
    • Listening for an inbound connection after sending an ACCEPT message and receiving the data over uTP
    • Support for the block header content-key and validation of the block header.
    • Upon successful receipt of new content, performing neighborhood gossip.
  • C. Client support for the POKE mechanic.
    • Tracking of the nodes along the search path which did not have the content but should have been interested.
    • Once content was successfully retrieved, offering it to nodes along the search path.
  • D. Non critical but related functionality
    1. Management of the content database, eviction when full based on radius
    2. Validation that offered headers are indeed part of the canonical chain (depends on header accumulator)

What we are testing

  1. uTP based transmission of content
  2. The effective spread of new content using gossip
  3. The passive replication of existing content using POKE mechanic.
Client A B C D.i D.ii
Fluffy ✔️ ✔️ ✔️ ✔️ 🍕
Trin ✔️ ✔️ ✔️ ✔️ 🍕
Ultralight ✔️ ✔️ ✔️ ✔️

Phase Three: Head Tracking

TODO: we can no longer simply follow the chain with the highest TDD after merge. Need to implement light protocol for Eth2

NOTE: establish the "correct" solution and then look for a quick solution that will get us the needed functionality quickly.

Phase Four: Full Content Transmission

During this phase we will flesh out the remaining functionality for transmission of content, specifically, large payloads that must be sent over uTP

  • A. Clients support CONTENT response that contain a uTP connection ID
    • Upon receipt of a CONTENT response with a uTP connection-id, the client initiates a uTP stream and receives the content payload over the stream.
  • B. Clients serve FINDCONTENT requests for all content types (headers, bodies, receipt bundles, master and epoch accumulators)
  • C. Clients support block body and receipt bundle content types
  • D. Received block body and receipt content is validated using the block header.

What we are testing

  1. support for remaining content types (bodies and receipt bundles)
  2. clients implement full support for FINDCONTENT/CONTENT including full uTP based content transmission.
Client A B C D
Fluffy ✔️ ✔️ ✔️ ✔️
Trin ✔️ 🍕 ✔️ ✔️
Ultralight ✔️ ✔️ 🍕 ✔️

Phase Five: The Road To Production

TODO: what is left?

NOTE: visibility into the state of the network is probably valuable at this stage.

NOTE: verifying cross client compatibility will need to be done

@pipermerriam pipermerriam changed the title [WIP] Staged Test Network Plan Staged Test Network Plan Dec 8, 2021
@pipermerriam
Copy link
Member Author

Proposal for DevConnect goals:

  1. Each client has documentation on how to install and launch a node which will connect to the current testnet bootnodes.
  2. Ultralight has some documentation on how to launch and run the in-browser stuff.
  3. Hosting of at least a subset of the header chain available via the testnet.

@ogenev
Copy link
Member

ogenev commented Mar 1, 2022

3. Hosting of at least a subset of the header chain available via the testnet.

Does this include 1) and 2) from Phase Two: Gossip?

@pipermerriam
Copy link
Member Author

Does this include 1) and 2) from Phase Two: Gossip?

I think it can be done in whatever the fastest/quickest way that works is which doesn't diverge too far from the existing roadmap. Don't want us doing extra work that doesn't lead us towards the final product... but if we can have uTP streams by then in any capacity then huzzah!

@acolytec3
Copy link
Contributor

Have updated for Ultralight

@kdeme
Copy link
Collaborator

kdeme commented Sep 14, 2022

As have I for Fluffy

@ogenev
Copy link
Member

ogenev commented Sep 16, 2022

Updated for Trin

@pipermerriam
Copy link
Member Author

is 🍕 a no? partial?

@acolytec3
Copy link
Contributor

is 🍕 a no? partial?

Partial in our case

@ogenev
Copy link
Member

ogenev commented Sep 20, 2022

is pizza a no? partial?

Partial/in progress for Trin. Updated POKE with ❌ as It is not yet implemented.

@carver
Copy link
Contributor

carver commented Nov 11, 2022

POKE in trin is merged, I updated the table

@kdeme
Copy link
Collaborator

kdeme commented Oct 16, 2024

Closing as this is completed since a while for all clients mentioned in the issue tracker.

@kdeme kdeme closed this as completed Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants