Merge pull request #127 from rauljordan/client-revamp

Rearchitecting the Sharding Node, its Lifecycle, and Services

README.md

@@ -2,7 +2,7 @@
 
 ![Travis Build](https://travis-ci.org/prysmaticlabs/geth-sharding.svg?branch=master)
 
-This is the main repository for the sharding implementation of the go-ethereum client by [Prysmatic Labs](https://prysmaticlabs.com). For the original, go-ethereum project, refer to the following [link](https://github.com/ethereum/go-ethereum).
+This is the main repository for the sharding implementation for the go-ethereum project by [Prysmatic Labs](https://prysmaticlabs.com). For the original, go-ethereum project, refer to the following [link](https://github.com/ethereum/go-ethereum).
 
 Before you begin, check out our [Sharding Reference Implementation Doc](https://github.com/prysmaticlabs/geth-sharding/blob/master/sharding/README.md). This doc serves as the single source of truth for our team, our milestones, and details on the different components of our architecture.
 
@@ -17,7 +17,7 @@ Interested in contributing? Check out our [Contribution Guidelines](#contributio
     -   [Running a Local Geth Node](#running-a-local-geth-node)
     -   [Transaction Generator](#transaction-generator)
     -   [Becoming a Notary](#becoming-a-notary)
-    -   [Running a Collation Proposal Client](#running-a-collation-proposal-client)
+    -   [Running a Collation Proposal Node](#running-a-collation-proposal-node)
 -   [Testing](#testing)
 -   [Contribution Guidelines](#contribution-guidelines)
 -   [License](#license)
@@ -107,26 +107,32 @@ Our system outlined below follows the [Minimal Sharding Protocol](https://ethres
 To deposit ETH and join as a notary in the Sharding Manager Contract, run the following command:
 
 ```
-geth sharding-notary --deposit --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
+geth sharding --actor "notary" --deposit --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
 ```
 
-This will extract 1000ETH from your account balance and insert you into the SMC's notaries. Then, the program will listen for incoming block headers and notify you when you have been selected as to vote on proposals for a certain shard in a given period. Once you are selected, your sharding client will download collation information to check for data availability on vote on proposals that have been submitted via the `addHeader` function on the SMC.
+This will extract 1000ETH from your account balance and insert you into the SMC's notaries. Then, the program will listen for incoming block headers and notify you when you have been selected as to vote on proposals for a certain shard in a given period. Once you are selected, your sharding node will download collation information to check for data availability on vote on proposals that have been submitted via the `addHeader` function on the SMC.
 
-Concurrently, you will need to run another client that is tasked with processing transactions into collations and submitting them to the SMC via the `addHeader` function. 
+Concurrently, you will need to run another service that is tasked with processing transactions into collations and submitting them to the SMC via the `addHeader` function. 
 
-## Running a Collation Proposal Client
+## Running a Collation Proposal Node
 
 ```
-geth sharding-proposer --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
+geth sharding --actor "proposer" --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
 ```
 
-This client is tasked with processing pending transactions into blobs within collations by serializing data into collation bodies. It is responsible for submitting proposals (collation headers) to the SMC via the `addHeader` function.
+This node is tasked with processing pending transactions into blobs within collations by serializing data into collation bodies. It is responsible for submitting proposals (collation headers) to the SMC via the `addHeader` function.
+
+## Running an Observer Node
+
+    geth sharding --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
+
+Omitting the `--actor` flag will launch a simple observer service attached to the sharding client that is able to listen to changes happening throughout the sharded Ethereum network.
 
 # Making Changes
 
 ## Rebuilding the Sharding Manager Contract Bindings
 
-The Sharding Manager Contract is built in Solidity and deployed to the geth node upon launch of the client if it does not exist in the network at a specified address. If there are any changes to the SMC's code, the Golang bindigs must be rebuilt with the following command.
+The Sharding Manager Contract is built in Solidity and deployed to a running geth node upon launch of the sharding node if it does not exist in the network at a specified address. If there are any changes to the SMC's code, the Golang bindigs must be rebuilt with the following command.
 
     go generate github.com/ethereum/go-ethereum/sharding
     # OR

cmd/geth/main.go

@@ -119,6 +119,7 @@ var (
 		utils.GpoPercentileFlag,
 		utils.ExtraDataFlag,
 		utils.DepositFlag,
+		utils.ActorFlag,
 		configFileFlag,
 	}
 
@@ -168,8 +169,7 @@ func init() {
 		attachCommand,
 		javascriptCommand,
 		// See shardingcmd.go:
-		notaryClientCommand,
-		proposerClientCommand,
+		shardingCommand,
 		// See misccmd.go:
 		makecacheCommand,
 		makedagCommand,

cmd/geth/shardingcmd.go

@@ -1,46 +1,69 @@
 package main
 
 import (
-	"github.com/ethereum/go-ethereum/sharding/notary"
-	"github.com/ethereum/go-ethereum/sharding/proposer"
+	"fmt"
 
 	"github.com/ethereum/go-ethereum/cmd/utils"
+	"github.com/ethereum/go-ethereum/sharding"
+	"github.com/ethereum/go-ethereum/sharding/node"
+	"github.com/ethereum/go-ethereum/sharding/notary"
+	"github.com/ethereum/go-ethereum/sharding/observer"
+	"github.com/ethereum/go-ethereum/sharding/proposer"
 	cli "gopkg.in/urfave/cli.v1"
 )
 
 var (
-	notaryClientCommand = cli.Command{
-		Action:    utils.MigrateFlags(notaryClient),
-		Name:      "sharding-notary",
-		Aliases:   []string{"shard-notary"},
-		Usage:     "Start a sharding notary client",
+	shardingCommand = cli.Command{
+		Action:    utils.MigrateFlags(shardingCmd),
+		Name:      "sharding",
+		Usage:     "Start a sharding-enabled node",
 		ArgsUsage: "[endpoint]",
-		Flags:     []cli.Flag{utils.DataDirFlag, utils.PasswordFileFlag, utils.NetworkIdFlag, utils.IPCPathFlag, utils.DepositFlag},
+		Flags:     []cli.Flag{utils.ActorFlag, utils.DataDirFlag, utils.PasswordFileFlag, utils.NetworkIdFlag, utils.IPCPathFlag, utils.DepositFlag},
 		Category:  "SHARDING COMMANDS",
 		Description: `
-Launches a sharding notary client that connects to a running geth node and submit collations to a Sharding Manager Contract. This feature is a work in progress.
-`,
-	}
-	proposerClientCommand = cli.Command{
-		Action:    utils.MigrateFlags(proposerClient),
-		Name:      "sharding-proposer",
-		Aliases:   []string{"shard-proposer"},
-		Usage:     "Start a sharding proposer client",
-		ArgsUsage: "[endpoint]",
-		Flags:     []cli.Flag{utils.DataDirFlag, utils.PasswordFileFlag, utils.NetworkIdFlag, utils.IPCPathFlag},
-		Category:  "SHARDING COMMANDS",
-		Description: `
-Launches a sharding proposer client that connects to a running geth node and proposes collations to notary node. This feature is a work in progress.
+Launches a sharding node that manages services related to submitting collations to a Sharding Manager Contract, notary and proposer services, and shardp2p connections. This feature is a work in progress.
 `,
 	}
 )
 
-func notaryClient(ctx *cli.Context) error {
-	c := notary.NewNotary(ctx)
-	return c.Start()
+// shardingCmd is the main cmd line entry point for starting a sharding-enabled node.
+// A sharding node launches a suite of services including notary services,
+// proposer services, and a shardp2p protocol.
+func shardingCmd(ctx *cli.Context) error {
+	// configures a sharding-enabled node using the cli's context.
+	shardingNode, err := node.NewNode(ctx)
+	if err != nil {
+		return fmt.Errorf("could not initialize node instance: %v", err)
+	}
+	if err := registerShardingServices(shardingNode); err != nil {
+		return fmt.Errorf("could not start sharding node: %v", err)
+	}
+	defer shardingNode.Close()
+	// starts a connection to a geth node and kicks off every registered service.
+	return shardingNode.Start()
 }
 
-func proposerClient(ctx *cli.Context) error {
-	p := proposer.NewProposer(ctx)
-	return p.Start()
+// registerShardingServices sets up either a notary or proposer
+// sharding service dependent on the ClientType cli flag. We should be defining
+// the services we want to register here, as this is the geth command entry point
+// for sharding.
+func registerShardingServices(n node.Node) error {
+	actorFlag := n.Context().GlobalString(utils.ActorFlag.Name)
+
+	err := n.Register(func() (sharding.Service, error) {
+		if actorFlag == "notary" {
+			return notary.NewNotary(n)
+		} else if actorFlag == "proposer" {
+			return proposer.NewProposer(n)
+		}
+		return observer.NewObserver(n)
+	})
+
+	if err != nil {
+		return fmt.Errorf("failed to register the main sharding services: %v", err)
+	}
+
+	// TODO(prestonvanloon) registers the shardp2p service.
+	// we can do n.Register and initialize a shardp2p.NewServer() or something like that.
+	return nil
 }

cmd/utils/flags.go

@@ -533,11 +533,14 @@ var (
 		Usage: "Minimum POW accepted",
 		Value: whisper.DefaultMinimumPoW,
 	}
-
-	//Sharding Settings
+	// Sharding Settings
 	DepositFlag = cli.BoolFlag{
 		Name:  "deposit",
-		Usage: "To become a notary with your sharding client, " + new(big.Int).Div(sharding.NotaryDeposit, new(big.Int).Exp(big.NewInt(10), big.NewInt(18), nil)).String() + " ETH will be deposited into SMC",
+		Usage: "To become a notary in a sharding node, " + new(big.Int).Div(sharding.NotaryDeposit, new(big.Int).Exp(big.NewInt(10), big.NewInt(18), nil)).String() + " ETH will be deposited into SMC",
+	}
+	ActorFlag = cli.StringFlag{
+		Name:  "actor",
+		Usage: `use the --actor notary or --actor proposer to start a notary or proposer service in the sharding node. If omitted, the sharding node registers an Observer service that simply observes the activity in the sharded network`,
 	}
 )
 

sharding/README.md

@@ -119,9 +119,9 @@ To concretize these phases, we will be releasing our implementation of sharding
 
 Our current work is focused on creating a localized version of phase 1, quadratic sharding that would include the following:
 
--   A minimal, **sharding client** system that will interact with a **Sharding Manager Contract** on a locally running geth node
+-   A minimal, **sharding node** system that will interact with a **Sharding Manager Contract** on a locally running geth node
 -   Ability to deposit ETH into the SMC through the command line and to be selected as a notary by the local **SMC** in addition to the ability to withdraw the ETH staked
--   A **proposer client** that listens for pending tx’s, creates collations, and submits them to the SMC
+-   A **proposer** that listens for pending tx’s, creates collations, and submits them to the SMC
 -   Ability to inspect the shard states and visualize the working system locally
 
 
@@ -147,7 +147,7 @@ ETA: To Be determined
 
 # Go-Ethereum Sharding Alpha Implementation
 
-Prysmatic Labs will begin by focusing its implementation entirely on the **Ruby Release** from our roadmap. We plan on being as pragmatic as possible to create something that can be locally run by any developer as soon as possible. Our initial deliverable will center around a command line tool that will serve as an entrypoint into a sharding client that allows staking to become a notary, a proposer client that allows for the creation of collation proposals, shard  state local storage, and on-chain voting of collation headers via the Sharding Manager Contract.
+Prysmatic Labs will begin by focusing its implementation entirely on the **Ruby Release** from our roadmap. We plan on being as pragmatic as possible to create something that can be locally run by any developer as soon as possible. Our initial deliverable will center around a command line tool that will serve as an entrypoint into a sharding node that allows staking to become a notary, proposer, manages shard state local storage, and does on-chain voting of collation headers via the Sharding Manager Contract.
 
 Here is a full reference spec explaining how our initial system will function:
 
@@ -157,19 +157,19 @@ Our implementation revolves around 5 core components:
 
 -   A **locally-running geth node** that spins up an instance of the Ethereum blockchain and mines on the Proof of Work chain
 -   A **Sharding Manager Contract (SMC)** that is deployed onto this blockchain instance
--   A **sharding client** that connects to the running geth node through JSON-RPC, provides bindings to the SMC
--   A **notary client** that allows users to stake ETH into the SMC and be selected as a notary in a certain period on a shard
--   A **proposer client** that is tasked with processing pending tx's into collations that are then submitted to the SMC. In phase 1, proposers _do not_ execute state, but rather just serialize pending tx data into possibly valid/invalid data blobs.
+-   A **sharding node** that connects to the running geth node through JSON-RPC, provides bindings to the SMC
+-   A **notary service** that allows users to stake ETH into the SMC and be selected as a notary in a certain period on a shard
+-   A **proposer service** that is tasked with processing pending tx's into collations that are then submitted to the SMC. In phase 1, proposers _do not_ execute state, but rather just serialize pending tx data into possibly valid/invalid data blobs.
 
 Our initial implementation will function through simple command line arguments that will allow a user running the local geth node to deposit ETH into the SMC and join as a notary that is randomly assigned to a shard in a certain period.
 
 A basic, end-to-end example of the system is as follows:
 
-1.  _**User starts a notary client and deposits 1000ETH into the SMC:**_ the sharding client connects to a locally running geth node and asks the user to confirm a deposit from his/her personal account.
+1.  _**User starts a sharding node and deposits 1000ETH into the SMC:**_ the sharding node connects to a locally running geth node and asks the user to confirm a deposit from his/her personal account.
 
 2.  _**Client connects & listens to incoming headers from the geth node and assigns user as notary on a shard per period:**_ The notary is selected in CURRENT_PERIOD + LOOKEAD_PERIOD (which is around a 5 minutes notice) and must download data for collation headers submitted in that time period.
 
-3.  _**Concurrently, a proposer client processes pending transaction data into blobs:**_ the proposer client will create collation bodies and submit their headers to the SMC. In Phase 1, it is important to note that we will _not_ have any state execution. Proposers will just serialize pending tx into fixed collation body sizes without executing them for state transition validity.
+3.  _**Concurrently, a proposer protocol processes pending transaction data into blobs:**_ the proposer client will create collation bodies and submit their headers to the SMC. In Phase 1, it is important to note that we will _not_ have any state execution. Proposers will just serialize pending tx into fixed collation body sizes without executing them for state transition validity.
 
 5.  _**The set of notaries vote on collation headers as canonical unitl the period ends:**_ the headers that received >= 2/3 votes are accepted as canonical.
 
@@ -181,17 +181,17 @@ Now, we’ll explore our architecture and implementation in detail as part of th
 
 Our Ruby Release requires users to start a local geth node running a localized, private blockchain to deploy the **SMC** into. Users can spin up a notary client as a command line entrypoint into geth while the node is running as follows:
 
-    geth sharding-notary --deposit --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
+    geth sharding --actor "notary" --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345 --deposit
 
-This will extract 1000ETH from the user's account balance and insert him/her into the SMC's notaries. Then, the program will listen for incoming block headers and notify the user when he/she has been selected as to vote on collations for a certain shard in a given period. Once you are selected, the sharding client will download collation information to check for data availability on vote on proposals that have been submitted via the `addHeader` function on the SMC.
+This will extract 1000ETH from the user's account balance and insert him/her into the SMC's notaries. Then, the program will listen for incoming block headers and notify the user when he/she has been selected as to vote on collations for a certain shard in a given period. Once you are selected, the sharding node will download collation information to check for data availability on vote on proposals that have been submitted via the `addHeader` function on the SMC.
 
 Users can also run a proposer client that is tasked with processing transactions into collations and submitting them to the SMC via the `addHeader` function. 
 
-    geth sharding-proposer --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
+    geth sharding --actor "proposer" --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
 
 This client is tasked with processing pending transactions into blobs within collations by serializing data into collation bodies. It is responsible for submitting proposals (collation headers) to the SMC via the `addHeader` function.
 
-The sharding client begins to work by its main loop, which involves the following steps:
+The sharding node begins to work by its main loop, which involves the following steps:
 
 1.  _**Subscribe to incoming block headers:**_ the client will begin by issuing a subscription over JSON-RPC for block headers from the running geth node.