Skip to content

Commit

Permalink
Rename slave -> worker
Browse files Browse the repository at this point in the history
Signed-off-by: Thane Thomson <connect@thanethomson.com>
  • Loading branch information
thanethomson committed Nov 1, 2021
1 parent d8fb3d5 commit fa3b67e
Show file tree
Hide file tree
Showing 15 changed files with 894 additions and 884 deletions.
88 changes: 49 additions & 39 deletions README.md
Expand Up @@ -14,20 +14,24 @@ not strictly adhered to prior to a v1.0 release, so breaking API changes can
emerge with minor version releases.**

## Requirements

`tm-load-test` is currently tested using Go v1.17.

## Building

To build the `tm-load-test` binary in the `build` directory:

```bash
make
```

## Usage

`tm-load-test` can be executed in one of two modes: **standalone**, or
**master/slave**.
**master/worker**.

### Standalone Mode

In standalone mode, `tm-load-test` operates in a similar way to `tm-bench`:

```bash
Expand All @@ -42,64 +46,67 @@ To see a description of what all of the parameters mean, simply run:
tm-load-test --help
```

### Master/Slave Mode
In master/slave mode, which is best used for large-scale, distributed load
testing, `tm-load-test` allows you to have multiple slave machines connect to
a single master to obtain their configuration and coordinate their operation.
### Master/Worker Mode

In master/worker mode, which is best used for large-scale, distributed load
testing, `tm-load-test` allows you to have multiple worker machines connect to a
single master to obtain their configuration and coordinate their operation.

The master acts as a simple WebSockets host, and the slaves are WebSockets
The master acts as a simple WebSockets host, and the workers are WebSockets
clients.

On the master machine:

```bash
# Run tm-load-test with similar parameters to the standalone mode, but now
# specifying the number of slaves to expect (--expect-slaves) and the host:port
# to which to bind (--bind) and listen for incoming slave requests.
# Run tm-load-test with similar parameters to the standalone mode, but now
# specifying the number of workers to expect (--expect-workers) and the host:port
# to which to bind (--bind) and listen for incoming worker requests.
tm-load-test \
master \
--expect-slaves 2 \
--expect-workers 2 \
--bind localhost:26670 \
-c 1 -T 10 -r 1000 -s 250 \
--broadcast-tx-method async \
--endpoints ws://tm-endpoint1.somewhere.com:26657/websocket,ws://tm-endpoint2.somewhere.com:26657/websocket
```

On each slave machine:
On each worker machine:

```bash
# Just tell the slave where to find the master - it will figure out the rest.
tm-load-test slave --master localhost:26680
# Just tell the worker where to find the master - it will figure out the rest.
tm-load-test worker --master localhost:26680
```

For more help, see the command line parameters' descriptions:

```bash
tm-load-test master --help
tm-load-test slave --help
tm-load-test worker --help
```

### Endpoint Selection Strategies

As of v0.5.1, an endpoint selection strategy can now be given to `tm-load-test`
as a parameter (`--endpoint-select-method`) to control the way in which
as a parameter (`--endpoint-select-method`) to control the way in which
endpoints are selected for load testing. There are several options:

1. `supplied` (the default) - only use the supplied endpoints (via the
1. `supplied` (the default) - only use the supplied endpoints (via the
`--endpoints` parameter) to submit transactions.
2. `discovered` - only use endpoints discovered through the supplied endpoints
(by way of crawling the Tendermint peers' network info), but do not use any
of the supplied endpoints.
3. `any` - use both the supplied and discovered endpoints to perform load
3. `any` - use both the supplied and discovered endpoints to perform load
testing.

**NOTE**: These selection strategies only apply if, and only if, the
`--expect-peers` parameter is supplied and is non-zero. The default behaviour
if `--expect-peers` is not supplied is effectively the `supplied` endpoint
**NOTE**: These selection strategies only apply if, and only if, the
`--expect-peers` parameter is supplied and is non-zero. The default behaviour if
`--expect-peers` is not supplied is effectively the `supplied` endpoint
selection strategy.

### Minimum Peer Connectivity
As of v0.6.0, `tm-load-test` can now wait for a minimum level of P2P
connectivity before starting the load testing. By using the

As of v0.6.0, `tm-load-test` can now wait for a minimum level of P2P
connectivity before starting the load testing. By using the
`--min-peer-connectivity` command line switch, along with `--expect-peers`, one
can restrict this.

Expand All @@ -109,11 +116,13 @@ minimum address book size is. Once the minimum address book size reaches the
configured value, the load testing can begin.

### Customizing

To implement your own client type to load test your own Tendermint ABCI
application, see the [`loadtest` package docs here](./pkg/loadtest/README.md).

## Monitoring
As of v0.4.1, `tm-load-test` exposes a number of metrics when in master/slave

As of v0.4.1, `tm-load-test` exposes a number of metrics when in master/worker
mode, but only from the master's web server at the `/metrics` endpoint. So if
you bind your master node to `localhost:26670`, you should be able to get these
metrics from:
Expand All @@ -125,30 +134,31 @@ curl http://localhost:26670/metrics
The following kinds of metrics are made available here:

* Total number of transactions recorded from the master's perspective (across
all slaves)
* Total number of transactions sent by each slave
* The status of the master node, which is a gauge that indicates one of the
all workers)
* Total number of transactions sent by each worker
* The status of the master node, which is a gauge that indicates one of the
following codes:
* 0 = Master starting
* 1 = Master waiting for all peers to connect
* 2 = Master waiting for all slaves to connect
* 2 = Master waiting for all workers to connect
* 3 = Load test underway
* 4 = Master and/or one or more slave(s) failed
* 5 = All slaves completed load testing successfully
* The status of each slave node, which is also a gauge that indicates one of the
following codes:
* 0 = Slave connected
* 1 = Slave accepted
* 2 = Slave rejected
* 4 = Master and/or one or more worker(s) failed
* 5 = All workers completed load testing successfully
* The status of each worker node, which is also a gauge that indicates one of
the following codes:
* 0 = Worker connected
* 1 = Worker accepted
* 2 = Worker rejected
* 3 = Load testing underway
* 4 = Slave failed
* 5 = Slave completed load testing successfully
* Standard Prometheus-provided metrics about the garbage collector in
* 4 = Worker failed
* 5 = Worker completed load testing successfully
* Standard Prometheus-provided metrics about the garbage collector in
`tm-load-test`
* The ID of the load test currently underway (defaults to 0), set by way of the
`--load-test-id` flag on the master

## Aggregate Statistics

As of `tm-load-test` v0.7.0, one can now write simple aggregate statistics to
a CSV file once testing completes by specifying the `--stats-output` flag:

Expand All @@ -159,10 +169,10 @@ tm-load-test -c 1 -T 10 -r 1000 -s 250 \
--endpoints ws://tm-endpoint1.somewhere.com:26657/websocket,ws://tm-endpoint2.somewhere.com:26657/websocket \
--stats-output /path/to/save/stats.csv

# From the master in master/slave mode
# From the master in master/worker mode
tm-load-test \
master \
--expect-slaves 2 \
--expect-workers 2 \
--bind localhost:26670 \
-c 1 -T 10 -r 1000 -s 250 \
--broadcast-tx-method async \
Expand Down
14 changes: 7 additions & 7 deletions cmd/tm-load-test/main.go
Expand Up @@ -4,8 +4,8 @@ import (
"github.com/informalsystems/tm-load-test/pkg/loadtest"
)

const appLongDesc = `Load testing application for Tendermint with optional master/slave mode.
Generates large quantities of arbitrary transactions and submits those
const appLongDesc = `Load testing application for Tendermint with optional master/worker mode.
Generates large quantities of arbitrary transactions and submits those
transactions to one or more Tendermint endpoints. By default, it assumes that
you are running the kvstore ABCI application on your Tendermint network.
Expand All @@ -17,24 +17,24 @@ To run the application in a similar fashion to tm-bench (STANDALONE mode):
To run the application in MASTER mode:
tm-load-test \
master \
--expect-slaves 2 \
--expect-workers 2 \
--bind localhost:26670 \
--shutdown-wait 60 \
-c 1 -T 10 -r 1000 -s 250 \
--broadcast-tx-method async \
--endpoints ws://tm-endpoint1.somewhere.com:26657/websocket,ws://tm-endpoint2.somewhere.com:26657/websocket
To run the application in SLAVE mode:
tm-load-test slave --master localhost:26680
tm-load-test worker --master localhost:26680
NOTES:
* MASTER mode exposes a "/metrics" endpoint in Prometheus plain text format
which shows total number of transactions and the status for the master and
all connected slaves.
* The "--shutdown-wait" flag in MASTER mode is specifically to allow your
all connected workers.
* The "--shutdown-wait" flag in MASTER mode is specifically to allow your
monitoring system some time to obtain the final Prometheus metrics from the
metrics endpoint.
* In SLAVE mode, all load testing-related flags are ignored. The slave always
* In SLAVE mode, all load testing-related flags are ignored. The worker always
takes instructions from the master node it's connected to.
`

Expand Down
2 changes: 1 addition & 1 deletion pkg/loadtest/README.md
Expand Up @@ -83,7 +83,7 @@ func main() {
panic(err)
}
// The loadtest.Run method will handle CLI argument parsing, errors,
// configuration, instantiating the load test and/or master/slave
// configuration, instantiating the load test and/or master/worker
// operations, etc. All it needs is to know which client factory to use for
// its load testing.
loadtest.Run(&loadtest.CLIConfig{
Expand Down
32 changes: 16 additions & 16 deletions pkg/loadtest/cli.go
Expand Up @@ -59,7 +59,7 @@ func buildCLI(cli *CLIConfig, logger logging.Logger) *cobra.Command {
rootCmd.PersistentFlags().StringVar(&cfg.BroadcastTxMethod, "broadcast-tx-method", "async", "The broadcast_tx method to use when submitting transactions - can be async, sync or commit")
rootCmd.PersistentFlags().StringSliceVar(&cfg.Endpoints, "endpoints", []string{}, "A comma-separated list of URLs indicating Tendermint WebSockets RPC endpoints to which to connect")
rootCmd.PersistentFlags().StringVar(&cfg.EndpointSelectMethod, "endpoint-select-method", SelectSuppliedEndpoints, "The method by which to select endpoints")
rootCmd.PersistentFlags().IntVar(&cfg.ExpectPeers, "expect-peers", 0, "The minimum number of peers to expect when crawling the P2P network from the specified endpoint(s) prior to waiting for slaves to connect")
rootCmd.PersistentFlags().IntVar(&cfg.ExpectPeers, "expect-peers", 0, "The minimum number of peers to expect when crawling the P2P network from the specified endpoint(s) prior to waiting for workers to connect")
rootCmd.PersistentFlags().IntVar(&cfg.MaxEndpoints, "max-endpoints", 0, "The maximum number of endpoints to use for testing, where 0 means unlimited")
rootCmd.PersistentFlags().IntVar(&cfg.PeerConnectTimeout, "peer-connect-timeout", 600, "The number of seconds to wait for all required peers to connect if expect-peers > 0")
rootCmd.PersistentFlags().IntVar(&cfg.MinConnectivity, "min-peer-connectivity", 0, "The minimum number of peers to which each peer must be connected before starting the load test")
Expand Down Expand Up @@ -87,35 +87,35 @@ func buildCLI(cli *CLIConfig, logger logging.Logger) *cobra.Command {
}
},
}
masterCmd.PersistentFlags().StringVar(&masterCfg.BindAddr, "bind", "localhost:26670", "A host:port combination to which to bind the master on which to listen for slave connections")
masterCmd.PersistentFlags().IntVar(&masterCfg.ExpectSlaves, "expect-slaves", 2, "The number of slaves to expect to connect to the master before starting load testing")
masterCmd.PersistentFlags().IntVar(&masterCfg.SlaveConnectTimeout, "connect-timeout", 180, "The maximum number of seconds to wait for all slaves to connect")
masterCmd.PersistentFlags().StringVar(&masterCfg.BindAddr, "bind", "localhost:26670", "A host:port combination to which to bind the master on which to listen for worker connections")
masterCmd.PersistentFlags().IntVar(&masterCfg.ExpectWorkers, "expect-workers", 2, "The number of workers to expect to connect to the master before starting load testing")
masterCmd.PersistentFlags().IntVar(&masterCfg.WorkerConnectTimeout, "connect-timeout", 180, "The maximum number of seconds to wait for all workers to connect")
masterCmd.PersistentFlags().IntVar(&masterCfg.ShutdownWait, "shutdown-wait", 0, "The number of seconds to wait after testing completes prior to shutting down the web server")
masterCmd.PersistentFlags().IntVar(&masterCfg.LoadTestID, "load-test-id", 0, "The ID of the load test currently underway")

var slaveCfg SlaveConfig
slaveCmd := &cobra.Command{
Use: "slave",
var workerCfg WorkerConfig
workerCmd := &cobra.Command{
Use: "worker",
Short: "Start load test application in SLAVE mode",
Run: func(cmd *cobra.Command, args []string) {
logger.Debug(fmt.Sprintf("Slave configuration: %s", slaveCfg.ToJSON()))
if err := slaveCfg.Validate(); err != nil {
logger.Debug(fmt.Sprintf("Worker configuration: %s", workerCfg.ToJSON()))
if err := workerCfg.Validate(); err != nil {
logger.Error(err.Error())
os.Exit(1)
}
slave, err := NewSlave(&slaveCfg)
worker, err := NewWorker(&workerCfg)
if err != nil {
logger.Error("Failed to create new slave", "err", err)
logger.Error("Failed to create new worker", "err", err)
os.Exit(1)
}
if err := slave.Run(); err != nil {
if err := worker.Run(); err != nil {
os.Exit(1)
}
},
}
slaveCmd.PersistentFlags().StringVar(&slaveCfg.ID, "id", "", "An optional unique ID for this slave. Will show up in metrics and logs. If not specified, a UUID will be generated.")
slaveCmd.PersistentFlags().StringVar(&slaveCfg.MasterAddr, "master", "ws://localhost:26670", "The WebSockets URL on which to find the master node")
slaveCmd.PersistentFlags().IntVar(&slaveCfg.MasterConnectTimeout, "connect-timeout", 180, "The maximum number of seconds to keep trying to connect to the master")
workerCmd.PersistentFlags().StringVar(&workerCfg.ID, "id", "", "An optional unique ID for this worker. Will show up in metrics and logs. If not specified, a UUID will be generated.")
workerCmd.PersistentFlags().StringVar(&workerCfg.MasterAddr, "master", "ws://localhost:26670", "The WebSockets URL on which to find the master node")
workerCmd.PersistentFlags().IntVar(&workerCfg.MasterConnectTimeout, "connect-timeout", 180, "The maximum number of seconds to keep trying to connect to the master")

versionCmd := &cobra.Command{
Use: "version",
Expand All @@ -130,7 +130,7 @@ func buildCLI(cli *CLIConfig, logger logging.Logger) *cobra.Command {
}

rootCmd.AddCommand(masterCmd)
rootCmd.AddCommand(slaveCmd)
rootCmd.AddCommand(workerCmd)
rootCmd.AddCommand(versionCmd)
return rootCmd
}
Expand Down
32 changes: 16 additions & 16 deletions pkg/loadtest/config.go
Expand Up @@ -18,7 +18,7 @@ var validEndpointSelectMethods = map[string]interface{}{
}

// Config represents the configuration for a single client (i.e. standalone or
// slave).
// worker).
type Config struct {
ClientFactory string `json:"client_factory"` // Which client factory should we use for load testing?
Connections int `json:"connections"` // The number of WebSockets connections to make to each target endpoint.
Expand All @@ -40,16 +40,16 @@ type Config struct {

// MasterConfig is the configuration options specific to a master node.
type MasterConfig struct {
BindAddr string `json:"bind_addr"` // The "host:port" to which to bind the master node to listen for incoming slaves.
ExpectSlaves int `json:"expect_slaves"` // The number of slaves to expect before starting the load test.
SlaveConnectTimeout int `json:"connect_timeout"` // The number of seconds to wait for all slaves to connect.
ShutdownWait int `json:"shutdown_wait"` // The number of seconds to wait at shutdown (while keeping the HTTP server running - primarily to allow Prometheus to keep polling).
LoadTestID int `json:"load_test_id"` // An integer greater than 0 that will be exposed via a Prometheus gauge while the load test is underway.
BindAddr string `json:"bind_addr"` // The "host:port" to which to bind the master node to listen for incoming workers.
ExpectWorkers int `json:"expect_workers"` // The number of workers to expect before starting the load test.
WorkerConnectTimeout int `json:"connect_timeout"` // The number of seconds to wait for all workers to connect.
ShutdownWait int `json:"shutdown_wait"` // The number of seconds to wait at shutdown (while keeping the HTTP server running - primarily to allow Prometheus to keep polling).
LoadTestID int `json:"load_test_id"` // An integer greater than 0 that will be exposed via a Prometheus gauge while the load test is underway.
}

// SlaveConfig is the configuration options specific to a slave node.
type SlaveConfig struct {
ID string `json:"id"` // A unique ID for this slave instance. Will show up in the metrics reported by the master for this slave.
// WorkerConfig is the configuration options specific to a worker node.
type WorkerConfig struct {
ID string `json:"id"` // A unique ID for this worker instance. Will show up in the metrics reported by the master for this worker.
MasterAddr string `json:"master_addr"` // The address at which to find the master node.
MasterConnectTimeout int `json:"connect_timeout"` // The maximum amount of time, in seconds, to allow for the master to become available.
}
Expand Down Expand Up @@ -132,10 +132,10 @@ func (c MasterConfig) Validate() error {
if len(c.BindAddr) == 0 {
return fmt.Errorf("master bind address must be specified")
}
if c.ExpectSlaves < 1 {
return fmt.Errorf("master expect-slaves must be at least 1, but got %d", c.ExpectSlaves)
if c.ExpectWorkers < 1 {
return fmt.Errorf("master expect-workers must be at least 1, but got %d", c.ExpectWorkers)
}
if c.SlaveConnectTimeout < 1 {
if c.WorkerConnectTimeout < 1 {
return fmt.Errorf("master connect-timeout must be at least 1 second")
}
if c.LoadTestID < 0 {
Expand All @@ -152,9 +152,9 @@ func (c Config) ToJSON() string {
return string(b)
}

func (c SlaveConfig) Validate() error {
if len(c.ID) > 0 && !isValidSlaveID(c.ID) {
return fmt.Errorf("Invalid slave ID \"%s\": slave IDs can only be lowercase alphanumeric characters", c.ID)
func (c WorkerConfig) Validate() error {
if len(c.ID) > 0 && !isValidWorkerID(c.ID) {
return fmt.Errorf("Invalid worker ID \"%s\": worker IDs can only be lowercase alphanumeric characters", c.ID)
}
if len(c.MasterAddr) == 0 {
return fmt.Errorf("master address must be specified")
Expand All @@ -165,7 +165,7 @@ func (c SlaveConfig) Validate() error {
return nil
}

func (c SlaveConfig) ToJSON() string {
func (c WorkerConfig) ToJSON() string {
b, err := json.Marshal(c)
if err != nil {
return fmt.Sprintf("%v", c)
Expand Down

0 comments on commit fa3b67e

Please sign in to comment.