tools: search/replace indexer/conduit in block-generator

tools/block-generator/README.md

@@ -1,6 +1,6 @@
 # Block Generator
 
-This tool is used for testing Indexer import performance. It does this by generating synthetic blocks which are sent by mocking the Algod REST API endpoints that Indexer uses.
+This tool is used for testing Conduit import performance. It does this by generating synthetic blocks which are sent by mocking the Algod REST API endpoints that Conduit uses.
 
 ## Scenario Configuration
 
@@ -9,11 +9,11 @@ Block generator uses a YAML config file to describe the composition of each rand
 2. Transaction type distribution
 3. Transaction type specific configuration
 
-At the time of writing, the block generator supports **payment** and **asset** transactions. The settings are hopefully, more or less, obvious. Distributions are specified as fractions of one, and the sum of all options must add up to one.
+At the time of writing, the block generator supports **payment** and **asset** transactions. The settings are hopefully, more or less, obvious. Distributions are specified as fractions of 1.0, and the sum of all options must add up to 1.0.
 
-Here is an example which uses all of the current options. Notice that the synthetic blocks are not required to follow algod limits, in this case the block size is specified as 19999, or four times larger than the current block size limit:
+Here is an example which uses all of the current options. Notice that the synthetic blocks are not required to follow algod limits, in this case the block size is specified as 19999:
 ```
-name: "Mixed (jumbo)"
+name: "Mixed (19,999)"
 genesis_accounts: 10000
 genesis_account_balance: 1000000000000
 tx_per_block: 19999
@@ -36,13 +36,15 @@ asset_delete_fraction: 0
 
 ## Modes
 
-The block generator can run in one of two modes, a standalone **daemon**, or a test suite **runner**
+The block generator can run in one of two _modes_:
+1. standalone **daemon**
+2. test suite **runner**
 
 ### daemon
 
-In standalone mode, a block-generator process starts and exposes the mock algod endpoints for **/genesis** and **/v2/blocks/{block}**. If you choose to query them manually, it only supports fetching blocks sequentially. This is due to the fact that it generates a pseudorandom stream of transactions and after each random transaction the state increments to the next.
+In standalone daemon mode, a block-generator process starts and exposes the mock algod endpoints for **/genesis** and **/v2/blocks/{block}**. If you choose to query them manually, it only supports fetching blocks sequentially. This is due to the fact that it generates a pseudorandom stream of transactions and after each random transaction the state increments to the next.
 
-Here is the help output: 
+Here is the help output for **daemon**: 
 ```bash
 ~$ ./block-generator daemon -h
 Start the generator daemon in standalone mode.
@@ -58,58 +60,89 @@ Flags:
 
 ### runner
 
-For our usage, we want to run the same set of tests consistently across many scenarios and with many different releases. The runner mode automates this process by starting the **daemon** with many different configurations, managing a postgres database, and running a separate indexer process configured to use them.
+The runner mode is well suited for runing the same set of tests consistently across many scenarios and for different releases. The runner mode automates this process by starting the **daemon** with many different configurations, managing a postgres database, and running a separate Conduit process configured to use them.
 
-The results of the testing are written to the directory specified by the **--report-directory** option, and include many different metrics. In addition to the report, the indexer log is written to this directory. The files are named according to the scenario file, and end in "report" or "log".
+The results of the testing are written to the directory specified by the **--report-directory** option, and include many different metrics. In addition to the report, the Conduit log is written to this directory. The files are named according to the scenario file, and end in "report" or "log".
 
 Here is an example report from running with a test duration of "1h":
 ```
-test_duration_seconds:3600
-test_duration_actual_seconds:3600.056457
-transaction_pay_total:30024226
-transaction_pay_create_total:614242
-early_average_import_time_sec:2.13
-early_cumulative_import_time_sec:1083.26
-early_average_imported_tx_per_block:19999.00
-early_cumulative_imported_tx_per_block:10179491
-early_average_block_upload_time_sec:NaN
-early_cumulative_block_upload_time_sec:0.00
-early_average_postgres_eval_time_sec:0.33
-early_cumulative_postgres_eval_time_sec:167.41
-early_imported_round:509
-early_overall_transactions_per_second:9397.09
-early_uptime_seconds:3600.06
-final_average_import_time_sec:2.35
-final_cumulative_import_time_sec:3602.62
-final_average_imported_tx_per_block:19999.00
-final_cumulative_imported_tx_per_block:30598470
-final_average_block_upload_time_sec:NaN
-final_cumulative_block_upload_time_sec:0.00
-final_average_postgres_eval_time_sec:0.33
-final_cumulative_postgres_eval_time_sec:507.38
-final_imported_round:1530
-final_overall_transactions_per_second:8493.40
-final_uptime_seconds:3600.06
+test_duration_seconds:30
+test_duration_actual_seconds:30.018076
+transaction_asset_close_total:472
+transaction_asset_create_total:711
+transaction_asset_optin_total:1230
+transaction_asset_xfer_total:468
+transaction_pay_total:1457
+transaction_pay_create_total:1472
+early_average_import_time_sec:0.05
+early_cumulative_import_time_sec:11.05
+early_average_imported_tx_per_block:10.00
+early_cumulative_imported_tx_per_block:2390
+early_imported_round:239
+early_overall_transactions_per_second:216.26
+early_uptime_seconds:30.02
+final_average_import_time_sec:0.05
+final_cumulative_import_time_sec:31.36
+final_average_imported_tx_per_block:10.00
+final_cumulative_imported_tx_per_block:5800
+final_imported_round:580
+final_overall_transactions_per_second:184.93
+final_uptime_seconds:30.02
 ```
 
-Here is the help output:
+Here is the help output for **runner**:
 ```bash
 ~$ ./block-generator runner -h
-Run test suite and collect results.
+Run an automated test suite using the block-generator daemon and a provided conduit binary. Results are captured to a specified output directory.
 
 Usage:
   block-generator runner [flags]
 
 Flags:
-      --cpuprofile string                   Path where Indexer writes its CPU profile.
+  -i, --conduit-binary string               Path to conduit binary.
+      --cpuprofile string                   Path where conduit writes its CPU profile.
   -h, --help                                help for runner
-  -i, --indexer-binary string               Path to indexer binary.
-  -p, --indexer-port uint                   Port to start the server at. This is useful if you have a prometheus server for collecting additional data. (default 4010)
-  -l, --log-level string                    LogLevel to use when starting Indexer. [error, warn, info, debug, trace] (default "error")
+  -k, --keep-data-dir                       If set the validator will not delete the data directory after tests complete.
+  -l, --log-level string                    LogLevel to use when starting conduit. [panic, fatal, error, warn, info, debug, trace] (default "error")
+  -p, --metrics-port uint                   Port to start the metrics server at. (default 9999)
   -c, --postgres-connection-string string   Postgres connection string.
   -r, --report-directory string             Location to place test reports.
       --reset                               If set any existing report directory will be deleted before running tests.
   -s, --scenario string                     Directory containing scenarios, or specific scenario file.
   -d, --test-duration duration              Duration to use for each scenario. (default 5m0s)
       --validate                            If set the validator will run after test-duration has elapsed to verify data is correct. An extra line in each report indicates validator success or failure.
 ```
+
+## Example Scenario Run using Conduit and Postgres - `run_runner.sh`
+
+A typical **runner** scenario involves:
+* a [scenario configuration](#scenario-configuration) file, e.g. [test_config.yml](./test_config.yml)
+* access to a `conduit` binary to query the block generator's mock Algod endpoint and ingest the synthetic blocks
+* a datastore -such as a postgres database- to collect `conduit`'s output
+* a `conduit` config file to define its import/export behavior
+
+`run_runner.sh` makes the following choices for the previous bullet points:
+* it can accept any scenario as its second argument, but defaults to [test_config.yml](./test_config.yml) when this isn't provided (this is a scenario with a lifetime of ~30 seconds)
+* knows how to import through a mock Algod running on port 11112 (which is the port the runner avails)
+* sets up a dockerized postgres database to receive conduit's output
+* configures `conduit` for these specs using [this config template](./runner/template/conduit.yml.tmpl)
+
+### Sample Run
+
+First you'll need to get a `conduit` binary. For example you can follow the [developer portal's instructions](https://developer.algorand.org/docs/get-details/conduit/GettingStarted/#installation) or run `go build .` inside of the directory `cmd/conduit` after downloading the `conduit` repo.
+
+Assume you've navigated to the `tools/block-generator` directory of 
+the `go-algorand` repo, and:
+* saved the conduit binary to `tools/block-generator/conduit`
+* created a block generator scenario config at `tools/block-generator/scenario.yml`
+
+Then you can execute the following command to run the scenario:
+```sh
+./run_runner.sh ./conduit scenario.yml 
+```
+
+### Scenario Report
+
+If all goes well, the run will generate a directory `tools/block-generator/OUTPUT_RUN_RUNNER_TEST` and in that directory you can see the statistics 
+of the run in `scenario.report`.
+

tools/block-generator/run_tests.sh

@@ -12,11 +12,11 @@ help() {
   echo " -v|--verbose    enable verbose script output."
   echo " -c|--connection-string"
   echo "                 PostgreSQL connection string."
-  echo " -i|--indexer    path to indexer binary."
-  echo " -s|--scenarios  path to indexer test scenarios."
+  echo " -i|--conduit    path to conduit binary."
+  echo " -s|--scenarios  path to conduit test scenarios."
   echo " -r|--report-dir directory where the report should be written."
   echo " -d|--duration   test duration."
-  echo " -l|--level      log level to pass to Indexer."
+  echo " -l|--level      log level to pass to conduit."
   echo " -g|--generator  block-generator binary to run the generator."
   exit
 }
@@ -33,7 +33,7 @@ while :; do
     GENERATOR_BINARY="${2-}"
     shift
     ;;
-  -i | --indexer)
+  -i | --conduit)
     CONDUIT_BINARY="${2-}"
     shift
     ;;
@@ -67,12 +67,12 @@ if [ -z "$CONNECTION_STRING" ]; then
 fi
 
 if [ -z "$CONDUIT_BINARY" ]; then
-  echo "Missing required indexer binary parameter (-i / --indexer)."
+  echo "Missing required conduit binary parameter (-i / --conduit)."
   exit 1
 fi
 
 if [ -z "$SCENARIOS" ]; then
-  echo "Missing required indexer test scenario parameter (-s / --scenarios)."
+  echo "Missing required conduit test scenario parameter (-s / --scenarios)."
   exit 1
 fi
 

tools/block-generator/runner/run.go

@@ -19,6 +19,7 @@ package runner
 import (
 	"bytes"
 	"context"
+
 	// embed conduit template config file
 	_ "embed"
 	"encoding/json"
@@ -101,7 +102,7 @@ func (r *Args) run() error {
 	baseName := filepath.Base(r.Path)
 	baseNameNoExt := strings.TrimSuffix(baseName, filepath.Ext(baseName))
 	reportfile := path.Join(r.ReportDirectory, fmt.Sprintf("%s.report", baseNameNoExt))
-	logfile := path.Join(r.ReportDirectory, fmt.Sprintf("%s.indexer-log", baseNameNoExt))
+	logfile := path.Join(r.ReportDirectory, fmt.Sprintf("%s.conduit-log", baseNameNoExt))
 	dataDir := path.Join(r.ReportDirectory, fmt.Sprintf("%s_data", baseNameNoExt))
 	// create the data directory.
 	if err := os.Mkdir(dataDir, os.ModeDir|os.ModePerm); err != nil {
@@ -127,7 +128,7 @@ func (r *Args) run() error {
 	defer func() {
 		// Shutdown generator.
 		if err := generatorShutdownFunc(); err != nil {
-			fmt.Printf("Failed to shutdown generator: %s\n", err)
+			fmt.Printf("failed to shutdown generator: %s\n", err)
 		}
 	}()
 
@@ -140,7 +141,7 @@ func (r *Args) run() error {
 	// create config file in the right data directory
 	f, err := os.Create(path.Join(dataDir, "conduit.yml"))
 	if err != nil {
-		return fmt.Errorf("creating conduit.yml: %v", err)
+		return fmt.Errorf("problem creating conduit.yml: %v", err)
 	}
 	defer f.Close()
 
@@ -151,7 +152,7 @@ func (r *Args) run() error {
 
 	err = t.Execute(f, conduitConfig)
 	if err != nil {
-		return fmt.Errorf("execute template file: %v", err)
+		return fmt.Errorf("problem executing template file: %v", err)
 	}
 
 	// Start conduit
@@ -162,7 +163,7 @@ func (r *Args) run() error {
 	defer func() {
 		// Shutdown conduit
 		if err := conduitShutdownFunc(); err != nil {
-			fmt.Printf("Failed to shutdown Conduit: %s\n", err)
+			fmt.Printf("failed to shutdown Conduit: %s\n", err)
 		}
 	}()
 
@@ -408,7 +409,7 @@ func startGenerator(configFile string, addr string, blockMiddleware func(http.Ha
 	}, generator
 }
 
-// startConduit starts the indexer.
+// startConduit starts the conduit binary.
 func startConduit(dataDir string, conduitBinary string) (func() error, error) {
 	cmd := exec.Command(
 		conduitBinary,
@@ -427,9 +428,9 @@ func startConduit(dataDir string, conduitBinary string) (func() error, error) {
 
 	return func() error {
 		if err := cmd.Process.Signal(os.Interrupt); err != nil {
-			fmt.Printf("failed to kill indexer process: %s\n", err)
+			fmt.Printf("failed to kill conduit process: %s\n", err)
 			if err := cmd.Process.Kill(); err != nil {
-				return fmt.Errorf("failed to kill indexer process: %w", err)
+				return fmt.Errorf("failed to kill conduit process: %w", err)
 			}
 		}
 		if err := cmd.Wait(); err != nil {

tools/block-generator/runner/runner.go

@@ -48,14 +48,14 @@ func init() {
 	RunnerCmd.Flags().StringVarP(&runnerArgs.PostgresConnectionString, "postgres-connection-string", "c", "", "Postgres connection string.")
 	RunnerCmd.Flags().DurationVarP(&runnerArgs.RunDuration, "test-duration", "d", 5*time.Minute, "Duration to use for each scenario.")
 	RunnerCmd.Flags().StringVarP(&runnerArgs.ReportDirectory, "report-directory", "r", "", "Location to place test reports.")
-	RunnerCmd.Flags().StringVarP(&runnerArgs.LogLevel, "log-level", "l", "error", "LogLevel to use when starting Indexer. [panic, fatal, error, warn, info, debug, trace]")
-	RunnerCmd.Flags().StringVarP(&runnerArgs.CPUProfilePath, "cpuprofile", "", "", "Path where Indexer writes its CPU profile.")
+	RunnerCmd.Flags().StringVarP(&runnerArgs.LogLevel, "log-level", "l", "error", "LogLevel to use when starting Conduit. [panic, fatal, error, warn, info, debug, trace]")
+	RunnerCmd.Flags().StringVarP(&runnerArgs.CPUProfilePath, "cpuprofile", "", "", "Path where Conduit writes its CPU profile.")
 	RunnerCmd.Flags().BoolVarP(&runnerArgs.ResetReportDir, "reset", "", false, "If set any existing report directory will be deleted before running tests.")
 	RunnerCmd.Flags().BoolVarP(&runnerArgs.RunValidation, "validate", "", false, "If set the validator will run after test-duration has elapsed to verify data is correct. An extra line in each report indicates validator success or failure.")
 	RunnerCmd.Flags().BoolVarP(&runnerArgs.KeepDataDir, "keep-data-dir", "k", false, "If set the validator will not delete the data directory after tests complete.")
 
 	RunnerCmd.MarkFlagRequired("scenario")
-	RunnerCmd.MarkFlagRequired("indexer-binary")
+	RunnerCmd.MarkFlagRequired("conduit-binary")
 	RunnerCmd.MarkFlagRequired("postgres-connection-string")
 	RunnerCmd.MarkFlagRequired("report-directory")
 }