IGNITE-4212 Ignite Benchmarking Simplification and Automation / IGNIT…

…E-4478 Fixing documentation and resolving lack of usability

(cherry picked from commit 1265793)

modules/yardstick/DEVNOTES-standalone.txt

@@ -11,5 +11,6 @@ Artifacts can be found in `/target/assembly directory.`
 Writing Ignite Benchmarks
 =========================
 
-All benchmarks extend `AbstractBenchmark` class. A new benchmark should also extend this abstract class
-and implement `test` method. This is the method which actually tests performance.
+All benchmarks extend `AbstractBenchmark` class. A new benchmark should also extend
+this abstract class and implement `test` method. This method is the entry point for
+a benchmark execution.

modules/yardstick/README.txt

@@ -1,61 +1,73 @@
 Yardstick Ignite Benchmarks
 ===========================
-Apache Ignite benchmarks are written on top of Yardstick Framework (https://github.com/gridgain/yardstick) and allow
-you to measure performance of various Apache Ignite components and modules.
+Apache Ignite benchmarks are written on top of Yardstick Framework (https://github.com/gridgain/yardstick)
+and allow you to measure performance of various Apache Ignite components and modules.
 
 The documentation below describes how to execute and configure already assembled benchmarks. If you need to add new
 benchmarks or build existing one then refer to instruction from `DEVNOTES.txt` in source directory.
 
-Visit Yardstick Repository (https://github.com/gridgain/yardstick) for detailed information in regards resulting graphs
-generation and how the frameworks works.
+Visit Yardstick Repository (https://github.com/gridgain/yardstick) for detailed information in regards resulting
+graphs generation and how the frameworks works.
 
 
 Running Ignite Benchmarks Locally
-==========================================
+=================================
 
-The simplest way to start with benchmarking is to use one of the executable scripts available under `benchmarks\bin` directory:
+The simplest way to start with benchmarking is to use one of the executable scripts available under `benchmarks\bin`
+directory:
 
-./bin/benchmark-run-all.sh config/benchmark-atomic-put.properties
+./bin/benchmark-run-all.sh config/benchmark-sample.properties
 
-The command above will benchmark the cache put operation for a distributed atomic cache. The results of the benchmark will be added
-to auto-generated `results-{DATE-TIME}` directory.
+The command above will benchmark the cache put operation for a distributed atomic cache. The results of the
+benchmark will be added to an auto-generated `output/results-{DATE-TIME}` directory.
 
-If `./bin/benchmark-run-all.sh` command is executed as is without any parameters and modifications in configurations files then
-all the available benchmarks will be executed on a local machine using `config/benchmark.properties` configuration.
+If `./bin/benchmark-run-all.sh` command is executed as is without any parameters and modifications in configurations
+files then all the available benchmarks will be executed on a local machine using `config/benchmark.properties` configuration.
 
-To get more information about available benchmarks and configuration parameters refer to “Provided Benchmarks” and
-“Properties And Command Line Arguments” sections below.
+In case of any issue refer to the logs that are added to an auto-generated
+`output/logs-{DATE-TIME}` directory.
+
+To get more information about available benchmarks and configuration parameters refer
+to “Provided Benchmarks” and “Properties And Command Line Arguments” sections below.
 
 
 Running Ignite Benchmarks Remotely
-=========================================
+==================================
+
+To benchmark Apache Ignite across several remote hosts the following steps need
+to be done:
 
-For running Ignite benchmarks on remote hosts you need to upload Ignite-Yardstick to each one of your remote hosts.
+1. Go to `config/ignite-remote-config.xml` and replace
+`<value>127.0.0.1:47500..47509</value>` with actual IPs of all the remote hosts.
+Refer to the documentation section below if you prefer to use other kind of IP finder:
+https://apacheignite.readme.io/docs/cluster-config
 
-NOTE: The path to the uploaded Ignite-Yardstick should be exactly the same on each host.
+2. Go to `config/benchmark-remote-sample.properties` and replace the `localhost` with
+actual IPs of the remote hosts in the following places:
+SERVERS='localhost,localhost'
+DRIVERS='localhost'
 
-Then you need to make some changes in config files:
+DRIVERS is the remote hosts where benchmarks execution will be and driven.
 
-1. You need to comment or delete the
-        <property name="localHost" value="127.0.0.1"/>
-line in `config/ignite-localhost-config.xml` file.
+Replace `localhost` occurrences in the same places in
+`config/benchmark-remote.properties` files if you plan to execute a full set of
+benchmarks available.
 
-2. You need to replace all the `127.0.0.1` addresses in `ignite-localhost-config.xml` file by actual IPs of your remote
-servers. You can add or delete lines with IP addresses if you want to run benchmarks on different number of hosts.
-There must be at least one IP address in the list.
+3. Upload Ignite Yardstick Benchmarks to one of your DRIVERS host in its own working directory.
 
-3. You need to replace the `localhost` strings by actual IP of your servers in the lines
-SERVERS='localhost,localhost'
-DRIVERS='localhost'
-in `config/benchmark-atomic-put.properties` file.
+4. Log in on the remote host that will be the DRIVER and execute the following command:
 
-Then use the following command:
+./bin/benchmark-run-all.sh config/benchmark-remote-sample.properties
 
-./bin/benchmark-run-all.sh config/benchmark-atomic-put.properties
+By default, all the necessary files will be automatically uploaded from the host in which you run the command above to
+every other host to the same path. If you prefer to do it manually set the AUTO_COPY variable in property file to `false`.
 
+The command above will benchmark the cache put operation for a distributed atomic cache. The results of the benchmark
+will be added to an auto-generated `output/results-{DATE-TIME}` directory.
 
-It is recommended to create some copies of original config files, edit these copies and then use as a
-parameter for `./bin/benchmark-run-all.sh` script.
+If you want to execute all the available benchmarks across the remote hosts then
+execute the following command on the DRIVER side:
+./bin/benchmark-run-all.sh config/benchmark-remote.properties
 
 
 Provided Benchmarks
@@ -90,9 +102,9 @@ The following benchmarks are provided:
 
 Properties And Command Line Arguments
 =====================================
-Note that this section only describes configuration parameters specific to Ignite benchmarks,
-and not for Yardstick framework. To run Ignite benchmarks and generate graphs, you will need to run them using
-Yardstick framework scripts in `bin` folder.
+Note that this section only describes configuration parameters specific to Ignite benchmarks, and not for Yardstick
+framework. To run Ignite benchmarks and generate graphs, you will need to run them using Yardstick framework scripts in
+`bin` folder.
 
 Refer to Yardstick Documentation (https://github.com/gridgain/yardstick) for common Yardstick properties
 and command line arguments for running Yardstick scripts.
@@ -102,7 +114,8 @@ The following Ignite benchmark properties can be defined in the benchmark config
 * `-b <num>` or `--backups <num>` - Number of backups for every key
 * `-cfg <path>` or `--Config <path>` - Path to Ignite configuration file
 * `-cs` or `--cacheStore` - Enable or disable cache store readThrough, writeThrough
-* `-cl` or `--client` - Client flag
+* `-cl` or `--client` - Client flag. Use this flag if you running more than one `DRIVER`, otherwise additional drivers
+would behave like a `servers`.
 * `-nc` or `--nearCache` - Near cache flag
 * `-nn <num>` or `--nodeNumber <num>` - Number of nodes (automatically set in `benchmark.properties`), used to wait for
     the specified number of nodes to start
@@ -141,4 +154,4 @@ Use Ignite Apache JIRA (https://issues.apache.org/jira/browse/IGNITE) to file bu
 
 License
 =======
-Yardstick Ignite is available under Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0.html) Open Source license.
+Yardstick Ignite is available under Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0.html) Open Source license.

modules/yardstick/config/benchmark-atomic-win.properties

@@ -51,7 +51,7 @@ set DRIVER_HOSTS=localhost
 :: set REMOTE_USER=
 
 :: Run configuration which contains all benchmarks.
-:: Note that each benchmark is set to run for 300 seconds (5 mins) with warm-up set to 60 seconds (1 minute).
+:: Note that each benchmark is set to run for 300 seconds (5 min) with warm-up set to 60 seconds (1 minute).
 set CONFIGS=^
 -cfg %SCRIPT_DIR%\..\config\ignite-localhost-config.xml -b 1 -w 60 -d 300 -t 64 -sm PRIMARY_SYNC -dn IgniteGetBenchmark -sn IgniteNode -ds atomic-get-1-backup,^
 -cfg %SCRIPT_DIR%\..\config\ignite-localhost-config.xml -b 1 -w 60 -d 300 -t 64 -sm PRIMARY_SYNC -dn IgniteGetOffHeapBenchmark -sn IgniteNode -ds atomic-get-offheap-1-backup,^

modules/yardstick/config/benchmark-atomic.properties

@@ -55,16 +55,31 @@ DRIVER_HOSTS=localhost
 # Number of nodes, used to wait for the specified number of nodes to start.
 nodesNum=$((`echo ${SERVER_HOSTS} | tr ',' '\n' | wc -l` + `echo ${DRIVER_HOSTS} | tr ',' '\n' | wc -l`))
 
+# Backups count.
+b=1
+
+# Warmup.
+w=60
+
+# Duration.
+d=300
+
+# Threads count.
+t=64
+
+# Sync mode.
+sm=PRIMARY_SYNC
+
 # Run configuration.
-# Note that each benchmark is set to run for 300 seconds (5 mins) with warm-up set to 60 seconds (1 minute).
+# Note that each benchmark is set to run for 300 seconds (5 min) with warm-up set to 60 seconds (1 minute).
 CONFIGS="\
--cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b 1 -w 60 -d 300 -t 64 -sm PRIMARY_SYNC -dn IgniteGetBenchmark -sn IgniteNode -ds atomic-get-1-backup,\
--cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b 1 -w 60 -d 300 -t 64 -sm PRIMARY_SYNC -dn IgniteGetOffHeapBenchmark -sn IgniteNode -ds atomic-get-offheap-1-backup,\
--cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b 1 -w 60 -d 300 -t 64 -sm PRIMARY_SYNC -dn IgniteGetOffHeapValuesBenchmark -sn IgniteNode -ds atomic-get-offheap-val-1-backup,\
--cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b 1 -w 60 -d 300 -t 64 -sm PRIMARY_SYNC -dn IgnitePutGetBenchmark -sn IgniteNode -ds atomic-put-get-1-backup,\
--cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b 1 -w 60 -d 300 -t 64 -sm PRIMARY_SYNC -dn IgnitePutGetOffHeapBenchmark -sn IgniteNode -ds atomic-put-get-offheap-1-backup,\
--cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b 1 -w 60 -d 300 -t 64 -sm PRIMARY_SYNC -dn IgnitePutGetOffHeapValuesBenchmark -sn IgniteNode -ds atomic-put-get-offheap-val-1-backup,\
--cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b 1 -w 60 -d 300 -t 64 -sm PRIMARY_SYNC -dn IgnitePutBenchmark -sn IgniteNode -ds atomic-put-1-backup,\
--cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b 1 -w 60 -d 300 -t 64 -sm PRIMARY_SYNC -dn IgnitePutOffHeapBenchmark -sn IgniteNode -ds atomic-put-offheap-1-backup,\
--cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b 1 -w 60 -d 300 -t 64 -sm PRIMARY_SYNC -dn IgnitePutOffHeapValuesBenchmark -sn IgniteNode -ds atomic-put-offheap-val-1-backup\
+-cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgniteGetBenchmark -sn IgniteNode -ds atomic-get-${b}-backup,\
+-cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgniteGetOffHeapBenchmark -sn IgniteNode -ds atomic-get-offheap-${b}-backup,\
+-cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgniteGetOffHeapValuesBenchmark -sn IgniteNode -ds atomic-get-offheap-val-${b}-backup,\
+-cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgnitePutGetBenchmark -sn IgniteNode -ds atomic-put-get-${b}-backup,\
+-cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgnitePutGetOffHeapBenchmark -sn IgniteNode -ds atomic-put-get-offheap-${b}-backup,\
+-cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgnitePutGetOffHeapValuesBenchmark -sn IgniteNode -ds atomic-put-get-offheap-val-${b}-backup,\
+-cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgnitePutBenchmark -sn IgniteNode -ds atomic-put-${b}-backup,\
+-cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgnitePutOffHeapBenchmark -sn IgniteNode -ds atomic-put-offheap-${b}-backup,\
+-cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgnitePutOffHeapValuesBenchmark -sn IgniteNode -ds atomic-put-offheap-val-${b}-backup\
 "

modules/yardstick/config/benchmark-bin-identity.properties

@@ -24,7 +24,6 @@ JVM_OPTS=${JVM_OPTS}" -DIGNITE_QUIET=false"
 
 # Uncomment to enable concurrent garbage collection (GC) if you encounter long GC pauses.
 JVM_OPTS=${JVM_OPTS}" \
--Xloggc:./gc${now0}.log \
 -XX:+PrintGCDetails \
 -verbose:gc \
 -XX:+UseParNewGC \
@@ -48,7 +47,7 @@ BENCHMARK_DEFAULT_PROBES=ThroughputLatencyProbe,PercentileProbe,DStatProbe
 # Packages where the specified benchmark is searched by reflection mechanism.
 BENCHMARK_PACKAGES=org.yardstickframework,org.apache.ignite.yardstick
 
-# Restart servers for each benchmark.
+# Flag which indicates to restart the servers before every benchmark execution.
 RESTART_SERVERS=true
 
 # Probe point writer class name.
@@ -81,14 +80,11 @@ t=64
 # Sync mode.
 sm=PRIMARY_SYNC
 
-# Jobs.
-j=10
-
 # Run configuration which contains all benchmarks.
-# Note that each benchmark is set to run for 300 seconds (5 mins) with warm-up set to 60 seconds (1 minute).
+# Note that each benchmark is set to run for 300 seconds (5 min) with warm-up set to 60 seconds (1 minute).
 CONFIGS="\
--cfg ${SCRIPT_DIR}/../config/ignite-bin-multicast-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgniteFieldsBinaryIdentityGetBenchmark -sn IgniteNode -ds ${ver}fields-bin-id-atomic-get-1-backup,\
--cfg ${SCRIPT_DIR}/../config/ignite-bin-multicast-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgniteFieldsBinaryIdentityPutBenchmark -sn IgniteNode -ds ${ver}fields-bin-id-atomic-put-1-backup,\
--cfg ${SCRIPT_DIR}/../config/ignite-bin-multicast-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgniteLegacyBinaryIdentityGetBenchmark -sn IgniteNode -ds ${ver}legacy-bin-id-atomic-get-1-backup,\
--cfg ${SCRIPT_DIR}/../config/ignite-bin-multicast-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgniteLegacyBinaryIdentityPutBenchmark -sn IgniteNode -ds ${ver}legacy-bin-id-atomic-put-1-backup\
+-cfg ${SCRIPT_DIR}/../config/ignite-bin-multicast-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgniteFieldsBinaryIdentityGetBenchmark -sn IgniteNode -ds ${ver}fields-bin-id-atomic-get-${b}-backup,\
+-cfg ${SCRIPT_DIR}/../config/ignite-bin-multicast-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgniteFieldsBinaryIdentityPutBenchmark -sn IgniteNode -ds ${ver}fields-bin-id-atomic-put-${b}-backup,\
+-cfg ${SCRIPT_DIR}/../config/ignite-bin-multicast-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgniteLegacyBinaryIdentityGetBenchmark -sn IgniteNode -ds ${ver}legacy-bin-id-atomic-get-${b}-backup,\
+-cfg ${SCRIPT_DIR}/../config/ignite-bin-multicast-config.xml -nn ${nodesNum} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgniteLegacyBinaryIdentityPutBenchmark -sn IgniteNode -ds ${ver}legacy-bin-id-atomic-put-${b}-backup\
 "

modules/yardstick/config/benchmark-cache-load-win.properties

@@ -54,7 +54,7 @@ set DRIVER_HOSTS=localhost
 set ops=PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL PUT_IF_ABSENT REPLACE SCAN_QUERY SQL_QUERY CONTINUOUS_QUERY
 
 :: Run configuration which contains all benchmarks.
-:: Note that each benchmark is set to run for 300 seconds (5 mins) with warm-up set to 60 seconds (1 minute).
+:: Note that each benchmark is set to run for 300 seconds (5 min) with warm-up set to 60 seconds (1 minute).
 set CONFIGS=^
 -cfg %SCRIPT_DIR%\..\config\ignite-cache-load-config.xml -r 10000 -pa 500 -bs 50 -b 1 -w 20 -d 30 -t 64 -sm PRIMARY_SYNC -dn IgniteCacheRandomOperationBenchmark -sn IgniteNode -ds cache-random-benchmark-1-backup -nn 1^
   -ltqf config\queries.sql -ltops %ops%

modules/yardstick/config/benchmark-cache-load.properties

@@ -21,7 +21,6 @@ JVM_OPTS=${JVM_OPTS}" -DIGNITE_QUIET=false"
 
 # Uncomment to enable concurrent garbage collection (GC) if you encounter long GC pauses.
 JVM_OPTS=${JVM_OPTS}" \
--Xloggc:./gc${now0}.log \
 -XX:+PrintGCDetails \
 -verbose:gc \
 -XX:+UseParNewGC \
@@ -79,14 +78,13 @@ pa=500000
 r=1000000
 
 # Batch size
-
 bs=50
 
 # Allow Operation
 ops="PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL PUT_IF_ABSENT REPLACE SCAN_QUERY SQL_QUERY CONTINUOUS_QUERY"
 
 # Run configuration.
-# Note that each benchmark is set to run for 300 seconds (5 mins) with warm-up set to 60 seconds (1 minute).
+# Note that each benchmark is set to run for 300 seconds (5 min) with warm-up set to 60 seconds (1 minute).
 CONFIGS="\
 -cfg ${SCRIPT_DIR}/../config/ignite-cache-load-config.xml -r ${r} -pa ${pa} -bs ${bs} -b ${b} -w ${w} -d ${d} -t ${t} -sm ${sm} -dn IgniteCacheRandomOperationBenchmark -sn IgniteNode -ds cache-random-benchmark-${b}-backup -nn ${nodesNum} \
 -ltqf ${SCRIPT_DIR}/../config/queries.sql -ltops ${ops}"