airframe-benchmark: Aggregate benchmark programs into one project (#573)

* Merge benchmarks into airframe-benchmark
* Upgrade sbt-jmh plugin
* Run benchmark within test
* Add JSON JMH benchmark
* Allow filtering benchmark package
* Reorganize json benchmark groups
* Fix readme
* Fix warmup/measurement time config
* Add int/double json array benchmark
* Remove uninteresting performance comparison
* Add warning
* Use the latest circe and json4s-jackson
* Build benchmark only for Scala 2.12

airframe-msgpack-benchmark/README.md → airframe-benchmark/README.md

@@ -1,24 +1,28 @@
 airframe-msgpack-benchmark
 ===
 
-MessagePack benchmark program based on [JMH](https://openjdk.java.net/projects/code-tools/jmh/).
+Airframe benchmark program based on [JMH](https://openjdk.java.net/projects/code-tools/jmh/).
 
 
-## How To Build
-```
-$ ./sbt
-> msgpackBenchmark/pack
+### Download
 
-# To install this program to $HOME/local/bin, run:
-> msgpackBenchmark/packInstall
-```
+- Download the latest tar.gz package from [here](https://oss.sonatype.org/content/repositories/snapshots/org/wvlet/airframe/airframe-benchmark_2.12/)
+- Download this archive, and unpack it with `tar xvfz`
+- Moved to the unpacked folder, then run `./bin/airframe-benchmark bench msgpack`
 
 ### Running the benchmark
 ```
-$ cd airframe-msgpack-benchmark/target/pack
-# In another terminal, run this command. The result will be written to a json file: 
-$ ./bin/airframe-msgpack-benchmark bench -f json
+$ cd airframe-benchmark
 
+# Run MessagePack benchmark
+$ ./bin/airframe-benchmark bench msgpack
+
+# Run JSON benchmark
+$ ./bin/airframe-benchmark bench json
+
+# Run MessagePack benchmaark and write the results to a json file
+# Run MessagePack benchmark
+$ ./bin/airframe-benchmark bench msgpack -f json
 ...
 
 # Run complete. Total time: 00:00:31
@@ -41,26 +45,48 @@ MessagePackJava.unpack:intArray      thrpt   50  121146.462 ±  3870.203  ops/s
 PackBenchmark.packInt                thrpt   50  105945.238 ±  2251.499  ops/s
 ```
 
-### Usage
+### General Usage
 ```
-$ airframe-msgpack-benchmark bench --help
-usage: bench
+$ airframe-benchmark bench --help
+usage: bench [targetPackage]
   Run a benchmark
 
 [global options]
- -h, --help             display help message
- -f [RESULTFORMAT]      Result format: text, csv, scsv, json, latex
- -o [RESULTOUTPUT]      Result output file name
- -mt:[MEASUREMENTTIME]  measurement time (default: 0.1s)
- -wt:[WARMUPTIME]       warmup time (default: 0.1s)
+ -h, --help         display help message
+ -f [RESULTFORMAT]  Result format: text, csv, scsv, json, latex
+ -o [RESULTOUTPUT]  Result output file name
+ -wt:[WARMUPTIME]   warmup time (default: 0.1s)
 [options]
  -i, --iteration:[ITERATION]     The number of iteration (default: 10)
- -w, --warmup:[WARMUPITERATION]  The number of warm-up iteration (default: 5)
- -f, --fork-count:[FORKCOUNT]    Fork Count (default: 5)
+ -w, --warmup:[WARMUPITERATION]  The number of warm-up iteration (default: 3)
+ -mt:[MEASUREMENTTIME]           measurement time (default: 1s)
 ```
 
-### Download
+## Building The Benchmark Runner
+```
+$ ./sbt
+> benchmark/pack
 
-- [Initial version](https://oss.sonatype.org/content/repositories/snapshots/org/wvlet/airframe/airframe-msgpack-benchmark_2.12/19.3.4+12-0a821f46+20190315-1700-SNAPSHOT/airframe-msgpack-benchmark_2.12-19.3.4+12-0a821f46+20190315-1700-SNAPSHOT.tar.gz)
-- Download this archive, and unpack it with `tar xvfz`
-- Moved to the unpacked folder, then run `./bin/airframe-msgpack-benchmark bench`
+# To install this program to $HOME/local/bin, run:
+> benchmark/packInstall
+```
+
+# For Developers
+
+## Running Benchmark While Developing Airframe
+
+```
+$ ./sbt
+
+# Run JSON benchmark:
+> benchmark/testOnly * -- -n json
+
+# Run benchmark for measuring JSON parse time:
+> benchmark/testOnly * -- -n json-perf
+
+# Run Msgpack benchmark:
+> benchmark/testOnly * -- -n msgpack
+
+# Run all JMH benchmarks:
+> benchmark/run bench-quick
+```