HiBench 6.0

We are happy to announce HiBench-6.0, a major release of HiBench. This release includes new features like more flexible build, Spark 2.0 support, more workloads, better configuration and a new streaming benchmark that supports Spark Streaming, Flink, Storm and Gearpump. It also includes many bug fixes to the previous release.

Flexible Build

HiBench 6.0 supports building only benchmarks for specific frameworks. Building all the benchmarks in HiBench could be time consuming because the hadoop benchmark relies on 3rd party tools like Mahout and Nutch. You can now only build for a specific framework to speed up the build process. If you are only interested in a single workload in HiBench, you can also only build a single module.

Spark 2.0 Support

Apache Spark 2.0 is a new major release with a few API changes. One of the features of HiBench 6.0 is to fully support Spark 1.6 and Spark 2.0. You can choose the Spark version when building HiBench and test these two Spark versions.

New Streaming Benchmark

A new streaming benchmark is included in HiBench 6.0. The new streaming benchmark supports Spark Streaming, Flink, Storm/Trident and Gearpump. There are four workloads identity, repartition, stateful wordcount and FixWindow. A common metrics framework was developed to assess all these streaming frameworks.

New Workloads

A new workload named NWeight for Spark is added. NWeight is an iterative graph-parallel algorithm implemented by Spark GraphX and Pregel. The algorithm computes associations between two vertices that are n-hop away.

Configurations

The configurations are split into several files for different frameworks. Each framework has its own configuration file. This simplifies the configurations because you may only interested in some of the frameworks. A few unnecessary configurations are also removed.

Continuous Integration

HiBench 6.0 uses Travis CI for continuous integration which builds the project and runs a set of workloads for testing.

Contributors

The following developers contributed to this release:
Adam Roberts (@a-roberts)
Andrew Audibert(@aaudiber)
Carlos Eduardo Moreira dos Santos(@cemsbr)
Carson Wang(@carsonwang)
Chenzhao Guo(@gczsjdy)
Daoyuan Wang(@adrian-wang)
Ge Chen(@princhenee)
Harschware(@harschware)
Huafeng Wang(@huafengw)
James Bogosian(@bogosj)
Jayanth(@prajay)
Ling Zhou(@lingzhouHZ)
Liye Zhang(@liyezhang556520)
Lun Gao(@gallenvara)
Mahmoud Ismail(@maismail)
Manu Zhang(@manuzhang)
Monta Yashi(@84monta)
Naresh Gundla(@nareshgundla)
Pengfei Xuan(@pfxuan)
Robert Schmidtke(@robert-schmidtke)
Shilei Qian(@qiansl127)
Tony Zhao(@touchdown)
Wei Mao(@mwws)
Xianyang Liu(@ConeyLiu)
Yanbing Zhang(@zybing)
Yi Cui(@Moonlit-Sailor)
ZheHan Wang(@han-wang)

Thanks to everyone who contributed! We are looking forward to more contributions from every one for the next release.