Kafka benchmark measures Thread.sleep(100)

[farquet]

On all VMs I tested, the iteration time for the kafka benchmark is very stable and the score is around 105ms per iteration. A JFR profile of the benchmark shows that the hot methods are the harness methods and Unsafe.park which gives the impression that the benchmark is just waiting. The very low CPU activity during the benchmark run is another hint of this.

Looking at the code, it appears that the iteration code is :
https://github.com/dacapobench/dacapobench/blob/dev-chopin/benchmarks/bms/kafka/src/org/dacapo/kafka/Launcher.java#L44-L45
which calls ClientRunner.runClient which does this :
https://github.com/dacapobench/dacapobench/blob/dev-chopin/benchmarks/bms/kafka/src/org/dacapo/kafka/ClientRunner.java#L25-L34
The system property that is the exit condition of the while loop is set to the desired value in this Kafka patch :
https://github.com/dacapobench/dacapobench/blob/dev-chopin/benchmarks/bms/kafka/kafka.patch#L11

So it looks like the Kafka task is executed very quickly and that most of the iteration time is spent in Thread.sleep(100) until the next check of the system property.

I double checked this by compiling the suite again after changing Thread.sleep(100) to Thread.sleep(50) and the iteration time goes down to around 55ms as expected.

[farquet]

Note that if the Kafka task fails, the benchmark will happily set the TaskState system property to DONE, finishing up the iteration. This may hide some underlying failures and skew the results.
You may want to set a different state if there is an error and watch for this to validate the output of the iteration :
https://github.com/dacapobench/dacapobench/blob/dev-chopin/benchmarks/bms/kafka/kafka.patch#L13