Skip to content

Commit

Permalink
Convert benchmarks to JUnit
Browse files Browse the repository at this point in the history
  • Loading branch information
Dave Syer committed Jun 15, 2020
1 parent 21c61b4 commit 79f5f76
Show file tree
Hide file tree
Showing 8 changed files with 125 additions and 99 deletions.
95 changes: 33 additions & 62 deletions benchmarks/README.adoc
View file
Expand Up @@ -2,82 +2,53 @@

== How to run

The Exec Maven Plugin will run the `benchmarks.jar` file to start the benchmarking process:
The Surefire Maven Plugin will run the benchmarking process:

```
$ cd benchmarks
$ mvn clean install exec:exec
$ mvn test -Dtest=StartupBenchmark
```

Outside Maven after building the benchmarks.jar you can just run:

```
$ cd benchmarks
$ java -jar target/benchmarks.jar
```
Outside Maven after building the benchmarks.jar you can just run the `StartupBenchmark` in your IDE.

= Results

== macBook Lofi Dewanto Result April 2020
- CPU: 2.7 GHz Quad-Core Intel Core i7
- Memory: 16 GB 2133 MHz LPDDR3

=== AspectJ 1.9.5, OpenJDK 11, JMH 1.23, Spring Boot 2.2.4
=== JDK 11

```
Result "com.bench.StartupBenchmark.spring":
3,539 ±(99.9%) 0,390 s/op [Average]
(min, avg, max) = (3,173, 3,539, 3,885), stdev = 0,258
CI (99.9%): [3,149, 3,929] (assumes normal distribution)

# Run complete. Total time: 00:29:58

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark (scale) Mode Cnt Score Error Units
StartupBenchmark.ltw N/A avgt 10 2,467 ± 0,256 s/op
StartupBenchmark.ltw_100 N/A avgt 10 2,742 ± 0,530 s/op
StartupBenchmark.spring v0_10 avgt 10 1,707 ± 0,090 s/op
StartupBenchmark.spring v1_10 avgt 10 2,098 ± 0,923 s/op
StartupBenchmark.spring v1_100 avgt 10 2,031 ± 0,169 s/op
StartupBenchmark.spring v10_50 avgt 10 2,885 ± 0,303 s/op
StartupBenchmark.spring v20_50 avgt 10 3,961 ± 0,466 s/op
StartupBenchmark.spring a0_10 avgt 10 2,304 ± 0,445 s/op
StartupBenchmark.spring a1_10 avgt 10 2,652 ± 0,692 s/op
StartupBenchmark.spring a1_100 avgt 10 2,666 ± 0,491 s/op
StartupBenchmark.spring a10_50 avgt 10 3,087 ± 0,277 s/op
StartupBenchmark.spring a10_100 avgt 10 3,379 ± 1,037 s/op
StartupBenchmark.spring a20_50 avgt 10 3,539 ± 0,390 s/op
class method scale median mean range
com.bench.StartupBenchmark ltw 2.398 2.473 0.113
com.bench.StartupBenchmark ltw_100 2.404 2.431 0.037
com.bench.StartupBenchmark spring v0_10 1.604 1.648 0.039
com.bench.StartupBenchmark spring v1_10 1.714 1.748 0.030
com.bench.StartupBenchmark spring v1_100 1.767 1.792 0.019
com.bench.StartupBenchmark spring v10_50 2.204 2.284 0.062
com.bench.StartupBenchmark spring v20_50 2.801 2.828 0.033
com.bench.StartupBenchmark spring a0_10 1.621 1.659 0.042
com.bench.StartupBenchmark spring a1_10 1.704 1.772 0.139
com.bench.StartupBenchmark spring a1_100 1.700 1.774 0.157
com.bench.StartupBenchmark spring a10_50 2.004 2.104 0.146
com.bench.StartupBenchmark spring a10_100 1.943 1.988 0.084
com.bench.StartupBenchmark spring a20_50 2.216 2.268 0.056
```

=== AspectJ 1.9.2, Oracle JDK 1.8, JMH 1.16, Spring Boot 2.1.1 -- Original Source before Move to newer Versions
=== JDK 8

```
Result "spring":
2,918 ±(99.9%) 0,056 s/op [Average]
(min, avg, max) = (2,862, 2,918, 2,971), stdev = 0,037
CI (99.9%): [2,862, 2,975] (assumes normal distribution)

# Run complete. Total time: 00:06:06

Benchmark (scale) Mode Cnt Score Error Units
StartupBenchmark.ltw N/A avgt 10 0,017 ± 0,001 s/op
StartupBenchmark.ltw_100 N/A avgt 10 0,017 ± 0,001 s/op
StartupBenchmark.spring v0_10 avgt 10 2,158 ± 0,131 s/op
StartupBenchmark.spring v1_10 avgt 10 2,235 ± 0,056 s/op
StartupBenchmark.spring v1_100 avgt 10 2,273 ± 0,076 s/op
StartupBenchmark.spring v10_50 avgt 10 2,908 ± 0,055 s/op
StartupBenchmark.spring v20_50 avgt 10 3,622 ± 0,041 s/op
StartupBenchmark.spring a0_10 avgt 10 2,083 ± 0,054 s/op
StartupBenchmark.spring a1_10 avgt 10 2,183 ± 0,040 s/op
StartupBenchmark.spring a1_100 avgt 10 2,219 ± 0,041 s/op
StartupBenchmark.spring a10_50 avgt 10 2,539 ± 0,044 s/op
StartupBenchmark.spring a10_100 avgt 10 2,567 ± 0,034 s/op
StartupBenchmark.spring a20_50 avgt 10 2,918 ± 0,056 s/op
class method scale median mean range
com.bench.StartupBenchmark ltw 2.020 2.044 0.063
com.bench.StartupBenchmark ltw_100 2.050 2.078 0.042
com.bench.StartupBenchmark spring v0_10 1.532 1.550 0.018
com.bench.StartupBenchmark spring v1_10 1.621 1.662 0.051
com.bench.StartupBenchmark spring v1_100 1.653 1.669 0.026
com.bench.StartupBenchmark spring v10_50 2.120 2.148 0.036
com.bench.StartupBenchmark spring v20_50 2.657 2.692 0.034
com.bench.StartupBenchmark spring a0_10 1.517 1.537 0.025
com.bench.StartupBenchmark spring a1_10 1.624 1.638 0.019
com.bench.StartupBenchmark spring a1_100 1.651 1.670 0.015
com.bench.StartupBenchmark spring a10_50 1.905 1.927 0.020
com.bench.StartupBenchmark spring a10_100 1.900 1.931 0.035
com.bench.StartupBenchmark spring a20_50 2.124 2.162 0.052
```

== Older Results
Expand Down
82 changes: 63 additions & 19 deletions benchmarks/pom.xml
View file
Expand Up @@ -65,15 +65,72 @@
<version>${jmh.version}</version>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>com.github.mp911de.microbenchmark-runner</groupId>
<artifactId>microbenchmark-runner-junit5</artifactId>
<version>0.2.0.RELEASE</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.github.mp911de.microbenchmark-runner</groupId>
<artifactId>microbenchmark-runner-extras</artifactId>
<version>0.2.0.RELEASE</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</exclusion>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>

</dependencies>

<profiles>
<profile>
<id>tools.jar</id>
<activation>
<jdk>[1.8,1.9)</jdk>
</activation>
<dependencies>
<dependency>
<groupId>com.sun</groupId>
<artifactId>tools</artifactId>
<version>1.8</version>
<scope>system</scope>
<systemPath>${java.home}/../lib/tools.jar</systemPath>
</dependency>
</dependencies>
</profile>
</profiles>

<build>
<finalName>benchmarks</finalName>
<plugins>
<plugin>
<artifactId>maven-shade-plugin</artifactId>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<finalName>benchmarks</finalName>
<annotationProcessorPaths>
<path>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
</path>
</annotationProcessorPaths>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
<plugin>
Expand All @@ -83,27 +140,14 @@
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<configuration>
<executable>java</executable>
<arguments>
<argument>-jar</argument>
<argument>${project.build.directory}/benchmarks.jar</argument>
</arguments>
</configuration>
</plugin>
</plugins>
</build>

<repositories>
<repository>
<id>jitpack.io</id>
<url>https://jitpack.io</url>
</repository>
<repository>
<id>spring-snapshots</id>
<name>Spring Snapshots</name>
Expand Down
0 .../java/com/bench/ProcessLauncherState.java → .../java/com/bench/ProcessLauncherState.java
View file
File renamed without changes.
13 changes: 7 additions & 6 deletions benchmarks/src/test/java/com/bench/ProcessLauncherStateTest.java
View file
@@ -1,9 +1,7 @@
package com.bench;

import static org.junit.Assert.assertTrue;
import static org.junit.jupiter.api.Assertions.assertTrue;

import org.junit.Ignore;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
Expand All @@ -14,11 +12,14 @@ class ProcessLauncherStateTest {

private final String CLASSPATH_PART_RESULT = "spring-boot-aspectj/benchmarks/target/test-classes/";

@BeforeEach
void setUp() throws Exception {
@Test
void testApplication() throws Exception {
ProcessLauncherState processLauncherState = new ProcessLauncherState("target");
processLauncherState.run();
processLauncherState.after();
}

@Ignore
@Test
void testGetClasspathJdk8() {
ProcessLauncherState processLauncherState = new ProcessLauncherState("", "");
String classpathName = processLauncherState.getClasspathJdk8();
Expand Down
27 changes: 19 additions & 8 deletions ...main/java/com/bench/StartupBenchmark.java → ...test/java/com/bench/StartupBenchmark.java
View file
Expand Up @@ -15,6 +15,9 @@
*/
package com.bench;

import org.junit.platform.commons.annotation.Testable;
import org.openjdk.jmh.annotations.AuxCounters;
import org.openjdk.jmh.annotations.AuxCounters.Type;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Fork;
Expand All @@ -28,27 +31,33 @@
import org.openjdk.jmh.annotations.TearDown;
import org.openjdk.jmh.annotations.Warmup;

@Measurement(iterations = 5)
@Warmup(iterations = 1)
import jmh.mbr.junit5.Microbenchmark;

@Measurement(iterations = 5, time = 1)
@Warmup(iterations = 1, time = 1)
@Fork(value = 2, warmups = 0)
@BenchmarkMode(Mode.AverageTime)
@Microbenchmark
public class StartupBenchmark {

private static final String M2_REPOSITORY_ASPECTJWEAVER = "/.m2/repository/org/aspectj/aspectjweaver/1.9.5/aspectjweaver-1.9.5.jar";

@Benchmark
@Testable
public void spring(SpringState state) throws Exception {
state.run();
}

@Benchmark
@Testable
public void ltw(ApplicationState state) throws Exception {
state.setProgArgs("--spring.aop.auto=false");
state.setJvmArgs("-javaagent:" + home() + M2_REPOSITORY_ASPECTJWEAVER);
state.run();
}

@Benchmark
@Testable
public void ltw_100(ApplicationState state) throws Exception {
state.setProgArgs("--bench.beans=100", "--spring.aop.auto=false");
state.setJvmArgs("-javaagent:" + home() + M2_REPOSITORY_ASPECTJWEAVER);
Expand All @@ -59,7 +68,8 @@ private String home() {
return System.getProperty("user.home");
}

@State(Scope.Benchmark)
@State(Scope.Thread)
@AuxCounters(Type.EVENTS)
public static class SpringState extends ProcessLauncherState {

public static enum Scale {
Expand Down Expand Up @@ -87,32 +97,33 @@ public String[] getArgs() {
}
}

@Param
@Param // ("a1_100")
private Scale scale = Scale.v1_10;

public SpringState() {
super("target");
}

@Setup(Level.Iteration)
@Setup(Level.Trial)
public void start() throws Exception {
setProgArgs(scale.getArgs());
}

@TearDown(Level.Iteration)
@TearDown(Level.Invocation)
public void stop() throws Exception {
super.after();
}
}

@State(Scope.Benchmark)
@State(Scope.Thread)
@AuxCounters(Type.EVENTS)
public static class ApplicationState extends ProcessLauncherState {

public ApplicationState() {
super("target", "--server.port=0");
}

@TearDown(Level.Iteration)
@TearDown(Level.Invocation)
public void stop() throws Exception {
super.after();
}
Expand Down
5 changes: 1 addition & 4 deletions benchmarks/src/test/java/com/example/InterceptorApplicationTests.java
View file
@@ -1,11 +1,8 @@
package com.example;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

@RunWith(SpringRunner.class)
@SpringBootTest(properties="logging.level.org.springframework.aop=TRACE")
public class InterceptorApplicationTests {

Expand Down
1 change: 1 addition & 0 deletions multi-ctw/pom.xml
View file
Expand Up @@ -6,6 +6,7 @@
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.2.4.RELEASE</version>
<relativePath/>
</parent>

<groupId>com.example</groupId>
Expand Down
1 change: 1 addition & 0 deletions multi-ltw/pom.xml
View file
Expand Up @@ -6,6 +6,7 @@
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.2.4.RELEASE</version>
<relativePath/>
</parent>

<groupId>com.example</groupId>
Expand Down

0 comments on commit 79f5f76

Please sign in to comment.