Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance microbenchmarks #209

Closed
wants to merge 9 commits into from
Closed

Performance microbenchmarks #209

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a10cbf4
wip basic benchmark
elefeint Dec 6, 2019
6dc3979
more benchmarks, cleaner structure
elefeint Dec 17, 2019
5fb9901
removed original met files source directory
elefeint Dec 19, 2019
c4866cc
restructured
elefeint Jan 9, 2020
d279e4f
move packages, fix checkstyle, add README
elefeint Jan 14, 2020
7a42cb2
fixed merge
elefeint Jan 14, 2020
3de70ba
added run results
elefeint Jan 14, 2020
97e1489
remove change from another pr
elefeint Jan 14, 2020
b5619d1
cd
elefeint Feb 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions cloud-spanner-r2dbc-performance-testing/README.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
== Set up data

These benchmarks use a public Met museum artwork data set in Cloud Spanner export format.
elefeint marked this conversation as resolved.
Show resolved Hide resolved
The dataset is included on the top level of this repository.

0. Unarchive `benchmark-data-met.tar.gz` into a folder `benchmark-data-met`.
elefeint marked this conversation as resolved.
Show resolved Hide resolved
0. Upload the folder into a valid GCP Storage bucket.
0. In a valid Spanner instance, select IMPORT/EXPORT tab.
0. Follow the import link.
0. Follow importing form instructions:
* Choose your bucket and folder as the source folder (don't forget the trailing slash).
* Give the name for a new database (the import will create it).
* Choose your region, confirm charges and hit Import.

This will create a dataflow job importing the data.
Once the job is done, you will have a table named `met_objects` with 200,074 rows.

If you are curious, this https://cloud.google.com/blog/products/gcp/when-art-meets-big-data-analyzing-200000-items-from-the-met-collection-in-bigquery[blog post] has more details about the data set.

== Run benchmarks:

```
cd cloud-spanner-r2dbc-performance-testing/

mvn clean install

java -jar -Dspanner.instance=[INSTANCE] -Dspanner.database=[DATABASE] -Dgcp.project=[PROJECT] target/benchmarks.jar
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why aren't we using Maven here to run? Like mvn:exec.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's possible, but it would require adding a main() method to have a runnable entry point and adding all benchmark classes to the configuration. Then if we want ability to exclude/include individual classes, we'd have to set up custom parameters that main method would pass into JMH runner.

Too much hassle given that realistically we are neither going to ever run those from an IDE nor make them a part of the build.

```

To run individual benchmark class (in this case, `DmlBenchmark`):
```
java -jar -Dspanner.instance=[INSTANCE] -Dspanner.database=[DATABASE] -Dgcp.project=[PROJECT] target/benchmarks.jar DmlBenchmark
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a parameter that controls the volume of the benchmark so that one can choose how long they want to run it for? In other words, if you want really accurate results, run for 30 minutes /1000 iterations, if you just want to get quick results after 2 minutes, pass some different parameters.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

those parameters come free with JMH actually. I'll document.

```


== Benchmark Results (2020/01/14)

Performing DDL with client library seems to have a lot of variability in performance.
This may well be by design -- GAX uses https://github.com/googleapis/gax-java/blob/a60bd347b69fbf725b0dd8ae18bbcf5b00b66b7b/gax-grpc/src/main/java/com/google/longrunning/stub/OperationsStubSettings.java#L242[exponential backoff] with initial delay of 100ms to execute long-running operations. R2DBC https://github.com/GoogleCloudPlatform/cloud-spanner-r2dbc/blob/8eff19dbbac92fbbb42473795e6d810cc77565ce/cloud-spanner-r2dbc/src/main/java/com/google/cloud/spanner/r2dbc/SpannerConnectionConfiguration.java#L155-L157[by default] queries every 5 seconds up to a timeout of 10 minutes.

For all other operations, the numbers look fairly similar between client library and R2DBC.

Full run output is in `benchmark-run-20200114-1.txt` and `benchmark-run-20200114-2.txt`.

=== Run 1

```
Benchmark Mode Cnt Score Error Units
DdlBenchmark.testDdlClientLibrary avgt 5 32.172 ± 66.025 s/op
DdlBenchmark.testDdlR2dbcDriver avgt 5 11.286 ± 0.269 s/op

DmlBenchmark.testDmlClientLibrary avgt 5 86.026 ± 11.228 ms/op
DmlBenchmark.testDmlR2dbcDriver avgt 5 83.427 ± 4.887 ms/op

QueryingBenchmark.tesFirstResponseQueryingClientLibrary avgt 5 25.980 ± 0.900 ms/op
QueryingBenchmark.testFirstResponseQueryingR2dbc avgt 5 26.667 ± 0.759 ms/op

QueryingBenchmark.testQueryingClientLibrary avgt 5 27.117 ± 1.408 ms/op
QueryingBenchmark.testQueryingR2dbc avgt 5 26.108 ± 0.758 ms/op

```

=== Run 2
```
Benchmark Mode Cnt Score Error Units
DdlBenchmark.testDdlClientLibrary avgt 5 27.348 ± 41.975 s/op
DdlBenchmark.testDdlR2dbcDriver avgt 5 11.355 ± 0.149 s/op

DmlBenchmark.testDmlClientLibrary avgt 5 86.507 ± 12.852 ms/op
DmlBenchmark.testDmlR2dbcDriver avgt 5 85.155 ± 0.719 ms/op

QueryingBenchmark.tesFirstResponseQueryingClientLibrary avgt 5 26.666 ± 1.662 ms/op
QueryingBenchmark.testFirstResponseQueryingR2dbc avgt 5 25.640 ± 0.820 ms/op

QueryingBenchmark.testQueryingClientLibrary avgt 5 26.690 ± 3.304 ms/op
QueryingBenchmark.testQueryingR2dbc avgt 5 26.470 ± 0.941 ms/op
```
Binary file added cloud-spanner-r2dbc-performance-testing/benchmark-data-met.tar.gz
View file
Open in desktop
Binary file not shown.
449 changes: 449 additions & 0 deletions cloud-spanner-r2dbc-performance-testing/benchmark-run-20200114-1.txt
View file
Open in desktop

Large diffs are not rendered by default.

449 changes: 449 additions & 0 deletions cloud-spanner-r2dbc-performance-testing/benchmark-run-20200114-2.txt
View file
Open in desktop

Large diffs are not rendered by default.

196 changes: 196 additions & 0 deletions cloud-spanner-r2dbc-performance-testing/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2014, Oracle America, Inc.
elefeint marked this conversation as resolved.
Show resolved Hide resolved
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

* Neither the name of Oracle nor the names of its contributors may be used
to endorse or promote products derived from this software without
specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
THE POSSIBILITY OF SUCH DAMAGE.
--><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<artifactId>cloud-spanner-r2dbc-parent</artifactId>
<groupId>com.google.cloud</groupId>
<version>0.2.0-SNAPSHOT</version>
</parent>

<groupId>com.google.cloud</groupId>
<artifactId>cloud-spanner-r2dbc-performance-testing</artifactId>
<version>0.2.0-SNAPSHOT</version>
<packaging>jar</packaging>

<name>JMH benchmark sample: Java</name>

<!--
This is the demo/sample template build script for building Java benchmarks with JMH.
Edit as needed.
-->

<dependencies>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>${jmh.version}</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
<scope>provided</scope>
</dependency>

<!-- R2DBC driver -->
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>cloud-spanner-r2dbc</artifactId>
<version>${project.version}</version>
</dependency>

<!-- Client library -->
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-spanner</artifactId>
</dependency>


<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>compile</scope>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't seem to be using junit and it almost never should be compile scope.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a test-only project, but yeah. (continues blaming archetype)

</dependency>
</dependencies>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<!--
JMH version to use with this project.
-->
<jmh.version>1.22</jmh.version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Properties usually go at the top of the file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blame the archetype. But yeah, I'll move it.


<!--
Java source/target to use for compilation.
-->
<javac.target>1.8</javac.target>

<!--
Name of the benchmark Uber-JAR to generate.
-->
<uberjar.name>benchmarks</uberjar.name>
</properties>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<compilerVersion>${javac.target}</compilerVersion>
<source>${javac.target}</source>
<target>${javac.target}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems only necessary because you're not using Maven to run the project.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JMH archetype generates this; if we want to have benchmarks running automatically, we can create our own configuration with programmatic JMH invocation.

<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>${uberjar.name}</finalName>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.openjdk.jmh.Main</mainClass>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
</transformers>
<filters>
<filter>
<!--
Shading signed JARs will fail without this.
http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar
-->
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<plugin>
<artifactId>maven-clean-plugin</artifactId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's really odd to see all of these plugins version-managed. I would skip doing it unless there is a specific reason for it.

<version>2.5</version>
</plugin>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be configured so it does not get released?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't be part of the release maven profile; on the other hand, can't hurt.

<version>2.8.1</version>
</plugin>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.1</version>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
</plugin>
<plugin>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>2.6</version>
</plugin>
<plugin>
<artifactId>maven-site-plugin</artifactId>
<version>3.3</version>
</plugin>
<plugin>
<artifactId>maven-source-plugin</artifactId>
<version>2.2.1</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
</plugin>
</plugins>
</pluginManagement>
</build>

</project>
Loading