Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: use virtual threads #1267

Draft
wants to merge 51 commits into
base: postgresql-dialect
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
960807e
perf: use virtual threads
olavloite Dec 14, 2023
1e24245
build: create jdk21 build profile
olavloite Dec 16, 2023
ff86fc0
build: update Docker images to jdk21
olavloite Dec 16, 2023
f9daa7f
build: remove Python install
olavloite Dec 16, 2023
bd02b1f
build: keep JDK in download folder
olavloite Dec 16, 2023
7e890c8
Merge branch 'postgresql-dialect' into virtual-threads
olavloite Dec 17, 2023
fe248d4
Merge branch 'postgresql-dialect' into virtual-threads
olavloite Dec 17, 2023
f1d4d2f
fix: install python 2
olavloite Dec 17, 2023
e931dc2
Merge branch 'postgresql-dialect' into virtual-threads
olavloite Dec 25, 2023
5bbc307
chore: update copyright year
olavloite Dec 25, 2023
f764427
Merge branch 'postgresql-dialect' into virtual-threads
olavloite Dec 28, 2023
056e947
feat: use virtual threads for connection
olavloite Dec 29, 2023
a799ccc
feat: automatically use virtual threads
olavloite Dec 29, 2023
cab5a69
test: add Java 21 to test matrix
olavloite Dec 29, 2023
2167537
test: run unit tests with virtual threads disabled
olavloite Dec 29, 2023
c5afcd6
test: reduce matrix for emulator
olavloite Dec 29, 2023
e505c18
test: run units with Java 11 also
olavloite Dec 29, 2023
e64b7d1
test: test with virtual threads in connection api
olavloite Dec 30, 2023
6296602
test: add benchmarks
olavloite Dec 31, 2023
6040d9d
feat: make parallelism configurable
olavloite Dec 31, 2023
cafc412
feat: print results at end
olavloite Dec 31, 2023
5adde6f
fix: calculate ops/s based on current runtime
olavloite Dec 31, 2023
dbf5b14
feat: add more benchmarks
olavloite Dec 31, 2023
96f000a
feat: make numChannels configurable
olavloite Dec 31, 2023
10d0721
feat: make benchmark methods configurable
olavloite Jan 1, 2024
684ab08
feat: add runners for JDBC + client lib
olavloite Jan 1, 2024
1a77e0e
feat: add gc stats
olavloite Jan 1, 2024
02d1097
feat: add benchmark runner for generated client
olavloite Jan 2, 2024
ca50c5e
fix: remove virtual threads config
olavloite Jan 2, 2024
02dc49e
perf: randomize pool for high qps
olavloite Jan 6, 2024
d9c0a05
feat: set min/max sessions for connections
olavloite Jan 6, 2024
888f56f
chore: make virtual threads configurable
olavloite Jan 7, 2024
b308003
chore: make everything configurable
olavloite Jan 8, 2024
9fdea45
feat: store results in database
olavloite Jan 8, 2024
21a0ffb
fix: use default session pool
olavloite Jan 8, 2024
b8f6933
feat: use optimization options
olavloite Jan 14, 2024
80bd849
feat: add option for using streaming sql for gapic
olavloite Jan 14, 2024
b6f78de
feat: increase virtual thread pool size
olavloite Jan 14, 2024
139bd46
perf: various optimizations
olavloite Jan 14, 2024
baacb71
feat: make virtual thread factor configurable
olavloite Jan 14, 2024
74c4c52
chore: reduce iterations for select100 tests
olavloite Jan 14, 2024
3d71b18
deps: stop maven from complaining
olavloite Jan 15, 2024
aeabd95
chore: reduce iterations for 100 rows in transaction
olavloite Jan 15, 2024
637fb80
feat: reduce number of rows in transaction
olavloite Jan 15, 2024
2d2c69e
fix: use array to select multiple rows
olavloite Jan 15, 2024
af50ae2
fix: use text as data type
olavloite Jan 15, 2024
1ccc78c
feat: support arrays for Spanner and gapic
olavloite Jan 15, 2024
69a463d
test: try without internal retries
olavloite Jan 15, 2024
2a04b52
chore: turn on checksums
olavloite Jan 15, 2024
47562d7
deps: update spanner deps
olavloite Jan 21, 2024
b2e8dfa
Merge branch 'virtual-threads' of github.com:GoogleCloudPlatform/pgad…
olavloite Jan 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 6 additions & 2 deletions .github/workflows/integration-emulator.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ env:
jobs:
integration-test-emulator:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
java: [11, 21]
# Add a PostgreSQL Docker container and the emulator.
services:
postgres:
Expand Down Expand Up @@ -40,7 +44,7 @@ jobs:
uses: actions/setup-java@v4
with:
distribution: zulu
java-version: 17
java-version: ${{matrix.java}}
- run: java -version
- name: Setup Go
uses: actions/setup-go@v5
Expand All @@ -65,7 +69,7 @@ jobs:
run: |
psql -h /pg -U postgres -c "CREATE DATABASE pgadapter"
- name: Run integration tests
run: mvn verify -B -Dclirr.skip=true -DskipUnits=true -DskipITs=false
run: mvn clean verify -B -Dclirr.skip=true -DskipUnits=true -DskipITs=false
env:
POSTGRES_HOST: /pg
POSTGRES_PORT: 5432
Expand Down
20 changes: 14 additions & 6 deletions .github/workflows/integration.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -76,12 +76,6 @@ jobs:
run: |
echo "EXCLUDED_INTEGRATION_TESTS=SlowTest" >> $GITHUB_ENV
- uses: actions/checkout@v4
- name: Setup Java
uses: actions/setup-java@v4
with:
distribution: zulu
java-version: 17
- run: java -version
- name: Setup Go
uses: actions/setup-go@v5
with:
Expand All @@ -104,6 +98,20 @@ jobs:
- name: Create PostgreSQL test database
run: |
psql -h /pg -U postgres -c "CREATE DATABASE pgadapter"
- name: Setup Java 11
uses: actions/setup-java@v4
with:
distribution: zulu
java-version: 11
- run: java -version
- name: Run unit tests without virtual threads
run: mvn test -B
- name: Setup Java 21
uses: actions/setup-java@v4
with:
distribution: zulu
java-version: 21
- run: java -version
- name: Run unit tests
run: mvn test -B -Ptest-all
- name: Auth
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/units.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
strategy:
fail-fast: false
matrix:
java: [8, 11, 17]
java: [8, 11, 17, 21]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
Expand Down Expand Up @@ -40,7 +40,7 @@ jobs:
strategy:
fail-fast: false
matrix:
java: [8, 11, 17]
java: [8, 11, 17, 21]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
Expand All @@ -56,7 +56,7 @@ jobs:
strategy:
fail-fast: false
matrix:
java: [8, 11, 17]
java: [8, 11, 17, 21]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
Expand Down
1 change: 1 addition & 0 deletions benchmarks/client-comparisons/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
tpcc-log.log
65 changes: 65 additions & 0 deletions benchmarks/client-comparisons/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# TPC-C Benchmark Application for PGAdapter and Cloud Spanner

This application implements the standard TPC-C benchmark and runs it against PGAdapter and a Cloud
Spanner database. It is a Java Spring Boot application. See [src/main/resources/application.properties](src/main/resources/application.properties)
for a full list of configuration options.

## Running

The application automatically starts PGAdapter together with the benchmark application. Supply the
Cloud Spanner database and your credentials using the below command. The `tpcc.benchmark-threads`
argument determines the number of threads that will execute test transactions in parallel.

```shell
mvn spring-boot:run -Dspring-boot.run.arguments="
--tpcc.benchmark-threads=8
--spanner.project=my-project
--spanner.instance=my-instance
--spanner.database=my-database
--pgadapter.credentials=/path/to/credentials.json
"
```


Load data:

```shell
mvn spring-boot:run -Dspring-boot.run.arguments="
--tpcc.benchmark-duration=PT600s
--tpcc.warehouses=10
--tpcc.benchmark-threads=1
--tpcc.load-data=true
--tpcc.truncate-before-load=false
--tpcc.run-benchmark=false
--tpcc.use-read-only-transactions=false
--tpcc.lock-scanned-ranges=false
--spanner.project=appdev-soda-spanner-staging
--spanner.instance=knut-test-ycsb
--spanner.database=tpcc2
--pgadapter.credentials=/home/loite/appdev-soda-spanner-staging.json
--pgadapter.disable-internal-retries=false
"
```


Run benchmark:

```shell
mvn spring-boot:run -Dspring-boot.run.arguments="
--tpcc.benchmark-duration=PT600s
--tpcc.warehouses=10
--tpcc.benchmark-threads=1
--tpcc.load-data=false
--tpcc.truncate-before-load=false
--tpcc.run-benchmark=true
--tpcc.use-read-only-transactions=true
--tpcc.lock-scanned-ranges=false
--spanner.project=appdev-soda-spanner-staging
--spanner.instance=knut-test-ycsb
--spanner.database=tpcc2
--pgadapter.enable-open-telemetry=true
--pgadatper.open-telemetry-sample-ratio=1.0
--pgadapter.credentials=/home/loite/appdev-soda-spanner-staging.json
--pgadapter.disable-internal-retries=true
"
```
85 changes: 85 additions & 0 deletions benchmarks/client-comparisons/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.1.5</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.google.cloud.spanner</groupId>
<artifactId>pgadapter-client-benchmarks</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>PGAdapter Client Benchmarks</name>
<description>Various Benchmarks for PGAdapter and other Cloud Spanner clients</description>
<properties>
<java.version>17</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<!-- Add the PostgreSQL JDBC driver -->
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.7.1</version>
</dependency>
<!-- Add the Cloud Spanner JDBC driver -->
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-spanner-jdbc</artifactId>
<version>2.15.0</version>
</dependency>
<!-- Add PGAdapter as a dependency, so we can start it in-process -->
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-spanner</artifactId>
<version>6.56.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-spanner-pgadapter</artifactId>
<version>0.27.2-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-extension-autoconfigure-spi</artifactId>
<version>1.33.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.14.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>com.spotify.fmt</groupId>
<artifactId>fmt-maven-plugin</artifactId>
<version>2.21.1</version>
<executions>
<execution>
<goals>
<goal>format</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

</project>