Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spanner - Closing session pool is slow for large pools #19

Closed
olavloite opened this issue Aug 11, 2019 · 1 comment · Fixed by #24 or #46
Closed

Spanner - Closing session pool is slow for large pools #19

olavloite opened this issue Aug 11, 2019 · 1 comment · Fixed by #24 or #46
Assignees
Labels
api: spanner type: feature request

Comments

@olavloite
Copy link
Contributor

@olavloite olavloite commented Aug 11, 2019

Calling Spanner#close() can be slow if the underlying session pool contains a large number of sessions. Spanner#close() will close and cleanup all resources that have been used by the instance, and one of these is the session pool. The method will block until all resources have been cleaned up.

The session pool deletes all its sessions when it is closed. This is done asynchronously using a fixed 8-worker thread executor. Each worker thread blocks while waiting for a session to be deleted. This means that closing a session pool with 1,000 sessions will have an execution time of T * 1000 / 8, where T is the total time needed to delete one session.

Session deletion could be further parallelized for large session pools in order to reduce the total execution time of Spanner#close().

@chingor13 chingor13 transferred this issue from googleapis/google-cloud-java Jan 7, 2020
@chingor13 chingor13 added the type: feature request label Jan 7, 2020
olavloite added a commit that referenced this issue Jan 9, 2020
Sessions should be closed using an async gRPC call in order
to speed up the closing of a large session pool. Instead of
using its own executor, which is limited to 8 worker threads,
to execute asynchronous delete session calls, the session pool
should use the asynchronous call option in gRPC. This allows a
larger number of asynchronous delete session calls to be executed
in parallel and speeds up closing a session pool with a large
number of sessions.

Testing against a real Cloud Spanner database with a session pool
containing 1,000 sessions shows the following performance for
closing the session pool:

Before (3 runs):
6603ms
8169ms
8367ms

After (3 runs):
1297ms
1710ms
1851ms

Fixes #19.
olavloite added a commit that referenced this issue Jan 14, 2020
Sessions should be closed using an async gRPC call in order
to speed up the closing of a large session pool. Instead of
using its own executor, which is limited to 8 worker threads,
to execute asynchronous delete session calls, the session pool
should use the asynchronous call option in gRPC. This allows a
larger number of asynchronous delete session calls to be executed
in parallel and speeds up closing a session pool with a large
number of sessions.

Testing against a real Cloud Spanner database with a session pool
containing 1,000 sessions shows the following performance for
closing the session pool:

Before (3 runs):
6603ms
8169ms
8367ms

After (3 runs):
1297ms
1710ms
1851ms

Fixes #19.
@olavloite olavloite self-assigned this Jan 22, 2020
@skuruppu skuruppu closed this in #24 Jan 23, 2020
skuruppu pushed a commit that referenced this issue Jan 23, 2020
* perf: close sessions async

Sessions should be closed using an async gRPC call in order
to speed up the closing of a large session pool. Instead of
using its own executor, which is limited to 8 worker threads,
to execute asynchronous delete session calls, the session pool
should use the asynchronous call option in gRPC. This allows a
larger number of asynchronous delete session calls to be executed
in parallel and speeds up closing a session pool with a large
number of sessions.

Testing against a real Cloud Spanner database with a session pool
containing 1,000 sessions shows the following performance for
closing the session pool:

Before (3 runs):
6603ms
8169ms
8367ms

After (3 runs):
1297ms
1710ms
1851ms

Fixes #19.

* fix: wait for test servers to terminate

* remove tracing for async call

* do not use directExecutor which could be a gRPC thread

* fix: return failed future instead of throwing exception

* fix: remove commented code
@olavloite
Copy link
Contributor Author

@olavloite olavloite commented Jan 23, 2020

Merge was reverted because of problems with the binary compatibility check.

@olavloite olavloite reopened this Jan 23, 2020
@google-cloud-label-sync google-cloud-label-sync bot added the api: spanner label Jan 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner type: feature request
Projects
None yet
2 participants