New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

workload: add pgx-based SQLRunner, use it in KV #31358

Merged
merged 1 commit into from Oct 15, 2018

Conversation

Projects
None yet
5 participants
@RaduBerinde
Member

RaduBerinde commented Oct 15, 2018

SQLRunner is a common facility for issuing SQL queries through pgx.
It supports multiple issuing methods, specified by the --method
flag:

  • prepare: each statement is prepared once per session and then
    reused.
  • noprepare: each statement is executed directly (which internally
    is equivalent to preparing and executing each statement
    separately).
  • simple: parameters are formatted as text and replaced in the query
    before it is issued.

KV is switched to use the new runner. The default prepare method
should be similar to the old code (except that we go through pgx).

Benchmark numbers (ops/s for kv80, single node GCE worker):

  • before: 30990 (541 stddev)
  • prepare (default): 31478 (331 stddev)
  • noprepare: 25506 (530 stddev)
  • simple: 29187 (599 stddev)

Release note: None

@cockroach-teamcity

This comment has been minimized.

Show comment
Hide comment
@cockroach-teamcity

cockroach-teamcity Oct 15, 2018

Member

This change is Reviewable

Member

cockroach-teamcity commented Oct 15, 2018

This change is Reviewable

@RaduBerinde

This comment has been minimized.

Show comment
Hide comment
@RaduBerinde

RaduBerinde Oct 15, 2018

Member

Full benchmark results (including a comparison with a gosql version of SQLRunner): https://docs.google.com/spreadsheets/d/1I028xgdHM9nqydMG3KblfzvZlXGfLRBXrC3xeqiilG8/edit?usp=sharing

Member

RaduBerinde commented Oct 15, 2018

Full benchmark results (including a comparison with a gosql version of SQLRunner): https://docs.google.com/spreadsheets/d/1I028xgdHM9nqydMG3KblfzvZlXGfLRBXrC3xeqiilG8/edit?usp=sharing

@danhhz

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained


pkg/workload/pgx_helpers.go, line 83 at r1 (raw file):

// Get returns one of the pools, in round-robin manner.
func (m *MultiConnPool) Get() *pgx.ConnPool {

Have you thought about how this will play with the partition affinity in tpcc? Keep the existing logic but with one MultiConnPool per url perhaps?

@RaduBerinde

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained


pkg/workload/pgx_helpers.go, line 83 at r1 (raw file):

Previously, danhhz (Daniel Harrison) wrote…

Have you thought about how this will play with the partition affinity in tpcc? Keep the existing logic but with one MultiConnPool per url perhaps?

Yeah, that should work fine.

@danhhz

:lgtm:

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale)

@nvanbenschoten

:lgtm:

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 2 stale)

workload: add pgx-based SQLRunner, use it in KV
SQLRunner is a common facility for issuing SQL queries through `pgx`.
It supports multiple issuing methods, specified by the `--method`
flag:
 - prepare: each statement is prepared once per session and then
   reused.
 - noprepare: each statement is executed directly (which internally
   is equivalent to preparing and executing each statement
   separately).
 - simple: parameters are formatted as text and replaced in the query
   before it is issued.

KV is switched to use the new runner. The default `prepare` method
should be similar to the old code (except that we go through `pgx`).

Benchmark numbers (ops/s for kv80, single node GCE worker):
  - before:            30990 (541 stddev)
  - prepare (default): 31478 (331 stddev)
  - noprepare:         25506 (530 stddev)
  - simple:            29187 (599 stddev)

Release note: None
@RaduBerinde

This comment has been minimized.

Show comment
Hide comment
@RaduBerinde

RaduBerinde Oct 15, 2018

Member

bors r+

Member

RaduBerinde commented Oct 15, 2018

bors r+

craig bot pushed a commit that referenced this pull request Oct 15, 2018

Merge #31358
31358: workload: add pgx-based SQLRunner, use it in KV r=RaduBerinde a=RaduBerinde

SQLRunner is a common facility for issuing SQL queries through `pgx`.
It supports multiple issuing methods, specified by the `--method`
flag:
 - prepare: each statement is prepared once per session and then
   reused.
 - noprepare: each statement is executed directly (which internally
   is equivalent to preparing and executing each statement
   separately).
 - simple: parameters are formatted as text and replaced in the query
   before it is issued.

KV is switched to use the new runner. The default `prepare` method
should be similar to the old code (except that we go through `pgx`).

Benchmark numbers (ops/s for kv80, single node GCE worker):
  - before:            30990 (541 stddev)
  - prepare (default): 31478 (331 stddev)
  - noprepare:         25506 (530 stddev)
  - simple:            29187 (599 stddev)

Release note: None

Co-authored-by: Radu Berinde <radu@cockroachlabs.com>
@craig

This comment has been minimized.

Show comment
Hide comment
@craig

craig bot commented Oct 15, 2018

Build succeeded

@craig craig bot merged commit 1abda3b into cockroachdb:master Oct 15, 2018

3 checks passed

GitHub CI (Cockroach) TeamCity build finished
Details
bors Build succeeded
Details
license/cla Contributor License Agreement is signed.
Details

@RaduBerinde RaduBerinde deleted the RaduBerinde:sqlrunner-kv-pgx branch Oct 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment