Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[YSQL] Reduce per-connection overhead for cache preloading #1091

Closed
m-iancu opened this issue Mar 29, 2019 · 2 comments
Closed

[YSQL] Reduce per-connection overhead for cache preloading #1091

m-iancu opened this issue Mar 29, 2019 · 2 comments
Assignees
@m-iancu
Labels
area/ysql Yugabyte SQL (YSQL) kind/enhancement This is an enhancement of an existing feature

Comments

@m-iancu
Copy link
Contributor

m-iancu commented Mar 29, 2019 •
edited
Loading

Currently, running sample apps with over 100 connections in a 3-node cluster is bottlenecked by all connections hitting the yb-master to preload their local caches.
Same issue can be observed on a local cluster when using enough connections.

To replicate:

  1. Start cluster
./bin/yb-ctl destroy; ./bin/yb-ctl create
  1. Run sample apps with large number of threads e.g. 99 (should see no ops for quite a while)
java -jar ./target/yb-sample-apps.jar --workload SqlInserts --nodes 127.0.0.1:5433 --num_threads_write 99 --num_threads_read 0 --num_unique_keys 1000000000

0 [main] INFO com.yugabyte.sample.Main  - Starting sample app...
26 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Using a randomly generated UUID : 4d852c5d-af09-44cf-b718-af6b834169d7
37 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - App: SqlInserts
37 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Run time (seconds): -1
37 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Adding node: 127.0.0.1:5433
38 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Num reader threads: 0, num writer threads: 99
38 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Num unique keys to insert: 1000000000
38 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Num keys to update: -1000000001
38 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Num keys to read: -1
38 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Value size: 0
38 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Restrict values to ASCII strings: false
38 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Perform sanity check at end of app run: false
38 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Table TTL (secs): -1
38 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Local reads: false
38 [main] INFO com.yugabyte.sample.common.CmdLineOpts  - Read only load: false
1088 [main] INFO com.yugabyte.sample.apps.SqlInserts  - Dropping any table(s) left from previous runs if any
5245 [main] INFO com.yugabyte.sample.apps.SqlInserts  - Created table: postgresqlkeyvalue
10251 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Uptime: 5006 ms |
15253 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Uptime: 10008 ms |
20257 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Uptime: 15012 ms |
25260 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Uptime: 20015 ms |
@m-iancu m-iancu self-assigned this Mar 29, 2019
yugabyte-ci pushed a commit that referenced this issue Apr 2, 2019
@m-iancu
[YSQL] #1090 #1091 Reduce per-connection overhead for cache preloading
465b6d6 
Summary:
- Use the relcache file to store relation cache data, add ysql catalog version to
  detect (and clean up) old (no longer valid) files.
- No longer preload catcache (only relcache) for backend connections except during
  initdb which makes heavy use of various relations and builtins.
- No longer increment catalog version for inserts, only for updates/deletes. Since we don't do
  negative caching, inserts (e.g. from CREATE TABLE) do not invalidate the cache, only
  updates/deletes (e.g. ALTER or DROP TABLE).
- Fix a cache refrence leak bug and clean up the logic for retry and cache-refresh due to
  catalog version mismatches.

Test Plan:
jenkins, TestPgCacheConsistency, sample apps with high number of connections:
java -jar ./target/yb-sample-apps.jar --workload SqlInserts --nodes 127.0.0.1:5433 --num_threads_write 99 --num_threads_read 0 --num_unique_keys 1000000000

Reviewers: mikhail, robert

Reviewed By: robert

Subscribers: yql, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D6412
@m-iancu
Copy link
Contributor Author

m-iancu commented Apr 2, 2019

Closed in 465b6d6.

@m-iancu m-iancu closed this as completed Apr 2, 2019
@kmuthukk
Copy link
Collaborator

kmuthukk commented Apr 2, 2019

nice work @m-iancu !

@m-iancu m-iancu mentioned this issue Apr 12, 2019
Closed
@kmuthukk kmuthukk added kind/enhancement This is an enhancement of an existing feature area/ysql Yugabyte SQL (YSQL) labels Apr 13, 2019
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@m-iancu m-iancu

Labels
area/ysql Yugabyte SQL (YSQL) kind/enhancement This is an enhancement of an existing feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@m-iancu @kmuthukk