Skip to content

Conversation

@benluddy
Copy link
Contributor

The RPC api.Registry/GetBundleForChannel is consumed only by
package-server and by declarative index conversion. Neither reads any
field other than CsvName and CsvJson. The package-server generates one
call to this RPC per channel per package per catalog and is
responsible for needlessly large CPU utilization and resident set
sizes on registry pods. Removing the additional per-request processing
and database queries that populate unused fields makes the registry
server better able to absorb the bursty load from package-server.

Before:

$ ghz -d '{"pkgName":"mongodb-enterprise","channelName":"stable"}' --insecure --concurrency=50 --duration=10s --call=api.Registry/GetBundleForChannel localhost:50051; top -b -n1 | grep opm

Summary:
  Count:	14677
  Total:	10.00 s
  Slowest:	164.41 ms
  Fastest:	2.81 ms
  Average:	34.00 ms
  Requests/sec:	1467.34

Response time histogram:
  2.808   [1]    |
  18.968  [3046] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  35.128  [5288] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  51.289  [3780] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  67.449  [1771] |∎∎∎∎∎∎∎∎∎∎∎∎∎
  83.609  [553]  |∎∎∎∎
  99.769  [143]  |∎
  115.929 [38]   |
  132.089 [5]    |
  148.249 [1]    |
  164.409 [2]    |

Latency distribution:
  10 % in 9.80 ms 
  25 % in 21.44 ms 
  50 % in 31.29 ms 
  75 % in 45.75 ms 
  90 % in 59.09 ms 
  95 % in 67.55 ms 
  99 % in 86.49 ms 

Status code distribution:
  [OK]            14628 responses   
  [Unavailable]   49 responses      

Error distribution:
  [49]   rpc error: code = Unavailable desc = transport is closing   

 518843 user  20   0 6576700 128252  30376 S  12.5   0.4   2:56.35 opm

After:

$ ghz -d '{"pkgName":"mongodb-enterprise","channelName":"stable"}' --insecure --concurrency=50 --duration=10s --call=api.Registry/GetBundleForChannel localhost:50051; top -b -n1 | grep opm

Summary:
  Count:	80470
  Total:	10.01 s
  Slowest:	47.58 ms
  Fastest:	0.18 ms
  Average:	6.16 ms
  Requests/sec:	8040.87

Response time histogram:
  0.185  [1]     |
  4.924  [54268] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  9.663  [3217]  |∎∎
  14.402 [4495]  |∎∎∎
  19.141 [10180] |∎∎∎∎∎∎∎∎
  23.880 [6790]  |∎∎∎∎∎
  28.619 [1223]  |∎
  33.358 [198]   |
  38.097 [37]    |
  42.836 [11]    |
  47.575 [1]     |

Latency distribution:
  10 % in 0.33 ms 
  25 % in 0.49 ms 
  50 % in 1.31 ms 
  75 % in 13.15 ms 
  90 % in 19.23 ms 
  95 % in 21.35 ms 
  99 % in 25.41 ms 

Status code distribution:
  [OK]            80421 responses   
  [Unavailable]   49 responses      

Error distribution:
  [49]   rpc error: code = Unavailable desc = transport is closing   

 518256 user  20   0 5986076  82300  30704 S   6.2   0.3   2:52.65 opm

The RPC api.Registry/GetBundleForChannel is consumed only by
package-server and by declarative index conversion. Neither reads any
field other than CsvName and CsvJson. The package-server generates one
call to this RPC per channel per package per catalog and is
responsible for needlessly large CPU utilization and resident set
sizes on registry pods. Removing the additional per-request processing
and database queries that populate unused fields makes the registry
server better able to absorb the bursty load from package-server.

Signed-off-by: Ben Luddy <bluddy@redhat.com>
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 31, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: benluddy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@benluddy benluddy requested a review from kevinrizza August 31, 2021 20:49
@openshift-ci openshift-ci bot requested review from ecordell and joelanford August 31, 2021 20:49
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 31, 2021
@codecov
Copy link

codecov bot commented Aug 31, 2021

Codecov Report

Merging #769 (021ba4b) into master (9bf77cf) will decrease coverage by 0.03%.
The diff coverage is 86.66%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #769      +/-   ##
==========================================
- Coverage   50.41%   50.37%   -0.04%     
==========================================
  Files         102      102              
  Lines        8725     8702      -23     
==========================================
- Hits         4399     4384      -15     
+ Misses       3473     3469       -4     
+ Partials      853      849       -4     
Impacted Files Coverage Δ
pkg/sqlite/query.go 64.61% <86.66%> (-0.02%) ⬇️
pkg/lib/bundle/supported_resources.go 100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9bf77cf...021ba4b. Read the comment docs.

SELECT operatorbundle.name, operatorbundle.csv FROM operatorbundle INNER JOIN channel
ON channel.head_operatorbundle_name = operatorbundle.name
WHERE channel.name = :channel AND channel.package_name = :package`
rows, err := s.db.QueryContext(ctx, query, sql.Named("channel", channel), sql.Named("package", pkg))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL sql.Named is a thing 👍

@kevinrizza
Copy link
Member

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 1, 2021
@openshift-merge-robot openshift-merge-robot merged commit 86907e1 into operator-framework:master Sep 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants