Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark AssignMutator.Mutate code #1437

Merged
merged 1 commit into from Jul 14, 2021

Conversation

willbeason
Copy link
Member

@willbeason willbeason commented Jul 13, 2021

Add benchmark tests for AssignMutator.Mutate.

In writing this, I found that mutating objects takes about 10x longer
than not mutating objects. Profiling reveals that we spend over 90% of
compute time in AssingMutator.Value(), where we convert from the raw
JSON value to a literal Go struct.

In a separate PR I'll suggest a change that resolves this performance
issue. Most likely we'll want to cache the assigned value. We can then
either reflect.DeepCopy the value or assign the cached value directly,
depending on whether we feel it safe.

Signed-off-by: Will Beason willbeason@google.com

@willbeason
Copy link
Member Author

willbeason commented Jul 13, 2021

Current benchmarks on my machine:

$ go test ./pkg/mutation/mutators -bench=. -run=NONE

goos: linux
goarch: amd64
pkg: github.com/open-policy-agent/gatekeeper/pkg/mutation/mutators
cpu: Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz
BenchmarkAssignMutator_Mutate/always_mutate_1-depth-12         	  370184	      2980 ns/op
BenchmarkAssignMutator_Mutate/always_mutate_2-depth-12         	  354416	      3005 ns/op
BenchmarkAssignMutator_Mutate/always_mutate_5-depth-12         	  384841	      3224 ns/op
BenchmarkAssignMutator_Mutate/always_mutate_10-depth-12        	  301633	      3581 ns/op
BenchmarkAssignMutator_Mutate/always_mutate_20-depth-12        	  271772	      4334 ns/op
BenchmarkAssignMutator_Mutate/never_mutate_1-depth-12          	11371996	       115.7 ns/op
BenchmarkAssignMutator_Mutate/never_mutate_2-depth-12          	 8351156	       161.0 ns/op
BenchmarkAssignMutator_Mutate/never_mutate_5-depth-12          	 5172600	       219.0 ns/op
BenchmarkAssignMutator_Mutate/never_mutate_10-depth-12         	 3369888	       335.2 ns/op
BenchmarkAssignMutator_Mutate/never_mutate_20-depth-12         	 2281078	       500.3 ns/op

@codecov-commenter
Copy link

codecov-commenter commented Jul 13, 2021

Codecov Report

Merging #1437 (73e6b72) into master (9e76b3d) will increase coverage by 0.05%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1437      +/-   ##
==========================================
+ Coverage   50.41%   50.46%   +0.05%     
==========================================
  Files          77       77              
  Lines        5108     5108              
==========================================
+ Hits         2575     2578       +3     
+ Misses       2182     2180       -2     
+ Partials      351      350       -1     
Flag Coverage Δ
unittests 50.46% <ø> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...onstrainttemplate/constrainttemplate_controller.go 54.04% <0.00%> (+0.97%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9e76b3d...73e6b72. Read the comment docs.

Copy link
Contributor

@maxsmythe maxsmythe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

We should deepcopy for safety, though if possible, we should deepcopy once per mutation session.

Add benchmark tests for AssignMutator.Mutate.

In writing this, I found that mutating objects takes about 10x longer
than not mutating objects. Profiling reveals that we spend over 90% of
compute time in AssingMutator.Value(), where we convert from the raw
JSON value to a literal Go struct.

In a separate PR I'll suggest a change that resolves this performance
issue. Most likely we'll want to cache the assigned value. We can then
either reflect.DeepCopy the value or assign the cached value directly,
depending on whether we feel it safe.

Signed-off-by: Will Beason <willbeason@google.com>
@willbeason willbeason merged commit 4c6fae2 into open-policy-agent:master Jul 14, 2021
@willbeason willbeason deleted the benchmark branch July 14, 2021 15:38
julianKatz pushed a commit to julianKatz/gatekeeper that referenced this pull request Jul 27, 2021
Add benchmark tests for AssignMutator.Mutate.

In writing this, I found that mutating objects takes about 10x longer
than not mutating objects. Profiling reveals that we spend over 90% of
compute time in AssingMutator.Value(), where we convert from the raw
JSON value to a literal Go struct.

In a separate PR I'll suggest a change that resolves this performance
issue. Most likely we'll want to cache the assigned value. We can then
either reflect.DeepCopy the value or assign the cached value directly,
depending on whether we feel it safe.

Signed-off-by: Will Beason <willbeason@google.com>
Signed-off-by: juliankatz <juliankatz@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants