OPA should automatically set GOMAXPROCS if needed #3328
Projects
Comments
tsandall
added this to TODO (Things That Should Be Done)
in Open Policy Agent
via automation
tsandall
moved this from TODO (Things That Should Be Done)
to Planned (Things We Are Going To Do)
in Open Policy Agent
tsandall
added a commit
to tsandall/opa
that referenced
this issue
Mar 30, 2021
This commit updates the OPA runtime to set the GOMAXPROCS environment variable if it was not previously set. The value is determined by looking at the cgroup CPU quota and determining how many CPUs that equals. This is important if OPA is running on hosts where the number of CPUs that the Go runtime would normally see is high compared to the CPU quota applied to the OPA process/container. If the ratio is high, then the Go runtime can't schedule the goroutines effectively and the process ends up getting throttled which can impact latency significantly. The results are quite promising. For example, this is the result from running the `vegeta` load test tool against v0.27.1 (using a sample HTTP API authorization policy and console decision logging enabled) w/ a CPU quota of "1.0". $ cat traffic.txt | vegeta attack --duration=60s --rate=500/1s | tee results.bin | vegeta report Requests [total, rate, throughput] 30000, 500.02, 500.01 Duration [total, attack, wait] 59.999207s, 59.9980205s, 1.1865ms Latencies [mean, 50, 95, 99, max] 18.860579ms, 1.104143ms, 123.456696ms, 222.282491ms, 393.8184ms Bytes In [total, mean] 2040000, 68.00 Bytes Out [total, mean] 28530000, 951.00 Success [ratio] 100.00% Status Codes [code:count] 200:30000 Error Set: And the same for this commit: $ cat traffic.txt | vegeta attack --duration=60s --rate=500/1s | tee results.bin | vegeta report Requests [total, rate, throughput] 30000, 500.02, 500.01 Duration [total, attack, wait] 59.9990695s, 59.9980199s, 1.0496ms Latencies [mean, 50, 95, 99, max] 1.818325ms, 1.038192ms, 4.078345ms, 13.789254ms, 219.8218ms Bytes In [total, mean] 2040000, 68.00 Bytes Out [total, mean] 28530000, 951.00 Success [ratio] 100.00% Status Codes [code:count] 200:30000 Error Set: Fixes open-policy-agent#3328 Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Open Policy Agent
automation
moved this from Planned (Things We Are Going To Do)
to Done
Mar 30, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
If the OPA container has CPU limits applied to it then the GOMAXPROCS should be set to ensure that the Go runtime can schedule goroutines properly. If GOMAXPROCS is unset, then the Go runtime will see ALL cores on the machine where the container is running (regardless of whether CPU limits/cgroups have been set.) This leads to throttling which hurts performance.
We should make this change inside the
github.com/open-policy-agent/opa/runtimepackage. When the runtime starts:This kind of behaviour is implemented in https://github.com/uber-go/automaxprocs. I'm not sure if we want to just vendor that or reimplement ourselves.
The text was updated successfully, but these errors were encountered: