Lock Contention

[maxsmythe]

What steps did you take and what happened:

When there is a config that syncs large numbers of objects with frequent updates, it can cause performance issues due to lock contention with the validation webhook.

We should find the lock contention (perhaps sync acquires a write lock and validation a read lock) and resolve it.

Here is where the write lock appears to be acquired:

https://github.com/open-policy-agent/frameworks/blob/9db6a6b27c9741bcf9c5f4d8c6e3ef230b49f41d/constraint/pkg/client/drivers/local/storages.go#L116-L123

Without changing how the storage backend works, perhaps the best solution is to throttle the rate at which write locks are acquired.

What did you expect to happen:

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Gatekeeper version:
• Kubernetes version: (use kubectl version):

[willbeason]

Added a feature request to OPA. The three line change I benchmarked reduced time to add data to OPA storage by >99%.

[willbeason]

open-policy-agent/opa#4709 is out for review.

We'll probably want to deepcopy the object before passing it to OPA storage, just to be safe, so for Gatekeeper the CPU time reduction will be 90%, but the time the transaction holds on the lock will be reduced 99%. This should make our max throughput (as far as the lock is concerned) about 100x higher if AddData is called in parallel.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[maxsmythe]

This still needs to be integrated. It's a scaling bottleneck for syncing high-throughput kinds