Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

harden redis and sync service. #712

Merged
merged 2 commits into from Mar 18, 2020
Merged

harden redis and sync service. #712

merged 2 commits into from Mar 18, 2020

Conversation

@raulk
Copy link
Member

raulk commented Mar 18, 2020

Fixes #708.

  • increase default sync service redis pool size from runtime.NumCPU()*10 (default) to 8192 to prevent starvation.
    • Starvation is very probable in the sidecar, given that we use the XREAD primitive, which blocks a connection.
    • The sidecar is especially exposed to this, as it is a single process that accumulates a lot of XREAD blocked conns to receive network change requests from instances.
    • 8192 is a suitable value because on local:docker, we won't run as many containers, and on cluster:k8s, sidecars only handle containers colocated on the node.
  • increase ulimits on infra containers to 1M open file descriptors.
  • expose pprof port of sidecar; enable GC traces for debuggability.
  • upgrade to go-redis v7.2.0.

additionally: harden throttling logic in local:docker runner.

* increase default sync service redis pool size from runtime.NumCPU()*10 to 8192 to
  prevent starvation. Starvation is very probable given that we use the XREAD primitive,
  which blocks a connection. The sidecar is especially exposed to this, as it is a
  single process that accumulates a lot of XREAD blocked conns to receive network
  change requests from instances. 8192 is a suitable value because on local:docker, we
  won't run as many containers, and on cluster:k8s, sidecars only handle containers
  colocated on the node.
* increase ulimits on infra containers to 1M open file descriptors.
* expose pprof port of sidecar; enable GC traces for debuggability.
* upgrade to go-redis v7.2.0.

additionally: harden throttling logic in local:docker runner.
@raulk raulk added this to the Testground v0.3 milestone Mar 18, 2020
@raulk raulk requested review from Stebalien and coryschwartz Mar 18, 2020
@raulk

This comment has been minimized.

Copy link
Member Author

raulk commented Mar 18, 2020

@aschmahmann @jacobheun This should fix all your woes with the local:docker runner, the sync service, and the sidecar.

Copy link
Collaborator

coryschwartz left a comment

I haven't run this, bu this looks fine as far as I can tell, once other comments are addressed.

@raulk raulk merged commit 9a98279 into master Mar 18, 2020
0 of 2 checks passed
0 of 2 checks passed
Travis CI - Branch Build Failed
Details
Travis CI - Pull Request Build Failed
Details
@raulk raulk deleted the feat/harden-redis branch Mar 18, 2020
@Robmat05 Robmat05 added status/done and removed status/waiting labels Mar 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

4 participants
You can’t perform that action at this time.