Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sdn: OVS pods require more memory #8166

Merged
merged 1 commit into from May 1, 2018
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
sdn: fix OOM issues with ovs-vswitchd on many-core machines
By default ovs-vswitchd creates #cores+1 pthreads, which each get
the RHEL/Fedora default 8MB of stack.  But for whatever reason,
ovs-vswitchd immediately dirties this region resulting in all 8MB
counted to RSS, leading to excessive memory consumption.

Work around that in two ways:

1) increase the OVS container memory limits
2) limit the number of threads ovs-vswitchd uses on machines
with more than 12 cores

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1571379
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1572797
  • Loading branch information
dcbw committed Apr 28, 2018
commit 6d9ad9d1ac4c95ea38a8b1aa7d94ac698724c755
13 changes: 11 additions & 2 deletions roles/openshift_sdn/files/sdn-ovs.yaml
Expand Up @@ -63,6 +63,15 @@ spec:
}
trap quit SIGTERM
/usr/share/openvswitch/scripts/ovs-ctl start --system-id=random

# Restrict the number of pthreads ovs-vswitchd creates to reduce the
# amount of RSS it uses on hosts with many cores
# https://bugzilla.redhat.com/show_bug.cgi?id=1571379
# https://bugzilla.redhat.com/show_bug.cgi?id=1572797
if [[ `nproc` -gt 12 ]]; then
ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=4
ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=10
fi
while true; do sleep 5; done
securityContext:
runAsUser: 0
Expand All @@ -83,10 +92,10 @@ spec:
resources:
requests:
cpu: 100m
memory: 200Mi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why bump request? What's the floor for a 10 node 100 pod cluster?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

58MB in my tests

memory: 300Mi
limits:
cpu: 200m
memory: 300Mi
memory: 400Mi

volumes:
- name: host-modules
Expand Down