Option to set TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES #2103
Conversation
| line: "TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES={{ ceph_tcmalloc_max_total_thread_cache }}" | ||
| when: | ||
| - etc_default_ceph.stat.exists | ||
| - ansible_os_family == 'Debian' |
There was a problem hiding this comment.
It's probably easier if you move the Debian condition above.
| @@ -452,6 +452,9 @@ os_tuning_params: | |||
| - { name: vm.swappiness, value: 10 } | |||
| - { name: vm.min_free_kbytes, value: "{{ vm_min_free_kbytes }}" } | |||
|
|
|||
| # For Debian & Red Hat/CentOS installs set TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES | |||
| #ceph_tcmalloc_max_total_thread_cache: 134217728 | |||
There was a problem hiding this comment.
Please uncomment it otherwise the tasks above will fail.
There was a problem hiding this comment.
d'oh - of course, initially i had wanted to use the package defaults - which arent necessarily 128MB - will fix.
roles/ceph-common/tasks/main.yml
Outdated
| include: configure_memory_allocator.yml | ||
| when: | ||
| - (ceph_origin == 'repository' or ceph_origin == 'distro') | ||
| - ceph_tcmalloc_max_total_thread_cache is defined |
There was a problem hiding this comment.
Also, we need a condition on filestore only osd_objectstore: filestore.
There was a problem hiding this comment.
Does bluestore not use tcmalloc? I thought it did - I havn't messed about with it to confirm - easy enough to fix this, just wanted to confirm.
There was a problem hiding this comment.
Yup, please leave it only for filestore.
| - ansible_os_family == 'Debian' | ||
| notify: | ||
| - restart ceph mons | ||
| - restart ceph osds |
There was a problem hiding this comment.
Let's restart the OSDs only since this is only affecting OSDs
andymcc
force-pushed
the
tcmalloc_settings
branch
from
037d874
to
2cffb0c
Compare
There was a problem hiding this comment.
Could also edit the container template like here: https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-osd/templates/ceph-osd-run.sh.j2#L64 to add -e TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES={{ ceph_tcmalloc_max_total_thread_cache }}. Thanks!
Again only on filestore.
andymcc
force-pushed
the
tcmalloc_settings
branch
from
2cffb0c
to
63b3e1e
Compare
|
Fixed those up. 1 note, I used a default of 0 (which will basically mean it gets skipped) - I'm trying to avoid a situation where existing clusters (on upgrades) run this and have their osds restarted for no reason, or worse they have tuned that value to a different setting and it gets force changed, and osds restarted - happy to change that but I'd rather not cause impact to running/existing clusters without the deployer/operator making a distinct choice (e.g. setting the var to non-default value). |
| @@ -62,6 +62,9 @@ expose_partitions "$1" | |||
| {% endif -%} | |||
| -e CLUSTER={{ cluster }} \ | |||
| -e OSD_DEVICE=/dev/${1} \ | |||
| {% if (ceph_tcmalloc_max_total_thread_cache | int) > 0 -%} | |||
There was a problem hiding this comment.
and osd_objectstore == 'filestore'
Use "ceph_tcmalloc_max_total_thread_cache" to set the TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES value inside /etc/default/ceph for Debian installs, or /etc/sysconfig/ceph for Red Hat/CentOS installs. By default this is set to 0, so the default package value will be used, if specified this value will be changed to match the variable, and ceph osd services will be restarted.
andymcc
force-pushed
the
tcmalloc_settings
branch
from
63b3e1e
to
7f6c391
Compare
|
one of these days ;) |
Use "ceph_tcmalloc_max_total_thread_cache" to set the
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES value inside /etc/default/ceph for
Debian installs, or /etc/sysconfig/ceph for Red Hat/CentOS installs.
By default this is not set, so the default package value will be used,
if specified this value will be changed to match the variable, and ceph
services will be restarted.