[jemalloc] Invalid conf pair: lg_dirty_mult:8 + Invalid conf pair: lg_chunk:18

[florianmagnin]

Hello,

On the latest Varnish release on Debian Buster, it seem's we cannot tweak anymore libjemalloc. So we have many leak of memory and OOM KILL.

Error:

Aug 11 15:05:02 XXX systemd1: Starting Varnish Cache, a high-performance HTTP accelerator...
Aug 11 15:05:02 XXX varnishd1866: <jemalloc>: Invalid conf pair: lg_dirty_mult:8
Aug 11 15:05:02 XXX varnishd1866: <jemalloc>: Invalid conf pair: lg_chunk:18
Aug 11 15:05:02 XXX varnishd1866: Warnings:
Aug 11 15:05:02 XXX varnishd1866: VCL compiled.
Aug 11 15:05:02 XXX varnishd1866: Debug: Version: varnish-6.4.0 revision 13f137934ec1cf14af66baf7896311115ee35598
Aug 11 15:05:02 XXX varnishd1866: Debug: Platform: Linux,4.19.0-10-cloud-amd64,x86_64,-junix,-smalloc,-sfile,-sdefault,-hcritbit
Aug 11 15:05:02 XXX varnishd1867: Version: varnish-6.4.0 revision 13f137934ec1cf14af66baf7896311115ee35598
Aug 11 15:05:02 XXX varnishd1867: Platform: Linux,4.19.0-10-cloud-amd64,x86_64,-junix,-smalloc,-sfile,-sdefault,-hcritbit
Aug 11 15:05:02 XXXvarnishd1866: Debug: Child (1880) Started
Aug 11 15:05:02 XXX varnishd1867: Child (1880) Started
Aug 11 15:05:03 XXX varnishd1867: Child (1880) said Child starts
Aug 11 15:05:03 XXX varnishd1867: Child (1880) said SMF.disk mmap'ed 75161927680 bytes of 75161927680
Aug 11 15:05:03 XXX systemd1: Started Varnish Cache, a high-performance HTTP accelerator.

Os Environment :

# uname -a
Linux XXX 4.19.0-10-cloud-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux

# aptitude show varnish
Package: varnish
Version: 6.4.0-1~buster
State: installed
Automatically installed: no
Priority: optional
Section: web
Maintainer: Varnish Software <opensource@varnish-software.com>
Architecture: amd64
Uncompressed Size: 14.6 M
Depends: libc6 (>= 2.14), libedit2 (>= 2.11-20080614-0), libjemalloc2 (>= 2.1.1), libncursesw6 (>= 6), libpcre3, libtinfo6 (>= 6), adduser, gcc, libc6-dev | libc6.1-dev | libc-dev
Suggests: varnish-dev
Conflicts: libvarnishapi1, varnish-dbg, varnish-doc
Replaces: libvarnishapi1, varnish-dbg, varnish-doc
Provides: libvarnishapi1, varnishd-abi-13f137934ec1cf14af66baf7896311115ee35598, varnishd-vrt (= 11.0)
Description: state of the art, high-performance web accelerator
Varnish Cache is a state of the art web accelerator written with performance and flexibility in mind. 
 
 Varnish Cache stores web pages in memory so web servers don't have to create the same web page over and over again. Varnish serves pages much faster than any application server; giving the website a significant speed up. 
 
 Some of the features include: 
 * A modern design 
 * VCL - a very flexible configuration language 
 * Load balancing with health checking of backends 
 * Partial support for ESI - Edge Side Includes 
 * URL rewriting 
 * Graceful handling of "dead" backends
Homepage: https://www.varnish-cache.org/
Tags: implemented-in::c, interface::daemon, network::server, protocol::http, role::program, uitoolkit::ncurses, use::proxying

# varnishd -V
varnishd (varnish-6.4.0 revision 13f137934ec1cf14af66baf7896311115ee35598)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2020 Varnish Software AS

# dpkg -l|grep -i malloc
ii libjemalloc2:amd64 5.1.0-3 amd64 general-purpose scalable concurrent malloc(3) implementation

# cat /etc/default/varnish
    START=yes
    RELOAD_VCL=1
    NFILES=131072
    MEMLOCK=82000
    #VARNISH_LISTEN_ADDRESS=
    VARNISH_LISTEN_PORT="80"
    VARNISH_ADMIN_LISTEN_ADDRESS="127.0.0.1"
    VARNISH_ADMIN_LISTEN_PORT="6082"
    VARNISH_VCL_CONF="/etc/varnish/XXX.vcl"
    VARNISH_SECRET_FILE=/etc/varnish/secret
    VARNISH_STORAGE="ram=malloc,57344m -s disk=file,/mnt/instance_store/varnishdb,71680m"
    MALLOC_CONF="lg_dirty_mult:8,lg_chunk:18"
    DAEMON_OPTS_EXTRA=""
    DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
    -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
    -f ${VARNISH_VCL_CONF} \
    -S ${VARNISH_SECRET_FILE} \
    -s ${VARNISH_STORAGE} \
    ${DAEMON_OPTS_EXTRA}"

Any ideas about that ?

Regards,
Florian

[bsdphk]

It looks like you are now using a newer jemalloc which no longer accepts those tweaks, that is not really a varnish problem though.

Suggest you try to find out first if you still need to tweak the new jemalloc and second what the appropriate paramters are now called.