Supervised systemd option not working with keydb 6

[kevinvalk]

Describe the bug
After updating my keydb installation to 6.0.13 (running on Debian), systemd keeps rebooting my keydb instances. After some investigation, systemd seems not to receive notification events anymore. Please note, I am running keydb with Type=notify in systemd and supervised systemd.

It may be "simply" due to the issue described here redis/redis#7217 (see also redis/redis@129d14e). Which would be resolved by installing pkg-config and libsystemd-dev package on the build server.

Log Files
The log is just fine and keeps doing this endlessly because of systemd keeps rebooting

5674:5674:M 14 Aug 2020 01:43:01.713 * RDB age 0 seconds
5674:5674:M 14 Aug 2020 01:43:01.713 * RDB memory usage when created 8.66 Mb
5674:5674:M 14 Aug 2020 01:43:01.713 * DB loaded from disk: 0.000 seconds
5674:5679:M 14 Aug 2020 01:43:01.714   Thread 0 alive.
5674:signal-handler (1597362361) Received SIGTERM scheduling shutdown...
5674:5679:M 14 Aug 2020 01:46:01.833 # User requested shutdown...
5674:5679:M 14 Aug 2020 01:46:01.833 * Saving the final RDB snapshot before exiting.
5674:5679:M 14 Aug 2020 01:46:01.845 * DB saved on disk
5674:5679:M 14 Aug 2020 01:46:01.845 * Removing the pid file.
5674:5679:M 14 Aug 2020 01:46:01.845 # KeyDB is now ready to exit, bye bye...
5715:5715:C 14 Aug 2020 01:46:02.198 # oO0OoO0OoO0Oo KeyDB is starting oO0OoO0OoO0Oo
5715:5715:C 14 Aug 2020 01:46:02.198 # KeyDB version=6.0.13, bits=64, commit=00000000, modified=0, pid=5715, just started
5715:5715:C 14 Aug 2020 01:46:02.198 # Configuration loaded
5715:5715:C 14 Aug 2020 01:46:02.198 # WARNING: server-threads is greater than this machine's core count.  Truncating to 1 threads
5715:5715:C 14 Aug 2020 01:46:02.199 # WARNING supervised by systemd - you MUST set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.


                                        KeyDB 6.0.13 (00000000/0) 64 bit

                                        Running in standalone mode
                                        Port: 6380
                                        PID: 5715

                     Join the KeyDB community! https://community.keydb.dev/



5715:5715:M 14 Aug 2020 01:46:02.205 # Server initialized
5715:5715:M 14 Aug 2020 01:46:02.205 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with KeyDB. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. KeyDB must be restarted after THP is disabled.
5715:5715:M 14 Aug 2020 01:46:02.206 * Loading RDB produced by version 6.0.13
5715:5715:M 14 Aug 2020 01:46:02.206 * RDB age 1 seconds
5715:5715:M 14 Aug 2020 01:46:02.206 * RDB memory usage when created 8.66 Mb
5715:5715:M 14 Aug 2020 01:46:02.206 * DB loaded from disk: 0.000 seconds
5715:5720:M 14 Aug 2020 01:46:02.206   Thread 0 alive.

To Reproduce
Running on

Linux mail01 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux
KeyDB server v=6.0.13 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=5055ac929b8ff5fb

Systemd unit file

[Unit]
Description=Advanced key-value store
After=network.target
Documentation=http://keydb.io/documentation, man:keydb-server(1)

[Service]
Type=notify
ExecStart=/usr/bin/keydb-server /etc/keydb/keydb-test.conf --supervised systemd --daemonize no
ExecStop=/bin/kill -s TERM $MAINPID
PIDFile=/run/keydb/keydb-test.pid
TimeoutStartSec=180
TimeoutStopSec=30
Restart=always
User=keydb
Group=keydb
RuntimeDirectory=keydb
RuntimeDirectoryMode=2755

UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/keydb/keydb-test
ReadWriteDirectories=-/var/log/keydb
ReadWriteDirectories=-/var/run/keydb

NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
#MemoryDenyWriteExecute=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# keydb-server can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
ProtectSystem=true
ReadWriteDirectories=-/etc/keydb

[Install]
WantedBy=multi-user.target
Alias=keydb-test.service

/etc/keydb/keydb-test.conf

include /etc/keydb/keydb.conf

## Network
port 6380
# unixsocket /var/run/keydb/keydb-test.sock
# unixsocketperm 700


## Database settings
dbfilename test.rdb
maxmemory 209715200
dir /var/lib/keydb/keydb-test


## Daemon
daemonize no
supervised systemd
loglevel notice
pidfile /run/keydb/keydb-test.pid
logfile /var/log/keydb/keydb-server-test.log

After running service keydb-test start, the command hangs and finally times out. The keydb-test server is actually during his time running, but systemd is not aware and will restart it after the default timeout.

● keydb-test.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/keydb-test.service; enabled; vendor preset: enabled)
   Active: activating (start) since Fri 2020-08-14 01:43:01 CEST; 1min 19s ago
     Docs: http://keydb.io/documentation,
           man:keydb-server(1)
 Main PID: 5674 (keydb-server)
    Tasks: 6 (limit: 1148)
   Memory: 13.5M
   CGroup: /system.slice/keydb-test.service
           └─5674 /usr/bin/keydb-server 127.0.0.1:6380

Aug 14 01:43:01 mail01 systemd[1]: Starting Advanced key-value store...

[kevinvalk]

Please note that all is well with the exact same configuration, system, etc when using forking mode without supervised.

Type=forking
ExecStart=/usr/bin/keydb-server /etc/keydb/keydb-test.conf --supervised no --daemonize yes

Hence I am really thinking the issue is related to redis issue redis/redis#7217

[JohnSully]

Did you compile KeyDB yourself or use a docker image? I think its that we're not building with systemd as part of our docker images.

[JohnSully]

Assigned to ben to compile with USE_SYSTEMD=yes

[kevinvalk]

Did you compile KeyDB yourself or use a docker image? I think its that we're not building with systemd as part of our docker images.

I am using the "official" PPA from https://docs.keydb.dev/docs/ppa-deb/ for Debian buster

[willbrowningme]

Having the same issue here with KeyDB server v=6.0.16 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=fa6f9f5195421066 from the official PPA on Ubuntu 20.04.

Tried the steps listed in redis/redis#7217 but doesn't fix it unfortunately.

It's strange because I'm running the exact same config setting and systemd service file with Redis server v=6.0.10 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=48afb8ef69256162 and it runs perfectly.

Here's my /etc/systemd/system/keydb_7000.service file:

[Unit]
Description=Advanced key-value store
After=network.target
Documentation=https://docs.keydb.dev/, man:keydb-server(1)

[Service]
Type=notify
ExecStart=/usr/bin/keydb-server /etc/keydb/cluster/7000/keydb_7000.conf --supervised systemd --daemonize no
ExecStop=/usr/bin/keydb-cli -h 127.0.0.1 -p 7000 shutdown
PIDFile=/run/keydb/keydb-server_7000.pid
TimeoutStartSec=180
TimeoutStopSec=30
Restart=always
User=keydb
Group=keydb
RuntimeDirectory=keydb
RuntimeDirectoryMode=2755

UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/keydb
ReadWriteDirectories=-/var/log/keydb
ReadWriteDirectories=-/var/run/keydb

NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
#MemoryDenyWriteExecute=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true

# keydb-server can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
ProtectSystem=true
ReadWriteDirectories=-/etc/keydb

[Install]
WantedBy=multi-user.target
Alias=keydb_7000.service

It just times out when trying to run sudo systemctl start keydb_7000.service.

Was looking at migrating to KeyDB after reading the blog post on TLS performance.

[willbrowningme]

Changing the Type to simple gets it running as expected so it must be an issue with the $NOTIFY_SOCKET.

[Service]
Type=simple

[VivekSainiEQ]

Hi all,

We weren't passing in the right flags for SYSTEMD support, the fix is coming in the next release.