[BUG] keydb not responding to SIGTERM during startup

[h0tw1r3]

Describe the bug

After issuing a service start, the systemd service start job never completes if certain conditions.
Depending on the issue, or how the service is configured (type=notify vs type=forking) it may not respond to service stop either.

To reproduce

Create a single node, configured as active-replica. Set the replicaof to a non-existent server to replicate a broken setup.

Start service: systemctl start keydb-server
Observe: systemctl list-jobs

You should notice a job is "stuck" in the start running state.
Check the journal. You should see expected errors:

MASTER <-> REPLICA sync started
Connecting to MASTER .....
Error condition on socket for SYNC: Connection refused

Try to stop the service: systemctl stop keydb-server
Systemd job will hang "forever".
Try to send a SIGTERM, keydb-server will ignore it.
Only way to reliably stop keydb-server is a SIGKILL.

Expected behavior

service should accept and respond to a SIGTERM during startup.

Additional information

Running debian buster, keydb package provided by EQ-Alpha

[VivekSainiEQ]

Hi @h0tw1r3,

Can you please share the keydb logs while this is happening? That may help to figure out exactly what's going on.

You can do this by running journalctl -u keydb-server (assuming you haven't changed the logfile directive to something else)

[VivekSainiEQ]

So there seem to be two issues at hand:

1. The job being stuck in the "starting" state was an issue relating to KeyDB not being compiled with the correct systemd support. This was fixed in Fixed bug where make USE_SYSTEMD=yes would not pass in correct flags #296, which is included with the latest KeyDB release (v6.2.0).

2. The inability to SIGTERM to kill seems to be a permissions issue with the dump.rdb file being unable to be open for writing on exit. As a result, the KeyDB server won't stop sue to being unable to save the .rdb file. The solution to this is to modify your KeyDB service settings/folder permissions such that the working directory (specified in the configs as dir, defaults to the location of the keydb-server executable) is able to be read and written by the systemd service.

Updating to the newest KeyDB release and updating your KeyDB service settings/folder permissions as necessary should fix the issues at hand. If after doing so these issues persist, please feel free to reopen this ticket.