Systemd detects exit 1 error when using shutdown command

[lework]

Version:

• os: centos 7.4 x64
• redis: 5.0.7

Conf:

redis

bind 0.0.0.0
port 7003
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
supervised systemd
pidfile /var/run/redis_7003.pid
loglevel notice
logfile /data/redis/7003/log/redis_7003.log
databases 16
save ""
maxclients 10000
maxmemory 512MB
maxmemory-policy volatile-ttl
dir /data/redis/7003/data
requirepass 123456
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes

systemd

[Unit]
Description=Redis persistent key-value database
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/redis/bin/redis-server /data/redis/7003/conf/redis.conf --supervised systemd
ExecReload=/bin/kill -USR2 $MAINPID
ExecStop=/usr/local/redis/bin/redis-cli -p 7003 -a 123456 shutdown;
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755
Restart=on-failure
LimitNOFILE=10240

[Install]
WantedBy=multi-user.target

Step

1. redis systemd status

[root@node130 ~]# systemctl status redis7003               
● redis7003.service - Redis persistent key-value database
   Loaded: loaded (/usr/lib/systemd/system/redis7003.service; enabled; vendor preset: disabled)
   Active: active (running) since 五 2020-01-17 17:35:35 CST; 27s ago
 Main PID: 8167 (redis-server)
    Tasks: 4
   Memory: 6.6M
   CGroup: /system.slice/redis7003.service
           └─8167 /usr/local/redis/bin/redis-server 0.0.0.0:7003 [cluster]

1月 17 17:35:35 node130 systemd[1]: Starting Redis persistent key-value database...
1月 17 17:35:35 node130 systemd[1]: Started Redis persistent key-value database.

2. exec shutdown

[root@node130 ~]# redis-cli -p 7003 -a 123456 shutdown
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
[root@node130 ~]# echo $?
0

command exit 0

3. redis systemd status

[root@node130 ~]# systemctl status redis7003          
● redis7003.service - Redis persistent key-value database
   Loaded: loaded (/usr/lib/systemd/system/redis7003.service; enabled; vendor preset: disabled)
   Active: active (running) since 五 2020-01-17 17:36:16 CST; 1s ago
  Process: 8176 ExecStop=/usr/local/redis/bin/redis-cli -p 7003 -a 123456 shutdown; (code=exited, status=1/FAILURE)
 Main PID: 8181 (redis-server)
    Tasks: 4
   Memory: 6.7M
   CGroup: /system.slice/redis7003.service
           └─8181 /usr/local/redis/bin/redis-server 0.0.0.0:7003 [cluster]

1月 17 17:36:16 node130 systemd[1]: Starting Redis persistent key-value database...
1月 17 17:36:16 node130 systemd[1]: Started Redis persistent key-value database.

status FAILURE, restart redis.

4. /var/log/message

[root@node130 ~]# tail /var/log/messages
Jan 17 17:35:35 node130 systemd: Started Redis persistent key-value database.
Jan 17 17:36:16 node130 redis-cli: Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Jan 17 17:36:16 node130 redis-cli: Could not connect to Redis at 127.0.0.1:7003: Connection refused
Jan 17 17:36:16 node130 systemd: redis7003.service: control process exited, code=exited status=1
Jan 17 17:36:16 node130 systemd: Unit redis7003.service entered failed state.
Jan 17 17:36:16 node130 systemd: redis7003.service failed.
Jan 17 17:36:16 node130 systemd: redis7003.service holdoff time over, scheduling restart.
Jan 17 17:36:16 node130 systemd: Stopped Redis persistent key-value database.
Jan 17 17:36:16 node130 systemd: Starting Redis persistent key-value database...
Jan 17 17:36:16 node130 systemd: Started Redis persistent key-value database.

I don't know why it returns this error.

[antirez]

Hello, I've no idea, if I do ./redis-server; echo $? I see "0" indeed. I don't use systemd so I'm not sure, maybe somebody can provide more hints.

[tjarlama]

From systemd man page:

ExecStop=
...
Also note that the stop operation is always performed if the service started successfully, even if the processes in the service terminated on their own or were killed. The stop commands must be prepared to deal with that case.

Jan 17 17:36:16 node130 redis-cli: Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Jan 17 17:36:16 node130 redis-cli: Could not connect to Redis at 127.0.0.1:7003: Connection refused
Jan 17 17:36:16 node130 systemd: redis7003.service: control process exited, code=exited status=1
Jan 17 17:36:16 node130 systemd: Unit redis7003.service entered failed state.

Problem is not the redis-server. It exits with 0 status code. But you specified ExecStop in unit file and it gets executed when main process exits. This will have a non-zero exit status because server is shutdown. Systemd consider it as a failure and restart service. It should work if you try without ExecStop.

[lework]

Sorry, I have some problems in my own thinking. Thanks for the guidance, i will close this.

[john-f-chamberlain]

The error results from the fact that /usr/libexec/redis-shutdown returns 1 when a password is used due to the error:
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

This will cause systemctl to claim that the shutdown failed to complete and marks the service as failed rather than inactive.

Without a password specified everything works as expected because redis-cli returns 0 due to not passing a password as an argument. I would consider this a bug that should be resolved, perhaps by adding an additional --ignore-password-warning flag or something similar.

A workaround that I have put in place for the time being is to simply add an echo statement to the bottom of the /usr/libexec/redis-shutdown script so it returns 0, but this obviously isn't ideal since any actual errors will now also be ignored.