st2 log rotation missing for mistral

[sibirajal]

Hello Team,

It appears that st2 logrotate file is not included the mistral service and it causing the disk space consumption on the ST2 server. I've tried to manually perform the reopen-log-files option for mistral service and it sends the USR1 signal to mistral service, then the mistral-server was not started back.

I had to start the mistral-server service manually to fix this abrupt process termination. Can we please fix the mistral-server process to handle the USR1 signal and include the mistral in st2 log rotation?

st2ctl running status:

st2ctl status

st2 components status

st2actionrunner PID: 28784
st2actionrunner PID: 28786
st2actionrunner PID: 28788
st2actionrunner PID: 28790
st2actionrunner PID: 28792
st2actionrunner PID: 28794
st2actionrunner PID: 28796
st2actionrunner PID: 28798
st2actionrunner PID: 28800
st2actionrunner PID: 28802
st2api PID: 28810
st2api PID: 29219
st2stream PID: 28825
st2stream PID: 28999
st2auth PID: 28838
st2auth PID: 29016
st2garbagecollector PID: 28851
st2notifier PID: 28898
st2resultstracker PID: 28927
st2rulesengine PID: 28939
st2sensorcontainer PID: 28951
st2chatops PID: 28971
mistral-server PID: 11152
mistral-api PID: 29011
mistral-api PID: 29156
mistral-api PID: 29157

Available component in reopen-log-files:
Usage: st2ctl {reopen-log-files}
positional arguments:
component Name of the st2 service to reopen the log files for.
st2actionrunner st2api st2stream st2auth st2garbagecollector st2notifier st2resultstracker st2rulesengine st2sensorcontainer st2chatops mistral

Mistral not exist in st2 logrotate file:

cat /etc/logrotate.d/st2 |grep mistral -i

Manually tried for mistral reopen-log-file:

st2ctl reopen-log-files mistral

Sending SIGUSR1 to mistral PID: 11152
Sending SIGUSR1 to mistral PID: 29011
Sending SIGUSR1 to mistral PID: 29156
Sending SIGUSR1 to mistral PID: 29157

st2 components status

st2actionrunner PID: 28784
st2actionrunner PID: 28786
st2actionrunner PID: 28788
st2actionrunner PID: 28790
st2actionrunner PID: 28792
st2actionrunner PID: 28794
st2actionrunner PID: 28796
st2actionrunner PID: 28798
st2actionrunner PID: 28800
st2actionrunner PID: 28802
st2api PID: 28810
st2api PID: 29219
st2stream PID: 28825
st2stream PID: 28999
st2auth PID: 28838
st2auth PID: 29016
st2garbagecollector PID: 28851
st2notifier PID: 28898
st2resultstracker PID: 28927
st2rulesengine PID: 28939
st2sensorcontainer PID: 28951
st2chatops PID: 28971
mistral-server is not running.
mistral-api PID: 29011
mistral-api PID: 29156
mistral-api PID: 29157

Mistral-server status:

systemctl status mistral-server

● mistral-server.service - Mistral workflow service (engine/executor)
Loaded: loaded (/usr/lib/systemd/system/mistral-server.service; enabled; vendor preset: disabled)
Active: failed (Result: signal) since Fri 2017-11-10 10:26:00 UTC; 59s ago
Process: 11152 ExecStart=/bin/sh -c . /opt/stackstorm/mistral/share/sysvinit/helpers; enabled_list -q server || exit 0; exec /opt/stackstorm/mistral/bin/mistral-server --server $(enabled_list server) $SERVER_ARGS (code=killed, signal=USR1)
Main PID: 11152 (code=killed, signal=USR1)

Nov 10 10:23:44 predaarqa1001.mgmt.messagelabs.net systemd[1]: Started Mistral workflow service (engine/executor).
Nov 10 10:23:44 predaarqa1001.mgmt.messagelabs.net systemd[1]: Starting Mistral workflow service (engine/executor)...
Nov 10 10:26:00 predaarqa1001.mgmt.messagelabs.net systemd[1]: mistral-server.service: main process exited, code=killed, status=10/USR1
Nov 10 10:26:00 predaarqa1001.mgmt.messagelabs.net systemd[1]: Unit mistral-server.service entered failed state.
Nov 10 10:26:00 predaarqa1001.mgmt.messagelabs.net systemd[1]: mistral-server.service failed.

After starting the mistral-server manually:

systemctl status mistral-server

● mistral-server.service - Mistral workflow service (engine/executor)
Loaded: loaded (/usr/lib/systemd/system/mistral-server.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2017-11-10 10:27:21 UTC; 2s ago
Main PID: 11376 (mistral-server)
CGroup: /system.slice/mistral-server.service
└─11376 /opt/stackstorm/mistral/bin/python /opt/stackstorm/mistral/bin/mistral-server --server engine,executor --config-file /etc/mistra...

Nov 10 10:27:21 predaarqa1001.mgmt.messagelabs.net systemd[1]: Started Mistral workflow service (engine/executor).
Nov 10 10:27:21 predaarqa1001.mgmt.messagelabs.net systemd[1]: Starting Mistral workflow service (engine/executor)...

[arm4b]

Thanks for the report,

st2mistral log rotation was included in recent StackStorm v2.5.0 (will be included in upcoming v2.5.1): StackStorm/st2-packages#468, following the customer report, probably yours.

[arm4b]

I also created an Issue #3851 in st2 to address the st2ctl reopen-log-files mistral command problems in future.

[arm4b]

@sibirajal Sorry for confusion, I double checked and the fix is not yet in v2.5.0.
Thanks to your report, we'll cherry pick this fix in upcoming v2.5.1 which will be released soon.

[arm4b]

For now, the temporary workaround for you before the v2.5.1 would be deploying the /etc/logrotate.d/mistral file on your servers with the following contents:

https://github.com/StackStorm/st2-packages/blob/master/packages/st2mistral/conf/logrotate.conf