Systemd service file for Turbinia server and worker #169

beamcodeup · 2018-03-07T01:57:53Z

Fix/Update init scripts #67

A new systemd service file for Turbinia. This will officially replace the scripts in turbinia/tools/gcp_init/.

Note: remember to delete the gcp init scripts in the future.

aarontp

Thanks for the PR. A few comments in line.

aarontp · 2018-03-08T22:21:11Z

tools/turbinia.service

+Group=turbinia
+Restart=always
+RestartSec=10
+ExecStartPre=/bin/sh -c 'sudo systemctl set-environment GOOGLE_APPLICATION_CREDENTIALS=/home/turbinia/turbinia-service-account-creds.json'


Is this the right way to set environment variables in this case? I don't know systemd much, but a quick search shows that there's also an 'Environment' key that can be set.

The problem with the Environment key is that it needs to be hardcoded, which is why i'm using execstartpre.

aarontp · 2018-03-08T22:22:32Z

tools/turbinia.service

+Restart=always
+RestartSec=10
+ExecStartPre=/bin/sh -c 'sudo systemctl set-environment GOOGLE_APPLICATION_CREDENTIALS=/home/turbinia/turbinia-service-account-creds.json'
+ExecStartPre=/bin/sh -c 'sudo systemctl set-environment TURBINIA_ROLE=$(/usr/bin/python /home/turbinia/turbinia_config_parser.py ROLE)'


Same comment about the environment, but it looks like in this case this might be the way to go because it does need an exec?

What user will this run as? Does this mean that turbinia will need sudo privs for this as well?

Referring to the User and Group attribute above, the processes will be ran as turbinia. However, in sudo, systemctl will run as root (which is required). Yes, turbinia will need sudo privs, which we have already set up in the /etc/sudoers.

Per Table 1 at https://www.freedesktop.org/software/systemd/man/systemd.exec.html, the prefix "+" set in any Exec*= denotes "the process is executed with full privileges". Therefore, sudo is not needed here. This means systemd will run Exec* as turbinia/turbinia unless they are denoted with special executiable prefixes. So, I applied the + prefix for ExecStartPre and ExecStartPost lines.

aarontp · 2018-03-08T22:29:47Z

tools/turbinia.service

+Group=turbinia
+Restart=always
+RestartSec=10
+ExecStartPre=/bin/sh -c 'sudo systemctl set-environment GOOGLE_APPLICATION_CREDENTIALS=/home/turbinia/turbinia-service-account-creds.json'


I think we should grab this filename from the config rather than hard-coding it here.

aarontp · 2018-03-08T22:37:27Z

tools/turbinia.service

+RestartSec=10
+ExecStartPre=/bin/sh -c 'sudo systemctl set-environment GOOGLE_APPLICATION_CREDENTIALS=/home/turbinia/turbinia-service-account-creds.json'
+ExecStartPre=/bin/sh -c 'sudo systemctl set-environment TURBINIA_ROLE=$(/usr/bin/python /home/turbinia/turbinia_config_parser.py ROLE)'
+ExecStart=/bin/sh -c '/home/turbinia/turbinia-dev2/bin/python /home/turbinia/turbinia/turbiniactl -L /tmp/turbinia-$TURBINIA_ROLE.log -o /tmp $TURBINIA_ROLE 2>> /tmp/turbinia-$TURBINIA_ROLE.stdout.log'


Do we need the '/bin/sh -c' and the python wrappers here? Why can't we execute turbiniactl directly?

Yes, without using /bin/sh, systemd complains that it cannot read this line because of the stderr symbol. To workaround, /bin/sh -c '' is used.

aarontp · 2018-03-08T22:37:52Z

tools/turbinia.service

+RestartSec=10
+ExecStartPre=/bin/sh -c 'sudo systemctl set-environment GOOGLE_APPLICATION_CREDENTIALS=/home/turbinia/turbinia-service-account-creds.json'
+ExecStartPre=/bin/sh -c 'sudo systemctl set-environment TURBINIA_ROLE=$(/usr/bin/python /home/turbinia/turbinia_config_parser.py ROLE)'
+ExecStart=/bin/sh -c '/home/turbinia/turbinia-dev2/bin/python /home/turbinia/turbinia/turbiniactl -L /tmp/turbinia-$TURBINIA_ROLE.log -o /tmp $TURBINIA_ROLE 2>> /tmp/turbinia-$TURBINIA_ROLE.stdout.log'


Don't we want to set the -o flag with a variable from the config?

Yes, I overlooked this one. the value for -o will be referenced to TMP_DIR in the config.

Done.

beamcodeup · 2018-03-15T17:54:39Z

@aarontp PTAL!

beamcodeup · 2018-03-20T21:41:38Z

After testing this extensively, I believe it would make more sense to have two seperate systemd services for, respectively, server and worker(s). Systemd supports Instantiated Services, which is great for workers per this doc. This overall approach would allow us to run both server and worker(s) daemons on the same machine. Currently working on this now.

beamcodeup · 2018-03-21T19:16:37Z

With systemd 236 or newer we can write directly to a file using StandardError=file:/some/path systemd/systemd#7198 instead of using stdout/stderr symbols in ExecStart. Unforutnately, the stable Debian 9.3 is shipped with systemd 232, per https://packages.debian.org/stretch/systemd.

systemd 232 supports:
inherit, null, tty, journal, syslog, kmsg, journal+console, syslog+console, kmsg+console or socket

For now, we can use the syntax /bin/sh -c ".... >> ...." for ExecStart. We could explore the use of syslog in the future for this project.

aarontp · 2018-04-10T04:47:14Z

LMK if you're waiting for anything from me on this. AFAIK we still want to split this into two init scripts for server and worker with most everything in a template. I think we talked about changing the ROLE config var into something like SERVER and PSQWORKER booleans so that we could run both roles on the same host. Thanks!

…itely if the parser returned error codes

beamcodeup · 2018-04-10T22:12:46Z

Systemd supports service instantiations from a single template file. So, this service file will serve as a template from which two units will be instantiated as "server" and "psqworker". This will eliminate the need of SERVER/PSQWORKER booleans.

beamcodeup · 2018-04-10T22:13:27Z

@aarontp PTAL

aarontp · 2018-04-10T23:54:23Z

tools/turbinia@.service

+Restart=on-abnormal
+RestartSec=10
+RestartPreventExitStatus=100 101 102
+ExecStartPre=+/bin/sh -c 'GOOGLE_APPLICATION_CREDENTIALS=$(/usr/bin/python /home/turbinia/turbinia_config_parser.py GCE_SERVICE_ACCOUNT_KEYS_FILE) && /bin/systemctl set-environment GOOGLE_APPLICATION_CREDENTIALS=$GOOGLE_APPLICATION_CREDENTIALS'


I think we want to remove the '/home/turbinia/turbinia_config_parser.py' hard-coded path here and assume that the script is somewhere in the $PATH, right?

Correct. It was a bad copy and paste. Fixed.

aarontp · 2018-04-11T00:00:10Z

tools/turbinia@.service

+Restart=on-abnormal
+RestartSec=10
+RestartPreventExitStatus=100 101 102
+ExecStartPre=+/bin/sh -c 'GOOGLE_APPLICATION_CREDENTIALS=$(/usr/bin/python /home/turbinia/turbinia_config_parser.py GCE_SERVICE_ACCOUNT_KEYS_FILE) && /bin/systemctl set-environment GOOGLE_APPLICATION_CREDENTIALS=$GOOGLE_APPLICATION_CREDENTIALS'


So, this effectively sets the env var twice here. I guess the second one is for the global systemd settings, why do we need both? Is that used for other calls in this script? I guess what I'm asking is why can't we just do:
ExecStartPre=+/bin/sh -c '/bin/systemctl set-environment GOOGLE_APPLICATION_CREDENTIALS=$(/usr/bin/python turbinia_config_parser.py GCE_SERVICE_ACCOUNT_KEYS_FILE)'

Same comment for the other env vars.

The key difference between those lines is which will return the exit code to systemd. In your proposed line, systemd will receive the exit code from systemctl (which is 2) if the parser failed with any exit code (standard codes, including our customized ones like 100-102). Based on our chat offline, those custom exit codes and RestartPreventExitStatus are not needed and that Restart=always. The only withdraw is that if the parser failed, the systemd will restart in loop. This can be addressed seperately with some kind of monitoring/alerting So, rolling this back.

aarontp · 2018-04-11T00:16:28Z

tools/turbinia@.service

+RestartPreventExitStatus=100 101 102
+ExecStartPre=+/bin/sh -c 'GOOGLE_APPLICATION_CREDENTIALS=$(/usr/bin/python /home/turbinia/turbinia_config_parser.py GCE_SERVICE_ACCOUNT_KEYS_FILE) && /bin/systemctl set-environment GOOGLE_APPLICATION_CREDENTIALS=$GOOGLE_APPLICATION_CREDENTIALS'
+ExecStartPre=+/bin/sh -c 'TURBINIA_TMPDIR=$(/usr/bin/python /home/turbinia/turbinia_config_parser.py TMP_DIR) && /bin/systemctl set-environment TURBINIA_TMPDIR=$TURBINIA_TMPDIR'
+ExecStart=/bin/sh -c '/usr/local/turbinia/turbiniactl -L $TURBINIA_TMPDIR/turbinia-%i.log -S -o $TURBINIA_TMPDIR %i 2>> $TURBINIA_TMPDIR/turbinia-%i.stdout.log'


It looks like this exec line will work for both server types, but if we ever want to change the flags per service, is there a good way to do that? Just curious for the future since I don't know systemd very well.

It's best to keep the service file simple that we do not need to change it very often.

There are several options to address your question. Will we foreseen any major additional arguments for both server types?

Pass args via %I

sudo systemctl start turbinia@"server arg1 arg2 arg3"

The service file would have something like:
Environment="ARGS=%I"
ExecStart=/usr/local/foo $ARGS

Create two seperate service files

Create a local file that contains args:
ARG1=-L
ARG2=-o

The service file would have:
EnvironmentFile=/this/local/args.conf
ExecStart=/usr/local/foo $ARG1 $ARG2

1 more option.

Manually override command arguments for an instantied service without editing the template file directly. ExecStart would have to be cleared then set like this:

systemctl edit turbinia@psqserver

then an empty text file opens up then add:

[Service]
ExecStart=
ExecStart=/bin/sh -c '/usr/local/turbinia/turbiniactl --new-flags foobar'

Ok, thanks for the response here. I don't have any particular flag changes that I foresee at the moment, but I think it's likely that they will change at some point, so I just wanted to understand what our options were.

beamcodeup · 2018-04-11T01:09:35Z

PTAL

aarontp

Thanks!

aarontp · 2018-04-12T17:12:51Z

tools/turbinia@.service

+RestartPreventExitStatus=100 101 102
+ExecStartPre=+/bin/sh -c 'GOOGLE_APPLICATION_CREDENTIALS=$(/usr/bin/python /home/turbinia/turbinia_config_parser.py GCE_SERVICE_ACCOUNT_KEYS_FILE) && /bin/systemctl set-environment GOOGLE_APPLICATION_CREDENTIALS=$GOOGLE_APPLICATION_CREDENTIALS'
+ExecStartPre=+/bin/sh -c 'TURBINIA_TMPDIR=$(/usr/bin/python /home/turbinia/turbinia_config_parser.py TMP_DIR) && /bin/systemctl set-environment TURBINIA_TMPDIR=$TURBINIA_TMPDIR'
+ExecStart=/bin/sh -c '/usr/local/turbinia/turbiniactl -L $TURBINIA_TMPDIR/turbinia-%i.log -S -o $TURBINIA_TMPDIR %i 2>> $TURBINIA_TMPDIR/turbinia-%i.stdout.log'


Ok, thanks for the response here. I don't have any particular flag changes that I foresee at the moment, but I think it's likely that they will change at some point, so I just wanted to understand what our options were.

* Systemd service file for Turbinia server and worker * replace hardcoded lines with config parsers * replacing more hardcoded lies * add a line to prevent the turbinia service from auto restarting infinitely if the parser returned error codes * turbina service units will be based on the template file * Change to restart=on-abnormal * remove custom exit codes, set restart as always, remove local vars

Systemd service file for Turbinia server and worker

d27664c

beamcodeup requested a review from aarontp March 7, 2018 01:57

aarontp reviewed Mar 8, 2018

View reviewed changes

replace hardcoded lines with config parsers

e44efce

replacing more hardcoded lies

cdc3219

beamcodeup added 2 commits April 10, 2018 12:50

add a line to prevent the turbinia service from auto restarting infin…

db99f53

…itely if the parser returned error codes

turbina service units will be based on the template file

2b5c483

Change to restart=on-abnormal

c28f6a7

aarontp reviewed Apr 11, 2018

View reviewed changes

remove custom exit codes, set restart as always, remove local vars

0225a3d

aarontp mentioned this pull request Apr 11, 2018

Add service account key path to config, and automatically source it #59

Closed

aarontp approved these changes Apr 12, 2018

View reviewed changes

aarontp merged commit ca58f58 into google:master Apr 12, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Systemd service file for Turbinia server and worker #169

Systemd service file for Turbinia server and worker #169

beamcodeup commented Mar 7, 2018

aarontp left a comment

aarontp Mar 8, 2018

beamcodeup Mar 8, 2018 •

edited

aarontp Mar 8, 2018

aarontp Mar 8, 2018

beamcodeup Mar 8, 2018

beamcodeup Mar 15, 2018

aarontp Mar 8, 2018

beamcodeup Mar 15, 2018

aarontp Mar 8, 2018

beamcodeup Mar 8, 2018

aarontp Mar 8, 2018

beamcodeup Mar 15, 2018

beamcodeup commented Mar 15, 2018

beamcodeup commented Mar 20, 2018 •

edited

beamcodeup commented Mar 21, 2018

aarontp commented Apr 10, 2018

beamcodeup commented Apr 10, 2018

beamcodeup commented Apr 10, 2018

aarontp Apr 10, 2018

beamcodeup Apr 11, 2018

aarontp Apr 11, 2018

beamcodeup Apr 11, 2018

aarontp Apr 11, 2018

beamcodeup Apr 11, 2018

beamcodeup Apr 11, 2018

aarontp Apr 12, 2018

beamcodeup commented Apr 11, 2018

aarontp left a comment

aarontp Apr 12, 2018

Systemd service file for Turbinia server and worker #169

Systemd service file for Turbinia server and worker #169

Conversation

beamcodeup commented Mar 7, 2018

aarontp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

beamcodeup Mar 8, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

beamcodeup commented Mar 15, 2018

beamcodeup commented Mar 20, 2018 • edited

beamcodeup commented Mar 21, 2018

aarontp commented Apr 10, 2018

beamcodeup commented Apr 10, 2018

beamcodeup commented Apr 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

beamcodeup commented Apr 11, 2018

aarontp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

beamcodeup Mar 8, 2018 •

edited

beamcodeup commented Mar 20, 2018 •

edited