WIP: Template units

[bcwaldon]

This will allow you to submit a unit like foo@.service, then start a job like foo@42.service.

[philips]

OMG

[yichengq]

Cool stuff!!

[jsdir]

This will definitely make things easier. 👍

[polvi]

What is the advantage of this over just submitting a foo@bar.service unit directly?

[bcwaldon]

@polvi I think you could ask the same question of systemd, ignoring the fact that fleet sits between you and systemd.

[polvi]

The difference with systemd is that it does not actually make a differentiation. The symbolic link thing is just a convention. You are still actually starting that distinct unit (could be a link or a copy). I think to make this idiomatic systemd, you would just create the links locally then fleetctl start them all.

[polvi]

i.e. with systemd you do not need to load/submit your template before you can use it. You just interact with the units, which happen to be a copy.

[bcwaldon]

@polvi you do have to 'submit' your template in the sense that it needs to be in /etc/systemd/system (or /run/... or /usr/lib64/...). The submit step in fleetctl is the equivalent of you either calling systemctl enable foo@.service or copying foo@.service to /etc/systemd/system. The major value in templated units is just that - you've got one template from which you can create many instances. If we don't support this, humans will have to keep a copy of the template around locally and create symbolic links for each of the instances back to the original template. This is mostly all for human convenience.

[polvi]

nope.

enable is used for telling systemd to load the unit at boot time, has nothing to do with templates. You do not need the template in the systemd search path to use it.

polvi-3 system # cat /root/test\@.service 
[Service]
Type=oneshot
ExecStart=/usr/bin/echo %i
polvi-3 system # pwd
/etc/systemd/system
polvi-3 system # ln -s /root/test\@.service test@foo.service
polvi-3 system # systemctl start test@foo.service
polvi-3 system # systemctl status test@foo.service
test@foo.service
   Loaded: loaded (/root/test@.service)
   Active: inactive (dead)

Mar 27 22:38:51 polvi-3 systemd[1]: Starting test@foo.service...
Mar 27 22:38:51 polvi-3 echo[3423]: foo
Mar 27 22:38:51 polvi-3 systemd[1]: Started test@foo.service.

Systemd just reads the unit as is. It does not do any magic outside of parsing the data between '@' and '.service' and using it as a template var.

[bcwaldon]

@polvi In your example, you put (linked) test@.service in /etc/systemd/system, which is a systemd search path. Am I misunderstanding you when you say 'you do not need the template in the systemd search path to use it'?

[polvi]

@bcwaldon it is in /root/.

polvi-3 system # ls -l /etc/systemd/system
total 4
lrwxrwxrwx 1 root root 19 Mar 27 22:38 test@foo.service -> /root/test@.service
polvi-3 system # ls -l /root/ # just random non-systemd search path 
total 4
-rw-r--r-- 1 root root 50 Mar 27 22:38 test@.service

[bcwaldon]

@polvi but the fact that it is in /etc/systemd/system means etcd "knows" about it...right? You couldn't start test@foo.service if your cwd was /root and the /etc/systemd/system/test@.service symlink did not exist.

[bcwaldon]

Oh - misread how your symlink was setup.

[polvi]

My point is that the template does nothing special with regards to the link. Here is another example. Note: systemd has no idea about the template unit now.

polvi-3 system # cp /root/test\@.service test@bar.service
polvi-3 system # ls
test@bar.service  test@foo.service
polvi-3 system # cat test\@bar.service 
[Service]
Type=oneshot
ExecStart=/usr/bin/echo %i
polvi-3 system # systemctl start test@bar.service
polvi-3 system # systemctl status !$
systemctl status test@bar.service
test@bar.service
   Loaded: loaded (/etc/systemd/system/test@bar.service; static)
   Active: inactive (dead)

Mar 27 22:49:58 polvi-3 systemd[1]: Starting test@bar.service...
Mar 27 22:49:58 polvi-3 echo[3553]: bar
Mar 27 22:49:58 polvi-3 systemd[1]: Started test@bar.service.

[polvi]

Just for clarity, foo is a symbolic link. bar is a copy of the file directly. Same result.

polvi-3 system # ls -l
total 8
-rw-r--r-- 1 root root 50 Mar 27 22:49 test@bar.service
lrwxrwxrwx 1 root root 19 Mar 27 22:38 test@foo.service -> /root/test@.service

[bcwaldon]

@polvi Here's the workflow I'm thinking of:

core@core-01 ~/fleet $ sudo mv sleep\@.service /etc/systemd/system
core@core-01 ~/fleet $ sudo systemctl daemon-reload
core@core-01 ~/fleet $ sudo systemctl start sleep@100.service
core@core-01 ~/fleet $ sudo systemctl status sleep@100.service
sleep@100.service
   Loaded: loaded (/etc/systemd/system/sleep@.service; static)
   Active: active (running) since Thu 2014-03-27 22:52:53 UTC; 3s ago
 Main PID: 6765 (sleep)
   CGroup: /system.slice/system-sleep.slice/sleep@100.service
           └─6765 /usr/bin/sleep 100

Mar 27 22:52:53 core-01 systemd[1]: Started sleep@100.service.

I put sleep@.service in the systemd search path, then I can start arbitrary instances of it.

[polvi]

OK, I get it now.

[robszumski]

You guys are doing a lot of talking about stuff on a single machine. If your template contains X-Conflicts that spreads it out over the cluster, does fleet drop the template and the instance of that template on each machine?

What happens when you submit a modified template into fleet. Do all of the instances start referencing the new template? Or does fleet reject it like it does now when you attempt to start something with the same name?

[bcwaldon]

@robszumski The payload would have to exist on every machine that's running a dependent job. So yes, the template unit file would be copied everywhere its needed.

Payloads will remain immutable, so you will still not be able to push an updated template.

[bcwaldon]

Closing so @unihorn can tackle this.

[apatil]

@bcwaldon is there a new PR up yet?

I was just about to post with a test result. fleetctl submit db@.service followed by fleetctl start db@1.service resulted in the following in systemctl status -n 100 --no-page | grep db@:

Apr 07 23:13:46 host-name fleet[3414]: I0407 23:13:46.922144 03414 event.go:36] EventJobOffered(db@1.service): passed all criteria, submitting JobBid
Apr 07 23:13:46 host-name fleet[3414]: I0407 23:13:46.922165 03414 agent.go:212] Submitting JobBid for Job(db@1.service)
Apr 07 23:13:46 host-name fleet[3414]: I0407 23:13:46.931579 03414 engine.go:115] Scheduled Job(db@1.service) to Machine(8fa2b7fd-2673-4d6f-af48-fdec62dbaf33)
Apr 07 23:13:46 host-name fleet[3414]: I0407 23:13:46.936742 03414 agent.go:144] Starting Job(db@1.service)
Apr 07 23:13:46 host-name fleet[3414]: I0407 23:13:46.937267 03414 manager.go:161] Writing systemd unit file db@.service
Apr 07 23:13:47 host-name fleet[3414]: E0407 23:13:47.273613 03414 manager.go:111] Failed to enable systemd unit db@1.service: No such file or directory

The service never started.

[bcwaldon]

@apatil This is going to be handled by @unihorn. No new code from me, this was just investigative.

[apatil]

OK, np. @unihorn would you mind pinging this PR when yours goes up? Also let me know if you need something tested.

[yichengq]

@apatil I am gonna investigate and support this feature in #303. You are more than welcome to discuss about the details and testing of this functionality. :)