wrappers: allow user mode systemd daemons #5822

jhenstridge · 2018-09-12T09:38:38Z

This patch extends the snap daemon support to support systemd running in user mode via a daemon-mode: user parameter.

The main differences are:

Installed to /etc/systemd/user instead of /etc/systemd/system
Can not depend on system units (and conversely system units can't depend on user units)
Sockets associated with user daemons are limited to user writable locations (e.g. $SNAP_USER_DATA).
snapd can not enable/disable user daemons during upgrades, as it can't communicate with the user instance(s) of systemd.

My eventual goal is to build D-Bus activation on top of this for both user session and system daemons.

I've been trying to write a spread test for this, but so far I've failed. My initial thought was to try and manually run /lib/systemd/systemd --user as the test user and then check how the daemon runs under that instance. Unfortunately that fails, and it looks like the only supported way of starting user instances is via pam_systemd + logind. It might be possible with the logind CreateSession method, but I haven't yet worked out what arguments to pass to get it to work.

zyga

Looks very reasonable! I only asked a question about the differences between the new user services and system services wrtd dependencies on the mount unit.

+1 - please get another review from @chipaca

wrappers/services.go

jhenstridge · 2018-09-12T11:39:16Z

And it looks like I was incorrect about enabling/disabling user services. It seems I want:

systemctl --global enable user-service-name

jhenstridge · 2018-09-13T11:21:05Z

So I worked out that I can get the user systemd instance started in the spread test by activating user@12345.service in the system instance. Two simple spread tests added showing a user daemon started with the default target, and user daemon sockets being created in the expected locations.

jhenstridge · 2018-09-14T01:16:08Z

I've disabled the new spread tests on a few systems:

Ubuntu Core: installing a snap with user services fails because /etc/systemd/user is not in the writable-paths list. If the core snap is updated the test will probably work, but core systems don't usually have a user session.
Ubuntu 14.04: it looks like the user@.service method of starting the user instance of systemd was added in version 205, while that distro release ships version 204.
Amazon Linux 2: this one also fails to load user@.service in the tests. I would have thought it would have a new enough systemd, but haven't investigated further since it is unlikely people will run a user session on this distro.

mvo5

Thanks for working on this. This looks really nice, I added some question inline.

mvo5 · 2018-09-17T13:34:01Z

interfaces/systemd/backend.go

@@ -85,7 +85,7 @@ func (b *Backend) Setup(snapInfo *snap.Info, confinement interfaces.ConfinementO
 	}
 	// Ensure the service is running right now and on reboots
 	for _, service := range changed {
-		if err := systemd.Enable(service); err != nil {
+		if err := systemd.Enable(service, false); err != nil {


I would prefer a flag argument here. Reading: systemd.Enable(service, true) does not convey much information. Something like systemd.Enable(service, systemd.UserInstance) or systemd.Enable(service, 0) is a bit more explicit.

snap/info_snap_yaml_test.go

mvo5 · 2018-09-17T14:06:55Z

tests/main/snap-user-service-socket-activation/task.yaml

+    echo "And the user mode systemd instance is started"
+    systemctl start user@"$(id -u test)".service
+
+    echo "It's sockets are created in the test user's directories"


Should we do a real end-to-end test here, i.e. have something connect to the socket and verify that the snap actually replies?

I was looking for a similar test for system daemons, and all I could find was tests/main/install-socket-activation, which isn't a full end-to-end test either. I agree that it would be a good idea to extend the test to do this.

I've updated the test to use a Python daemon implementation of the service that responds differently to the different sockets. This could be reused to test socket activation of system services.

mvo5 · 2018-09-17T14:08:23Z

tests/main/snap-user-service/task.yaml

+    systemctl start user@"$(id -u test)".service
+
+    echo "We can see the service running"
+    as_user systemctl --user status snap.test-snapd-user-service.test-snapd-user-service|MATCH "running"


(nitpick) I think we should check for Active: active just for good measure to ensure that the thing is still running.

mvo5 · 2018-09-17T14:13:52Z

wrappers/services.go

@@ -274,6 +290,10 @@ func StopServices(apps []*snap.AppInfo, reason snap.ServiceStopReason, inter int
 		if !app.IsService() || !osutil.FileExists(app.ServiceFile()) {
 			continue
 		}
+		// We can't stop services that run under a user mode systemd
+		if app.IsUserService() {


The fact that we don't stop services is different than for system services. Can this break user services?

It's more the fact that snapd can't talk to the user mode systemd instance (or multiple instances if you happen to have a multi-seat system).

As for what could break, it'd be the same things that can break for snap applications running over an upgrade or uninstall.

This is problematic. We either need to find a way to stop them or we need infrastructure enough to inform the snap that is about to be updated and for it to do the right thing with those?

@jhenstridge also does this mean that "snap start/stop" etc don't work for these services?

Correct: "snap start/stop" will not do anything because snapd can not talk to the user session systemd instance.

So we're essentially in the same position as existing long running snap applications (e.g. a messaging app started via xdg autostart that runs minimised and is only visible through the notifications it posts).

We are in a better position than the status quo though, in that if snapd grows a user session agent to handle upgrades, there is a defined method of restarting these daemons (unlike a long running GUI application).

wrappers/services.go

niemeyer

No changes required right now. Just marking it as I want to have a careful dive on this one, given that it's a major new area being opened.

jhenstridge · 2018-11-23T04:08:40Z

I've updated the branch to work with current master. I'm not running the snap-user-service tests on CentOS 7 for the same reason as Amazon Linux 2: it ships an old version of systemd where the current method of starting a user session (activating the user@$uid.service unit) doesn't work.

jdstrand · 2018-11-29T13:40:41Z

snap/info.go

+	case systemd.UserMode:
+		return dirs.SnapUserServicesDir
+	default:
+		panic("unknown systemd.InstanceMode")


Should this (and other places below) be systemd.DaemonMode?

I called the type InstanceMode, which is why I included that in the message. I'm open to changing it though. There's two ways it is used at the moment:

which instance of systemd are we talking to?

what mode does this service run in?

jdstrand · 2018-11-29T13:58:31Z

tests/main/snap-user-service-socket-activation/task.yaml

+    # Amazon Linux 2 gives error "Unit user@12345.service not loaded."
+    - -amazon-linux-2-*
+    # Centos 7 gives error "Unit user@12345.service not loaded."
+    - -centos-7-*


What is the user experience for someone installing a snap with a user service on these distributions (or others where user services aren't in use yet)?

The problem is that the method of launching the user mode systemd has changed over the course of releases.

With current versions of systemd, the supported method is to ask the pid 1 systemd instance to start the templated user@.service service. With older releases, my understanding is that systemd --user was invoked directly by logind.

As written, the test relies on the new method of launching the user instance, since trying directly execute it will fail with complaints about cgroups. I haven't looked into what would be necessary to get the test fixture running on these old systemd's, and wasn't sure how important it was.

My primary motive for this feature is for use in conjunction with dbus service activation. For that particular case, I plan to use the same method of supporting distros without a user instance of systemd as most services do currently: include both an Exec and SystemdService line in the dbus service activation file.

For other types of user services I don't have a good answer. It is basically the same situation as running a snap that uses XDG Autostart on a desktop that doesn't support it.

Would using session-tool avoid the problem of systemd changing how user sessions are started across releases?

Thanks for the detailed response. Sounds good :)

Would using session-tool avoid the problem?

I suspect it wouldn't. I booted up the Centos 7 live CD in a VM to see what its desktop session looked like, and there was no evidence of a user instance of systemd running.

Either it ships with a too-old systemd, or they intentionally disabled it so it wouldn't be part of their support burden.

jdstrand · 2018-11-29T14:02:22Z

wrappers/services.go

@@ -114,24 +114,26 @@ func StartServices(apps []*snap.AppInfo, inter interacter) (err error) {
 			if err == nil {
 				return
 			}
-			if e := stopService(sysd, app, inter); e != nil {
-				inter.Notify(fmt.Sprintf("While trying to stop previously started service %q: %v", app.ServiceName(), e))
+			if app.ServiceMode() == systemd.SystemMode {


I think you are doing the right thing here (ie, not mixing in user services with the snap start|stop|restart commands), but it makes me wonder if snap start|stop|restart should grow a --user option or similar for these...

It's not something that snapd can control though: in general snapd running as pid 0 is not going to be able to communicate with the systemd instance running as the user.

At the moment we don't do anything about snap apps running as the user during an upgrade, so I don't think it is that big a stretch to do the same for user services.

Note that this is something that (iirc), @chipaca is looking into. It would be possible for snapd to communicate with userd to do something sensible on refresh. Also note that my question is not meant to block this PR; just putting it out there for a potential followup.

wrappers/services_test.go

jdstrand

I looked at this PR from a security point of view (but had a few questions other questions inline). In general, this PR LGTM in that regard. One thing that came to mind is that, unlike system services, user services are likely to be coupled with typical desktop interfaces (eg, X11, desktop-legacy; either intentionally or devs taking a shotgun approach with their plugs) which could open up interesting (from a security POV) interactions between snaps. @jhenstridge, curious on your thoughts about adding review-tools check to warn if specifying 'daemon-mode: user' for a command that 'plugs: [ x11 ]' (or similar)?

jhenstridge · 2018-11-30T11:32:10Z

@jhenstridge, curious on your thoughts about adding review-tools check to warn if specifying 'daemon-mode: user' for a command that 'plugs: [ x11 ]' (or similar)?

I don't think it would be out of the ordinary for session D-Bus daemons (for instance) to want to talk to the display, so I don't think it is something review-tools should block.

While this does provide a new way for a snap author to have their code executed, is it that different to e.g. the way we allow apps to launch via XDG autostart?

jdstrand · 2018-11-30T16:47:29Z

@jhenstridge, curious on your thoughts about adding review-tools check to warn if specifying 'daemon-mode: user' for a command that 'plugs: [ x11 ]' (or similar)?

...

While this does provide a new way for a snap author to have their code executed, is it that different to e.g. the way we allow apps to launch via XDG autostart?

With autostart the user should be prompted, with this PR, the user session service is just there potentially long running in the background (eg, this makes something like a persistent keylogger easier). Again, X is deeply flawed from a security POV so not saying we should block this on that (users will always need to trust the publisher when a snap plugs x11, desktop-legacy, etc and this PR doesn't change that). I just wanted your thoughts and you answered my question wrt dbus services wanting to talk to the display.

pedronis · 2019-01-23T13:44:45Z

I'm closing this until there is a proposal (on the forum) of how to stop these services at refresh like we stop any other services, and what snap start/stop etc should do with them

pedronis

thank you

jhenstridge · 2020-04-21T10:16:14Z

Over two CI runs, I've gotten the following error:

##[error]2020-04-21 07:15:48 Error executing google:ubuntu-core-16-64:tests/main/snap-user-service (apr210650-369708) : 
...
+ snap install --dangerous /home/gopath/src/github.com/snapcore/snapd/tests/lib/snaps/test-snapd-user-service/test-snapd-user-service_1.0_all.snap
error: cannot perform the following tasks:
- Make snap "test-snapd-user-service" (unset) available to the system (Post http://0/v1/service-control: read unix @->/run/user/0/snapd-session-agent.socket: read: connection reset by peer)

This look like a left over session agent socket in root's XDG_RUNTIME_DIR, with nothing listening on the socket. It probably indicates another test not cleaning up correctly, but it does make me wonder if we should treat "connection reset by peer" errors on the initial dial as if that user's session agent is not present. If we can run into this problem in CI, then I wouldn't be surprised if it can happen on real systems too.

zyga

Hey James

I'm submitting my partial review. I still have a few more thorny files to read through to make sure I didn't miss any detail.

This branch will have some interaction with my app awareness work that will likely land shortly. I will be happy to work with you on ensuring the result of merging both branches is correct. I believe that I need to adjust some of the logic used to create transient scopes as well as to, perhaps, adjust the cgroup scanner (though I think it is correct, it should be tested for sure).

My main worry is the extensive use of user.sh that I would rather see sunset. I think using session-tool is a viable alternative and I would not be surprised if that would make all of the tests to pass on all but 14.04 systems.

I've left a few comments inline, please go through them and I promise that by tomorrow I will complete the review of the remaining parts.

zyga · 2020-04-17T13:58:12Z

tests/main/snap-user-service-restart-on-upgrade/task.yaml

+        touch agent-was-enabled
+    fi
+    snap set system experimental.user-daemons=true
+    #shellcheck source=tests/lib/user.sh


I would prefer this to be ported to session-tool instead. This can be done in a follow up.

zyga · 2020-04-17T13:59:30Z

tests/main/snap-user-service-socket-activation/task.yaml

+    # Amazon Linux 2 gives error "Unit user@12345.service not loaded."
+    - -amazon-linux-2-*
+    # Centos 7 gives error "Unit user@12345.service not loaded."
+    - -centos-7-*


Would using session-tool avoid the problem of systemd changing how user sessions are started across releases?

usersession/agent/rest_api.go

zyga · 2020-04-21T11:02:02Z

snap/validate.go

-			"invalid %q: must have a prefix of $SNAP_DATA, $SNAP_COMMON or $XDG_RUNTIME_DIR", fieldName)
+	switch socket.App.DaemonScope {
+	case SystemDaemon:
+		if !(strings.HasPrefix(path, "$SNAP_DATA/") || strings.HasPrefix(path, "$SNAP_COMMON/") || strings.HasPrefix(path, "$XDG_RUNTIME_DIR/")) {


Curiously, I think this is wrong. System daemons cannot refer to $XDG_RUNTIME_DIR, as that is only provided to sessions. On my Focal system system services no longer run in sessions so they do not have XDG_RUNTIME_DIR.

I would be happy to have a follow up that deals with this but please add a // TODO comment if you are going to push any more patches into this branch.

I don't think it makes sense either: given that systemd mounts a tmpfs over a user's $XDG_RUNTIME_DIR, it isn't clear the system instance of systemd can reliably provide sockets in that location (even ignoring the question of which user's XDG_RUNTIME_DIR it should refer to).

FWIW, it looks like this was introduced in PR #6327. I left it unchanged since there are probably some snaps making use of it right now (maybe something from the Mir team?).

We should look, after this PR, because I suspect it expands to something silly (unless we expand it ignoring the environment).

tests/main/user-session-env/task.yaml

zyga · 2020-04-21T11:20:28Z

wrappers/services.go

@@ -372,7 +485,7 @@ func StopServices(apps []*snap.AppInfo, reason snap.ServiceStopReason, inter int

 		// ensure the service is really stopped on remove regardless
 		// of stop-mode
-		if reason == snap.StopReasonRemove && !app.StopMode.KillAll() {
+		if reason == snap.StopReasonRemove && !app.StopMode.KillAll() && app.DaemonScope == snap.SystemDaemon {


Is there a TODO that captures stopping of user services?

As mentioned in previous comments I believe this block of code is unnecessary for user services, and possibly system services too on modern distros. Unless the service has KillMode=none or sets FinalKillSignal to something other than SIGKILL, systemd will have gone through these steps already. As we don't allow snaps to alter those settings the sysd.Stop() call should be reliable.

From what I've gathered, this was in place to deal with old buggy versions of systemd, which would be from before they supported a user instance of systemd.

zyga · 2020-04-21T11:25:26Z

snap/info_snap_yaml_test.go

+version: 42
+apps:
+ svc:
+   command: svc1


Nitpick: use svc for consistency with the service name.

zyga · 2020-04-21T11:27:50Z

tests/lib/snaps/test-snapd-user-service-sockets/bin/start

@@ -0,0 +1,53 @@
+#!/usr/bin/python3


We probably need the copyright header here.

zyga

Thank you so much for porting this to session-tool

I marked one case of missing --restore and asked a question about the purge step.

tests/main/snap-user-service/task.yaml

tests/main/snap-user-service-upgrade-failure/task.yaml

tests/main/snap-user-service-socket-activation/task.yaml

This is an attempt to either fix the intermittent hang during tear down, or make it fail reliably.

…emons

zyga · 2020-05-06T21:44:54Z

@jhenstridge I merged master into your branch just now, to hopefully get a green pass. We just landed a fix for the two pulseaudio tests that were picking up some test leaks and that had a host of issues of their own.

zyga

I reviewed just the spread tasks that contain now-redundant startups of .target units that session-tool --prepare now handles automatically.

tests/main/snap-user-service-restart-on-upgrade/task.yaml

zyga · 2020-05-06T21:46:22Z

tests/main/snap-user-service-socket-activation/task.yaml

+    echo "And the user mode systemd instance is started"
+    session-tool -u test --prepare
+
+    echo "And the user session has finished starting"


Those two can now go away

Removing these caused a failure on the Ubuntu 16.04 based systems (both classic and core).

From the look of it, session-tool --prepare will only wait for default.target if there is a systemd controlled session bus. Ubuntu 16.04's session bus is managed by Upstart, yet it has a functioning user mode systemd instance.

There's no dependency on the D-Bus session bus for many operations, even with more recent systemd. For instance, on 20.04:

$ strace --trace=connect systemctl --user is-active snapd.session-agent.service connect(3, {sa_family=AF_UNIX, sun_path="/run/user/1000/systemd/private"}, 33) = 0 inactive +++ exited with 3 +++

It is communicating with systemd over a private socket.

tests/main/snap-user-service-start-on-install/task.yaml

tests/main/snap-user-service-upgrade-failure/task.yaml

tests/main/snap-user-service/task.yaml

…ion-tool

"session-tool --prepare" was only trying to start default.target if it believed there was a systemd controlled D-Bus session bus. This is not the case on Ubuntu 16.04, yet it still has a working user instance of systemd. Instead try running "systemctl --user is-enabled default.target", which should only succeed if we can talk to the user instance of systemd.

anonymouse64 · 2020-07-29T16:58:14Z

Unfortunately I don't think this was cherry-picked to 2.45, but it will be in 2.46 now.

EndrII · 2020-11-06T06:57:05Z

Unfortunately I don't think this was cherry-picked to 2.45, but it will be in 2.46 now.

The current snapd version is 2.47. Why this feature still not merged?

anonymouse64 · 2020-11-25T17:51:58Z

The current snapd version is 2.47. Why this feature still not merged?

This feature was merged to master on May 7. If you have more questions about this feature, I would suggest creating a new forum post at forum.snapcraft.io, where more folks will see your questions.

zyga previously approved these changes Sep 12, 2018

View reviewed changes

wrappers/services.go Show resolved Hide resolved

jhenstridge force-pushed the user-daemons branch 2 times, most recently from 740f111 to 831eb09 Compare September 13, 2018 11:19

jhenstridge force-pushed the user-daemons branch 4 times, most recently from 371434b to 9d88f75 Compare September 14, 2018 01:11

mvo5 reviewed Sep 17, 2018

View reviewed changes

jhenstridge force-pushed the user-daemons branch 4 times, most recently from 48be212 to 6a7a002 Compare September 18, 2018 15:21

niemeyer previously requested changes Oct 8, 2018

View reviewed changes

jhenstridge force-pushed the user-daemons branch 2 times, most recently from 8df294d to 58465fe Compare November 23, 2018 03:11

jdstrand reviewed Nov 29, 2018

View reviewed changes

wrappers/services_test.go Show resolved Hide resolved

jdstrand reviewed Nov 29, 2018

View reviewed changes

jhenstridge mentioned this pull request Dec 4, 2018

interfaces: support D-Bus activation #6258

Closed

pedronis self-requested a review December 20, 2018 10:32

jhenstridge force-pushed the user-daemons branch from 58465fe to b659e12 Compare January 2, 2019 03:08

pedronis approved these changes Apr 20, 2020

View reviewed changes

jhenstridge added 2 commits April 20, 2020 15:54

cmd/snap-preseed: remove user daemon service files on reset

82e016a

Merge remote-tracking branch 'upstream/master' into user-daemons

f89220b

zyga reviewed Apr 21, 2020

View reviewed changes

tests: add some debug commands to snap-user-service spread test

b1016df

jhenstridge force-pushed the user-daemons branch from b625a4d to b1016df Compare April 23, 2020 05:41

jhenstridge added 2 commits May 4, 2020 12:18

Merge remote-tracking branch 'upstream/master' into user-daemons

d44a7cc

tests: port user daemons spread tests to session-tool

6144f2f

jhenstridge force-pushed the user-daemons branch from e713262 to 6144f2f Compare May 4, 2020 09:38

zyga reviewed May 4, 2020

View reviewed changes

tests/main/snap-user-service/task.yaml Show resolved Hide resolved

tests/main/snap-user-service-upgrade-failure/task.yaml Outdated Show resolved Hide resolved

tests/main/snap-user-service-socket-activation/task.yaml Outdated Show resolved Hide resolved

Merge remote-tracking branch 'upstream/master' into user-daemons

98df7bc

jhenstridge force-pushed the user-daemons branch from 3ded2e3 to 98df7bc Compare May 5, 2020 07:28

jhenstridge added 3 commits May 5, 2020 16:40

tests: add some debug output to snap-user-service-upgrade-failure test

991d61f

Merge remote-tracking branch 'upstream/master' into user-daemons

ea5ff93

wrappers: try making a request to the session agent before stopping it

d16c44f

This is an attempt to either fix the intermittent hang during tear down, or make it fail reliably.

jhenstridge force-pushed the user-daemons branch from 30b1079 to d16c44f Compare May 6, 2020 07:51

jhenstridge and others added 2 commits May 6, 2020 18:09

tests: add debug stanza to all snap-user-service spread tests

cf197f8

Merge branch 'master' of git://github.com/snapcore/snapd into user-da…

bf5fa36

…emons

zyga reviewed May 6, 2020

View reviewed changes

jhenstridge added 2 commits May 7, 2020 11:10

tests: no need to wait for default.target or sockets.target with sess…

10f582a

…ion-tool

mvo5 merged commit 21050a4 into snapcore:master May 7, 2020

jhenstridge mentioned this pull request May 18, 2020

schema: add support for the daemon-scope property on apps canonical/snapcraft#3129

Closed

4 tasks

anonymouse64 modified the milestones: 2.45, 2.46 Jul 29, 2020

wrappers: allow user mode systemd daemons #5822

wrappers: allow user mode systemd daemons #5822

Conversation

jhenstridge commented Sep 12, 2018

zyga left a comment

Choose a reason for hiding this comment

jhenstridge commented Sep 12, 2018

jhenstridge commented Sep 13, 2018

jhenstridge commented Sep 14, 2018

mvo5 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niemeyer left a comment • edited

Choose a reason for hiding this comment

jhenstridge commented Nov 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdstrand Nov 29, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdstrand Nov 29, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdstrand left a comment

Choose a reason for hiding this comment

jhenstridge commented Nov 30, 2018

jdstrand commented Nov 30, 2018

pedronis commented Jan 23, 2019

pedronis left a comment

Choose a reason for hiding this comment

jhenstridge commented Apr 21, 2020

zyga left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zyga left a comment

Choose a reason for hiding this comment

zyga commented May 6, 2020

zyga left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anonymouse64 commented Jul 29, 2020

EndrII commented Nov 6, 2020

anonymouse64 commented Nov 25, 2020

niemeyer left a comment •

edited

jdstrand Nov 29, 2018 •

edited

jdstrand Nov 29, 2018 •

edited

zyga left a comment •

edited