[feature-req] systemd based for hostname, datetime,... broken

[sjorge]

For example stuff like puppet, saltstack,... will trip over this. The tools in question are hostnamectl, datetimectl,...

There is an easy fix, they currently fail due to the lack of some cgroup and other stuff not being supported. But there is an easy workaround that survives upgrades and other changes.

      ## disable some functions because lx-brand (TODO: remove once implemented)
      for svc in systemd-timedated systemd-hostnamed; do
        override_dir=/etc/systemd/system/${svc}.service.d
        override_svc=${override_dir}/override.conf
        mkdir -p ${override_dir}

        echo "[Service]" > ${override_svc}
        echo "PrivateTmp=no" >> ${override_svc}
        echo "PrivateDevices=no" >> ${override_svc}
        echo "PrivateNetwork=no" >> ${override_svc}
        echo "ProtectSystem=no" >> ${override_svc}
        echo "ProtectHome=no" >> ${override_svc}

        systemctl daemon-reload
        systemctl restart ${svc}
      done

This creates override files for systemd units that get applied over the one provided by the package. This disables all isolation mechanism.

After this the daemons start up and all is well.

It would be great to have this happen in the building process. (This is also needed for systemd based debian and ubuntu images)

[chorrell]

Can this be characterized as an lx-brand issue? You mention:

due to the lack of some cgroup and other stuff not being supported

Is there an issue open for the missing cgroup etc support? I'd like to better understand what that is and if it's an known platform issue.

Also, it's not clear what's exactly is failing based on your description.

The hostnamectl and datetimectl commands are failing and they are used by puppet and saltstack?

[chorrell]

OK, confirmed that timedatectl and hostnamectl fail (at least on joyent_20160420T005724Z)

[chorrell]

Found this: https://smartos.org/bugview/OS-5304

[sjorge]

Those commands are the cli tooling on systemd based systems for basic config like TZ, hostname,...

Probably why cfgmgmt tools call them.

Having these working out of the box would be great.

They depend on a lot of things it seems. Not sure all those bits will get added to the brand code anytime soon.

The workaround is pretty generic though and pkgmgr friendly

[chorrell]

OK.

I'd like to better understand the settings we're disabling though. I'm assuming they are all cgroup related and probably relate to security/permissions. What's the downside for setting the following as opposed to their default settings?

PrivateTmp=no
PrivateDevices=no
PrivateNetwork=no
ProtectSystem=no
ProtectHome=no

[chorrell]

Some light reading for myself: https://www.freedesktop.org/software/systemd/man/systemd.exec.html

[chorrell]

And https://www.freedesktop.org/software/systemd/man/systemd.directives.html

[sjorge]

From memory:

# private fs namespace, /tmp and /var/tmp are only for the processes forked not for outsiders
PrivateTmp=no
# private fs namespace for /dev (I think)
PrivateDevices=no
# restrict processes' network access to loopback only
PrivateNetwork=no
# not sure, but it also needed to be no else it still failed
ProtectSystem=no
# restricts access /home and /root, they appear empty
ProtectHome=no

Meh close, your links are better.

[chorrell]

Right.

So the question I have is: Is overriding the defaults for those services worse than the services not working?

Probably not? It's not great that the services won't have the typical default restrictions, but given that timedatectl and hostnamectl fail without the overrides in pace it's probably a good compromise until this can be fixed in the platform.

[sjorge]

Exactly, It's not great that we have to disable it.

But it's better than cfgmgmt tools not working, other stuff that depends on it failing.

If I'm not mistaken some DE's (gnome) depend on this because the BSD's are implementing parts like hostnamed, logind,... to get newer gnome. Not that this is the primary use-case fpr systemd distro under lx. But it should work when the user installs it.

[chorrell]

Yeah, that's fair.

I'll work on getting these overrides in place for all the relevant lx-brand images (debian-8, ubuntu-16.04, basically all the ones with systemd)

[chorrell]

On a test ubuntu16.04 image I added the overrides but something isn't right:

# hostnamectl
Failed to create bus connection: No such file or directory

[chorrell]

# systemctl restart systemd-hostnamed
Failed to restart systemd-hostnamed.service: Unit dbus.socket not found.

[sjorge]

@chorrell looks like the 'dbus' package is not installed, I just checked the ubuntu 16.04 dataset. So it is failing on the dependant unit dbus.socket (which auto start dbus.service on first connect)

CentOS has the package intalled by default it seems, not sure about the debian dataset as I don't use that one.

[chorrell]

Yep, that seems to be it.

[sjorge]

Probably some odd upstream dependency issue, as the binaries for hostnamed seem to be provided as is. Unless dbus gets explicitly stripped by usjoyent.

[chorrell]

It's a pretty minimal install, so it might just not be included by default (I'm not omitting it).

[sjorge]

Ack, odd choice not to include dbus but to ship the systemd-xxxd services that depend on it... especially if it's something like the one dealing with hostname. Perhaps that should be a upstream bug.

[chorrell]

Maybe.

fwiw, the install is via debootstrap --components=main etc: https://github.com/joyent/ubuntu-lx-brand-image-builder/blob/master/install#L177

[sjorge]

I just did a minimal install 16.04 via PXE on a vm at work... dbus is missing too. (And our puppet boostapping failing). But $work is mostly a debian and RHEL shop, both have no issues with hostnamectl in a minimal install. shurgs

[chorrell]

Ha!

I've seen a lot of things like that with the various distros and what they consider "minimal" to be.

[sjorge]

I guess technically on a update config = destroy, create work flow. Setting the hostname once on install is all you need so a heavy dbus/systemd management system just for the hostname is overkill. (simple text files anyone? :p)

Anyway, thanks for looking into this so far. It's one ugly sticky mess.

[sjorge]

The nice thing is, if you place those files and install dbus afterwards it still works !

[chorrell]

Yeah, that was kind of a relief that it was that simple.

[chorrell]

The fix is in. New images should be available in a week or so.

[sjorge]

This should probably also go into debian-lx-brand-image-builder (for the debian-8 images)

[chorrell]

Yep, will be working on that too.

[sjorge]

Excellent!

[chorrell]

The following images were just released and they include the required systemd service overrides:

Ubuntu 16.04
https://images.joyent.com/images/675834a0-1267-11e6-81fe-2b1280d24244

CentOS 7
https://images.joyent.com/images/d61a1ef2-12db-11e6-ad97-770e083f1374

Debian 8
https://images.joyent.com/images/473a3a0c-12f2-11e6-acf9-4b8bbe1782ba

[sjorge]

Excellent, updated my base sets. Won't have time to re-provision today though.