Tilde ('~') does not expand to home of user actually running service start script

[der-gabe]

I discovered this issue while debugging why the PostgreSQL service would not start in my Docker container using systemctl.py.

The service definition specifies that the startup script should be run as user postgres who exists in the container and has the home directory /var/lib/pgsql.

The startup script (/usr/lib/postgresql-init) apparently expects to be run as user postgres and, at one point, does the following:

cd ~

A few lines later, it attempts to initialise the DB (if that hasn't been done before) sending stdout to a file named initlog, in the local working directory.

That all works fine, when the script is run by regular systemd, but systemctl.py apparently sets up the environment differently, so that ~ does not expand to the postgres user's home, but rather to /root.

This leads to the following errors being logged by the startup script:

/usr/lib/postgresql-init: line 42: cd: /root: Permission denied
Initializing PostgreSQL 9.6.9 at location /var/lib/pgsql/data
/usr/lib/postgresql-init: line 52: initlog: Permission denied
Initialisation failed. See //initlog .

… and lets the initialisation (and thus the service start) fail.

I was able to work around this issue, by putting a drop-in unit file into /usr/lib/systemd/system/postgresql.service.d that explicitly sets

[Service]
Environment="HOME=/var/lib/pgsql"

… but I really consider this a workaround, rather than a fix.

Ideally, when systemctl.py runs a shell script as some user, the tilde character ~ should expand to that users home directory (as specified in /etc/passwd) if used within the script. If only, because that seems to be the way systemd does it and there's at least one startup script out there that relies on this behaviour.

[manics]

I ran into a similar issue a few months ago- one of my systemd services was failing because some standard environment variables were missing.

I made my own fix manics@524ce62 but didn't have time to submit it as a PR, you've just reminded me about it.

The more general problem is the handling of environment variables. The systemd docs say only a fixed set should be defined. I did some investigation using this service file env-var-test.service to dump the variables for different users:

[Unit]
Description=xxx

[Service]
User=root
Type=forking
ExecStart=/bin/sh -c "(date; echo; /usr/bin/env) > /tmp/env-var-test.log"

User=root

SHELL=/bin/sh
USER=root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
PWD=/
LANG=en_US.UTF-8
SHLVL=1
HOME=/root
LOGNAME=root
_=/usr/bin/env

User=vagrant

SHELL=/bin/bash
USER=vagrant
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
PWD=/
LANG=en_US.UTF-8
SHLVL=1
HOME=/home/vagrant
LOGNAME=vagrant
_=/usr/bin/env

This doesn't seem to match the systemd docs, so my above commit only added these

[gdraheim]

Oh well, that points back to the HISTORY of systemctl.py as a simple wrapper looking for ExecStart statements in the *.service files to be run. It was actually nice to inherit extra environment variables when checking services.

In any case, the environment variables referenced in the standard systemd documentation should be atleast on a value that matches expectations. So when $USER / $HOME should be overridden then it is the way to go.

[manics]

@gdraheim Do you want me to go ahead and open a PR with my fix once I've retested it? Or do you have some other solution in mind?

[TheTechsTech]

I had similar issues while compiling programs like freepbx under docker, I ended up doing something like this:

COPY systemctl.py /usr/bin/systemctl.py
RUN cp -f /usr/bin/systemctl /usr/bin/systemctl.original \
    && chmod +x /usr/bin/systemctl.py \
    && cp -f /usr/bin/systemctl.py /usr/bin/systemctl

I create the user, i needed an asterisk user

RUN export USER=xxxx && adduser xxxx -m -c "xxxx User" 

At this point i have my *.service files already copied over.
I use original for issues that come up.

RUN systemctl.original disable/enable

End build with:

RUN systemctl stop firewalld \
    && systemctl.original disable dbus firewalld \
    && (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \
    systemd-tmpfiles-setup.service ] || rm -f $i; done); \
    rm -f /lib/systemd/system/multi-user.target.wants/*; \
    rm -f /lib/systemd/system/local-fs.target.wants/*; \
    rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
    rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
    rm -f /lib/systemd/system/basic.target.wants/*; \
    rm -f /lib/systemd/system/anaconda.target.wants/*; \
    rm -f /etc/dbus-1/system.d/*; \
    rm -f /etc/systemd/system/sockets.target.wants/*; 

And other things I found to cause issues and not usable in docker.

Start container up with.

ENTRYPOINT ["/usr/bin/systemctl","default","--init"]

[gdraheim]

@manics .... reviewing your patch, I see that you had completely replaced the env={} . Instead I would like to simply override the variables that are defined in the standard and where programs/services may expect them to be set.

Your link to the standard docs is extremely helpful for that. =>
https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Environment%20variables%20in%20spawned%20processes

[gdraheim]

and please guys (include @techno-express ), do report such problems, there's no need to make up a workaround when it can get fixed in the upstream project. ;)

[gdraheim]

So, what do we have:

• $PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
• $LANG= ... actually setlocale() with the value from /etc/locale.conf
• $USER, $LOGNAME, $HOME, $SHELL .. only asserted when User= is present, so "postgres" is valid but "root" is not
• $INVOCATION_ID ... I have never heard of that, but a random() number seems easy
• $XDG_RUNTIME_DIR ... expected for user-mode only
• $MAINPID .... already done
• $MANAGERPID ... well, okay, but for non-init systemctl.py this is a volatile info
• $LISTEN_FDS, $LISTEN_PID, $LISTEN_FDNAMES ... I'd skip that as socket activation is not supported by systemctl.py
• $NOTIFY_SOCKET ... already done (for Type=notify services)
• $WATCHDOG_PID, $WATCHDOG_USEC ... keep-alive is not supported by systemctl.py
• $TERM ... "for units connected to a terminal", which is used for getty/rescue/reboot. We can keep the env value for that
• $JOURNAL_STREAM .. when "StandardError=journal" .. did not find any service to use it
• $SERVICE_RESULT ... already done (only valid in ExecStop / ExecStopPost)
• $EXIT_CODE, $EXIT_STATUS ... same here
  `
  So we are left with requirements for $PATH, $LANG, $INVOCATION_ID, $MANAGERPID ... and in set-user mode we have $USER, $LOGNAME, $HOME, $SHELL, $XDG_RUNTIME_DIR.

Probably we can also support $USER/$HOME for the default root-user as well.

[gdraheim]

I added some code, present in v1.4.2416, that may fix this problem.

I don't have testcase so far, so it is up to you to give it a try.

[gdraheim]

Already in release v1.4.2456