PAM sessions are not being initialized properly

[gclen]

I have a configuration where mount points are being set up when a user authenticates with PAM. This worked on jupyterhub 0.8.1. However, this does not appear to work on newer versions of jupyterhub (I upgraded to 0.9.6).

I think the issue can be traced back to this commit where the interaction with PAM was moved to a thread. I think having authentication in a separate thread is probably okay (though I haven't tested this). However, opening a PAM session should probably be done in the spawned process since they are typically setting process/thread attributes. Reverting the changes in jupyterhub/auth.py fixed the issue for me but I think a proper fix is needed (this fix may just be reverting the changes).

I think taking another look at the commit above may also help with issues #2056, #2754, and #2406. Let me know if you need me to do more testing or need any other information.

[gatoniel]

Hi,

I am trying to use PAMAuthenticator with a non-root user JuypterHub controlled by systemd on CentOS 8. Since I am going to spawn singleuser servers with podman, I need PAM to open a session to set the correct context for each user. Right now this does not work, and I get several PAM related errors. This is the /etc/var/secure log

Mar 31 16:04:51 picard python3[9768]: pam_loginuid(login:session): Error writing /proc/self/loginuid: Operation not permitted
Mar 31 16:04:51 picard python3[9768]: pam_loginuid(login:session): set_loginuid failed
Mar 31 16:04:51 picard python3[9768]: pam_selinux(login:session): Error sending audit message.
Mar 31 16:04:51 picard python3[9768]: pam_systemd(login:session): Failed to create session: Access denied
Mar 31 16:04:51 picard python3[9768]: pam_unix(login:session): session opened for user testuser by (uid=991)

And this is the service file for JupyterHub for systemd:

[Unit]
Description=JupyterHub
After=syslog.target network.target podman-chp.service

[Service]
User=jupyterhub
AmbientCapabilities=CAP_SETUID CAP_SETGID
Environment="PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/jupyterhub/bin"
Environment="CONFIGPROXY_AUTH_TOKEN=754b5de95588a1c729d417809d3925cca09e61fc0255015d93b54d154ec6885b"
ExecStart=/opt/jupyterhub/bin/jupyterhub -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py

[Install]
WantedBy=multi-user.target

To make sure that /proc/self/loginuid is unset, I modified the pre_spawn_start function:

@run_on_executor
    def pre_spawn_start(self, user, spawner):
        """Open PAM session for user if so configured"""
        if not self.open_sessions:
            return
        try:
            proc = Popen(["cat", "/proc/self/loginuid"], stdout=PIPE)
            out, err = proc.communicate()
            self.log.info(out)
            pamela.open_session(user.name, service=self.service, encoding=self.encoding)
        except pamela.PAMError as e:
            self.log.warning("Failed to open PAM session for %s: %s", user.name, e)
            self.log.warning("Disabling PAM sessions from now on.")
            self.open_sessions = False

The /var/log/messages output was

Mar 31 16:04:51 picard jupyterhub[9768]: [I 2020-03-31 16:04:51.042 JupyterHub auth:956] b'4294967295'
Mar 31 16:04:51 picard dbus-daemon[1999]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.14225" (uid=991 pid=9768 comm="/opt/jupyterhub/bin/python3 /opt/jupyterhub/bin/ju" label="system_u:system_r:unconfined_service_t:s0") interface="org.freedesktop.login1.Manager" member="CreateSession" error name="(unset)" requested_reply="0" destination="org.freedesktop.login1" (uid=0 pid=2091 comm="/usr/lib/systemd/systemd-logind " label="system_u:system_r:systemd_logind_t:s0")
Mar 31 16:04:51 picard jupyterhub[9768]: [W 2020-03-31 16:04:51.078 JupyterHub auth:959] Failed to open PAM session for testuser: [PAM Error 14] Cannot make/remove an entry for the specified session
Mar 31 16:04:51 picard jupyterhub[9768]: [W 2020-03-31 16:04:51.078 JupyterHub auth:960] Disabling PAM sessions from now on.

The first line 4294967295 shows that loginuid is not set yet, as expected. The JupyterHub error log shows, that we have PAM Error 14.

I tried adding all of this additional CAPABILITIES in the service file: CAP_SYS_ADMIN CAP_MAC_ADMIN CAP_AUDIT_CONTROL CAP_CHOWN CAP_KILL CAP_DAC_READ_SEARCH CAP_DAC_OVERRIDE CAP_FOWNER CAP_SYS_TTY_CONFIG, but this did not change anything in the errors. Right now I am trying to resolve this issue. There are three possible directions for me:

1. The PAM error is SELinux related: Maybe I need to change the SELinux context of the JupyterHub process somehow. From the logs above I get, that the context is system_u:system_r:unconfined_service_t.
2. Maybe I need to add some more CAPABILITIES to the JupyterHub process.
3. Maybe @gclen is right and the PAM open session call, should be run somewhere else. @gclen , did you just remove the @run_on_executor decorator to revert changes? Then I could test your solution.

In addition, I think all three directions affect the PAM errors.

When I put c.PAMAuthenticator.open_sessions = False my instance runs and works with some workarounds, but I would like to use PAM session, because its more elegant.

[gclen]

I just reverted all of the changes in this commit: 4de6b39 and it seemed to work for me. If it also works for you it might be worth submitting a pull request with your changes.

[gclen]

In theory all you need to do is add

from tornado import gen

and change https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/auth.py#L948 and https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/auth.py#L960 to

@gen.coroutine

instead of

@run_on_executor

but I haven't tested this (and currently don't have access to my normal setup)

[gatoniel]

Ok, I took on my own way:

1. in my oppinion the pam_open_session call should definitely be made by a child of the JupyterHub process, since depending on the PAM service it alters /proc/self/loginuid. This should not be altered for the main process. I have not found a good way to do this: right now I make the call in the preexec_fn of subprocess.Popen.
2. I wrote a SELinux module to test, if my errors are related to SELinux. This made it also possible to transition a spawned single user server to a different domain.
3. I did not need to add new capabilities.

But I am still suck with one error. Here from /var/log/secure:

python3[22778]: pam_systemd(login:session): Failed to create session: Access denied

And here from /var/log/messages:

dbus-daemon[1999]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.18841" (uid=1000 pid=22879 comm="/opt/jupyterhub/bin/python3 /opt/jupyterhub/bin/ju" label="system_u:system_r:jupyterhubd_t:s0") interface="org.freedesktop.login1.Manager" member="CreateSession" error name="(unset)" requested_reply="0" destination="org.freedesktop.login1" (uid=0 pid=2091 comm="/usr/lib/systemd/systemd-logind " label="system_u:system_r:systemd_logind_t:s0")

I used --disable_dontaudit to see all avc errors in the audit logs. There are no more errors. So I think the above mentioned error comes from dbus, systemd-logind or polkit. But I havent looked into them. Does anyone have a good ressource for these tools? I cant find any hint, how to enable a new program to communicate with systemd-logind.

[gatoniel]

Hi, I worked it out now:

I added a set_preexec_fn routine to spawners. It is called by the authenticator on pre_spawn_start and sets a preexec_fn for the spawner. When the spawner makes a call to the singleuser notebook via subprocess.Popen the new preexec_fn is used. This function changes to root, runs through the PAM stack, and then changes to the correct uid, gids. The PAM handle is unique for the session and gets saved in the spawner, so it can be reused, when closing the PAM session in post_spawn_stop. This is needed, when users can start multiple singleuser notebooks, hence might have multiple open sessions. I forked jupyterhub and wrapspawner for my own usage and it works.

I think there is a more elegant way to make the authenticator run a pre_spawn_start hook in the beginning of the new subprocess of the spawner. Maybe someone else has a better idea?

[shawnweeks]

This is still broken on JupyterHub 1.4.2 when run with the sudospawner. Disabling PAM Sessions via c.PAMAuthenticator.open_sessions=False works fine.

[shawnweeks]

Something I found related to this is that on Ubuntu 20.04 server install is that you need to add the hub service user to the "utmp" group in addition to the "shadow" group. You still get some errors though. I'm going to keep digging into exactly what each step is doing.

Aug 30 23:47:17 mgmt-1 jupyterhub[23465]: sh: 1: cannot create /run/motd.dynamic.new: Permission denied
Aug 30 23:47:17 mgmt-1 python3[23449]: pam_keyinit(login:session): Unable to change GID to 1000 temporarily
Aug 30 23:47:17 mgmt-1 python3[23449]: pam_unix(login:session): session opened for user sweeks by (uid=998)
Aug 30 23:47:17 mgmt-1 python3[23449]: pam_systemd(login:session): Failed to create session: Access denied

[bkreider]

The shadow group means you can read all of the local hashed passwords. That's why it needs to run as root or be a member to work. It's related to your PAM configuration. If you use an LDAP backend and not shadow passwords, you can often run unprivileged when using PAM modules -- it all depends on what your PAM modules are doing.

That's why gatoniel is running in a preexec function.

[minrk]

Cross-linking #486 since this has been brought up several times, but our session lifecycle is not correct:

Quoting https://bugs.freedesktop.org/show_bug.cgi?id=62866:

A PAM service needs to invoke the PAM session hooks like this:

pam_open_session(h);
pid = fork();
if (pid == 0) {
exec();
}
waitpid(pid);
pam_close_session(h);
exit();

That's the only correct way. i.e. the PAM handle can only be used once, and both pam_open_session() and pam_close_session() need to be called in the parent -- not the child. Also, since the session hooks will set all kinds of stuff like resource limits, security labels, audit info, selinux labels, yadda yadda yadda, the parent must exit() after the close session hook.

So the only way to fix this fully robustly is to have a separate PAM process for each user (indeed for each Spawner instance) where we make PAM calls (including authenticate). It also means requiring password entry for every spawn if we want to avoid persisting passwords in a reusable form (which I think we should avoid), and hijacking LocalProcessSpawner so the authenticate process is indeed the parent of the single-user server.

There's further discussion in minrk/pamela#22 which includes a sketch of a path forward.