Authentication error after activate pam_tcb

[Izorkin]

Steps taken:

1. Build shadow 4.13 with the parameters --with-tcb --with-bcrypt --with-yescrypt.
2. Replaced pam_unix.so with pam_tcb.so in /etc/pam.d/* files:

# Account management.
account required /nix/store/mc3j8gfm9jafs2ywy9xq96whg10fmn3k-tcb-1.2/lib/security/pam_tcb.so shadow fork helper=/run/wrappers/bin/tcb_chkpwd

# Authentication management.
auth sufficient /nix/store/mc3j8gfm9jafs2ywy9xq96whg10fmn3k-tcb-1.2/lib/security/pam_tcb.so nullok shadow fork helper=/run/wrappers/bin/tcb_chkpwd
auth required pam_deny.so

# Password management.
password sufficient /nix/store/mc3j8gfm9jafs2ywy9xq96whg10fmn3k-tcb-1.2/lib/security/pam_tcb.so use_authtok shadow fork nullok write_to=tcb helper=/run/wrappers/bin/tcb_chkpwd

# Session management.
session required pam_env.so conffile=/etc/pam/environment readenv=0
session required /nix/store/mc3j8gfm9jafs2ywy9xq96whg10fmn3k-tcb-1.2/lib/security/pam_tcb.so helper=/run/wrappers/bin/tcb_chkpwd

3. Changed permissions for passwd and tcb_chkpwd:

-r-x--s--x 1 root shadow 17K Dec  4 21:59 /run/wrappers/bin/passwd
-r-x--s--x 1 root shadow 17K Dec  4 21:59 /run/wrappers/bin/tcb_chkpwd

4. Changed file /etc/nsswitch.conf:

shadow:    tcb

5. Added lines in /etc/login.defs:

USE_TCB yes
TCB_AUTH_GROUP yes

6. Added group auth
7. Running tcb_convert

After all changes, when I try to log in I get an error:

login[3552]: pam_tcb(login:auth): Credentials for user root unknown
login[3033]: pam_tcb(login:auth): Authentication failed for UNKNOWN USER from (uid=0)

[Izorkin]

Error log if change password.
From root:

passwd: Permission denied
passwd: password unchanged

passwd[5745]: pam_tcb(passwd:chauthtok): Username obtained: root

From user:

Changing password for example-user.
Current password:
passwd: Permission denied
passwd: password unchanged

passwd[5842]: pam_tcb(passwd:chauthtok): Username obtained: example-user
tcb_chkpwd[5845]: user unknown
passwd[5842]: pam_tcb(passwd:chauthtok): Authentication failed for example-user from example-user(uid=1000), for password management

cc @solardiz

[solardiz]

@Izorkin I don't know what exactly went wrong.

Did pam_tcb work for you in its pam_unix compatibility mode before you proceeded to fully convert to tcb? This is something I'd have tested first.

Anyway, I suggest you check file permissions:

ls -ld / /etc /etc/passwd /etc/shadow /etc/tcb /etc/tcb/root /etc/tcb/root/shadow

[Izorkin]

Did pam_tcb work for you in its pam_unix compatibility mode before you proceeded to fully convert to tcb?

What parameters are responsible for this? I am currently trying out tcb on a test virtual machine.

ls -ld / /etc /etc/passwd /etc/shadow /etc/tcb /etc/tcb/root /etc/tcb/root/shadow

drwxr-xr-x 17 root root 4096 Dec 4 15:27 /
drwxr-xr-x 25 root root 4096 Dec 5 15:03 /etc
-rw-r--r-- 1 root root 3784 Dec 5 15:03 /etc/passwd
-rw-r----- 1 root shadow 1096 Dec 5 15:03 /etc/shadow
drwx--x--- 47 root shadow 4096 Dec 4 21:29 /etc/tcb
drwx--s--- 2 root auth 4096 Dec 4 21:29 /etc/tcb/root
-rw-r----- 1 root auth 124 Dec 4 21:29 /etc/tcb/root/shadow

[solardiz]

My best guess is you didn't install the NSS module in place, so it's not picked up by your glibc... and do you even use glibc?

Did pam_tcb work for you in its pam_unix compatibility mode before you proceeded to fully convert to tcb?

What parameters are responsible for this?

Kind of none - this was supposed to work on its own, by simply replacing pam_unix with pam_tcb and not making any other changes yet.

[Izorkin]

Yes, using glibc. NSS loads library from PATH?

[Izorkin]

I was able to load libnss_tcb module. Local login does not work, but I can log in via SSH.
Log error:

systemd[1]: Starting User Manager for UID 0...
systemd[1315]: PAM failed: Insufficient credentials to access authentication data
systemd[1315]: user@0.service: Failed to set up PAM session: Operation not permitted
systemd[1315]: user@0.service: Failed at step PAM spawning /nix/store/2609pv1z9yfak7g6jw46jffi1pzkqfi1-systemd-251.7/lib/systemd/systemd: Operation not permitted
systemd[1]: user@0.service: Main process exited, code=exited, status=224/PAM
systemd[1]: user@0.service: Failed with result 'exit-code'.
systemd[1]: Failed to start User Manager for UID 0.
systemd[1]: Started Session 3 of User root.

[solardiz]

Yes, using glibc. NSS loads library from PATH?

No. (That would be a vulnerability when NSS is used by a SUID/SGID program.)

I guess glibc only checks one or a few hard-coded locations. We normally install libnss_tcb.so.2 into the same directory with glibc's own NSS modules.

[Izorkin]

I was able to load the module with the LD_LIBRARY_PATH environment:

# /etc/systemd/system/nscd.service
[Unit]
Before=nss-lookup.target nss-user-lookup.target
Description=Name Service Cache Daemon
Wants=nss-lookup.target nss-user-lookup.target
X-Restart-Triggers=/nix/store/div98sxg0d4n5idp1y9x07pbqvhd0k12-hosts /nix/store/nnmlhn3zbskrabpp7c5y5ixrrqvz0rrr-etc-nsswitch.conf /nix/store/f43r03jpnwnzm12al7286nizrav6ad9a-etc-nscd.conf

[Service]
Environment="LD_LIBRARY_PATH=/nix/store/mc3j8gfm9jafs2ywy9xq96whg10fmn3k-tcb-1.2/lib:/nix/store/2609pv1z9yfak7g6jw46jffi1pzkqfi1-systemd-251.7/lib"
Environment="LOCALE_ARCHIVE=/nix/store/xawd954bsgnqp0bixhnwyl1kswrqfki2-glibc-locales-2.35-163/lib/locale/locale-archive"
Environment="PATH=/nix/store/a7gvj343m05j2s32xcnwr35v31ynlypr-coreutils-9.1/bin:/nix/store/mydc6f4k2z73xlcz7ilif3v2lcaiqvza-findutils-4.9.0/bin:/nix/store/86bp03jkmsl6f92w0yzg4s59g5mhxwmy-gnugrep-3.7/bin:/nix/store/89zs7rms6x00xfq4dq6m7mjnhkr8a6r4-gnused-4.8/bin:/nix/store/2609pv1z9yfak7g6jw46jffi1pzkqfi1-systemd-251.7/bin:/nix/store/a7gvj343m05j2s32xcnwr35v31ynlypr-coreutils-9.1/sbin:/nix/store/mydc6f4k2z73xlcz7ilif3v2lcaiqvza-findutils-4.9.0/sbin:/nix/store/86bp03jkmsl6f92w0yzg4s59g5mhxwmy-gnugrep-3.7/sbin:/nix/store/89zs7rms6x00xfq4dq6m7mjnhkr8a6r4-gnused-4.8/sbin:/nix/store/2609pv1z9yfak7g6jw46jffi1pzkqfi1-systemd-251.7/sbin"
Environment="TZDIR=/nix/store/i16lgq16av602nfyws3ps8dd9yj36dwh-tzdata-2022f/share/zoneinfo"



ExecReload=/nix/store/57xv61c5zi8pphjbcwxxjlgc34p61ic9-glibc-2.35-163-bin/bin/nscd --invalidate passwd
ExecReload=/nix/store/57xv61c5zi8pphjbcwxxjlgc34p61ic9-glibc-2.35-163-bin/bin/nscd --invalidate group
ExecReload=/nix/store/57xv61c5zi8pphjbcwxxjlgc34p61ic9-glibc-2.35-163-bin/bin/nscd --invalidate hosts
ExecStart=!@/nix/store/57xv61c5zi8pphjbcwxxjlgc34p61ic9-glibc-2.35-163-bin/bin/nscd nscd
Group=nscd
NoNewPrivileges=true
PIDFile=/run/nscd/nscd.pid
PrivateTmp=true
ProtectHome=read-only
ProtectSystem=strict
RemoveIPC=true
Restart=always
RestrictSUIDSGID=true
RuntimeDirectory=nscd
Type=forking
User=nscd

[solardiz]

I was able to load the module with the LD_LIBRARY_PATH environment

That's good to know. Please note that LD_LIBRARY_PATH is ignored by SUID/SGID programs, avoiding the vulnerability I mentioned, but also being unusable to tell such programs, like passwd, to get an NSS module loaded. So that's not what you can really use here, beyond experimenting/testing.

/etc/systemd/system/nscd.service

That's the caching service only, and I think it's obsolete anyway. We don't normally run it at all on systems with tcb.

I think you just need to put the NSS module in place... but that may be inconsistent with NixOS philosophy? If so, maybe add /nix/store/mc3j8gfm9jafs2ywy9xq96whg10fmn3k-tcb-1.2/lib to /etc/ld.so.conf and rerun ldconfig.

I'll close this issue now since it's not an issue we need to track (nothing wrong with tcb, you're just installing it in an unintended way), but please feel free to continue adding comments anyway.

[Izorkin]

If so, maybe add /nix/store/mc3j8gfm9jafs2ywy9xq96whg10fmn3k-tcb-1.2/lib to /etc/ld.so.conf and rerun ldconfig.

This variant not working.
trace_passwd.log

[solardiz]

This variant not working.
trace_passwd.log

I'm not sure exactly why not, but here are two observations:

1. In the first comment, you show you specified:

# Password management.
password sufficient /nix/store/mc3j8gfm9jafs2ywy9xq96whg10fmn3k-tcb-1.2/lib/security/pam_tcb.so use_authtok shadow fork nullok write_to=tcb helper=/run/wrappers/bin/tcb_chkpwd

use_authtok assumes you have a previously stacked module ask for the new password, but you don't seem to have a prior module? Here's an example working use like we had it on Owl:

#%PAM-1.0
# $Owl: Owl/packages/pam/system-auth.pam,v 1.5 2011/07/17 10:05:28 solar Exp $
auth       required	pam_tcb.so shadow fork nullok prefix=$2y$ count=8
account    required	pam_tcb.so shadow fork
password   required	pam_passwdqc.so config=/etc/passwdqc.conf
password   required	pam_tcb.so use_authtok shadow write_to=tcb fork nullok prefix=$2y$ count=8
session    required	pam_tcb.so

If you omit the prior module (in our case, pam_passwdqc), then also omit use_authtok.

2. In the strace log, paths to libnss_tcb somehow vary:

openat(AT_FDCWD, "/nix/store/mc3j8gfm9jafs2ywy9xq96whg10fmn3k-tcb-1.2/lib/libnss_tcb.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/nix/store/mc3j8gfm9jafs2ywy9xq96whg10fmn3k-tcb-1.2/lib/libnss_tcb.so.2", O_RDONLY|O_CLOEXEC) = 3
[pid  1729] openat(AT_FDCWD, "/nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libnss_tcb.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libnss_tcb.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

[Izorkin]

Manually copying the library libnss_tcb.so.2 to /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/ helped fix the bug in operation!
My need to find a way to load library from /nix/store/mc3j8gfm9jafs2ywy9xq96whg10fmn3k-tcb-1.2/lib/

[Izorkin]

@solardiz is it safe to leave the tcb_chkpwd file in the /run/wrappers/bin directory?

-r-x--s--x 1 root shadow 17064 Dec  5 22:03 /run/wrappers/bin/tcb_chkpwd

Or move it to /run/wrappers/bin/chkpwd/

[solardiz]

is it safe to leave the tcb_chkpwd file in the /run/wrappers/bin directory?

-r-x--s--x 1 root shadow 17064 Dec  5 22:03 /run/wrappers/bin/tcb_chkpwd

Or move it to /run/wrappers/bin/chkpwd/

The extra directory layer provides an extra level of security separation, if you restrict permissions on the directory such that its content is only accessible by a group and you grant that group membership to just the right programs.

For example, you can set the directory permissions to 710 with group chkpwd and you then make programs needing this functionality (and only them) SGID chkpwd. From Owl documentation (on compatibility with third-party packages):

        Password checking helpers (xlockmore, xscreensaver).

On Red Hat Linux, the PAM modules' helper programs used to check a
user's own password are world-accessible.  On Owl, they're restricted
to group chkpwd.  This means that certain programs which need the
functionality need to be granted this new privilege.  This is known to
be needed for PAM-aware X screen locking programs such as the one
coming from package xlockmore.  To make it work on Owl, it needs to be
made SGID to group chkpwd, like this:

-rwx--s--x    1 root     chkpwd     751328 Mar 30  2001 /usr/X11R6/bin/xlock

Unfortunately, this trick does not appear to work with the newer
xscreensaver package.  xscreensaver appears to be too smart and drops
its additional privilege.

For the GNU screen package coming from Owl itself, we have:

# ls -la /usr/libexec/screen/
total 28
drwx--x---  2 root screen  4096 May 29  2016 .
drwxr-xr-x 12 root root    4096 May 29  2016 ..
-rwx--s--x  2 root shadow 10008 Jan 11  2015 tcb_chkpwd
-rwx--s--x  2 root utmp    5912 Jan 11  2015 utempter

and

-rwx--s--x 1 root screen 358008 Jan 11  2015 /usr/bin/screen

that's a trick to provide an SGID binary access to two limited-access SGID helper programs at once. Please note that we use hard-links there (note the link counts of 2 on the program binaries), managed by triggers in the corresponding packages. This is tricky, and can only reasonably be pulled off at distro level, with agreed upon conventions and with packages playing together well.

So if you're not going to modify other packages, you probably have to make this helper world-accessible. This isn't meant to introduce a vulnerability - it is just not the safest mode possible.

Summary: yes, it's kind of safe, and the alternative is probably overly complicated in your case.

[Izorkin]

Thanks for details!

Is there a patch to integrate tcb support into glibc?

[solardiz]

Is there a patch to integrate tcb support into glibc?

No, I think there is not.

[Izorkin]

No, I think there is not.

Eh, it would be easier to activate tcb support in NixOS.

[solardiz]

-rw-r----- 1 root shadow 1096 Dec 5 15:03 /etc/shadow

Please note that for security you need to remove /etc/shadow once you're comfortable with the conversion to tcb.

[Izorkin]

Did I describe the purpose of groups correctly?

  auth = 327; # Group that is allowed access to tcb shadow files
  chkpwd = 328; # Group that is allowed to change tcb shadow files
  sys = 329; # Group to temporarily lock all of tcb shadow files

[solardiz]

Did I describe the purpose of groups correctly?

No, maybe except for sys.

auth is a group that, only together with group shadow, grants read-only access to other users' shadow files. Most systems don't need this functionality, but it's OK to provide it.

chkpwd is a group that, if access to tcb_chkpwd is restricted to it, allows programs (typically screen locking ones) to check the user's password. As discussed, NixOS is unlikely to enable such restriction, and thus wouldn't need this group to exist at all.

sys is a group that commonly readily exists on systems even without our tcb, and which we reuse to temporarily block access to /etc/tcb during (un)conversion of a system between legacy shadow and tcb modes.

[Izorkin]

Thanks.

[Izorkin]

sys is a group that commonly readily exists on systems even without our tcb, and which we reuse to temporarily block access to /etc/tcb during (un)conversion of a system between legacy shadow and tcb modes.

The sys group does not exist in nixos. There are these standard groups:

root
wheel
disk
kmem
tty
floppy
uucp
lp
cdrom
tape
audio
video
dialout
nogroup
users
nixbld
utmp
adm
input
kvm
render
sgx
shadow

[flokli]

@solardiz we use nscd as a /non-caching/ NSS lookup dispatcher in NixOS.

The whole reasoning is documented in https://flokli.de/posts/2022-11-18-nsncd/.

Can you see any reason to why doing NSS lookups that way would not work?

[ldv-alt]

Since nscd is a threaded process, the use of static variables in libnss_tcb via libtcb might be problematic.
Anyway, please make sure you build tcb with -DENABLE_SETFSUGID, otherwise libtcb would use setreuid and setregid which affect the whole nscd process.

[Izorkin]

@ldv-alt require building package using cmake? Is there an alternative way to use this parametr?
When trying to build using cmake, an error is generated that the CMakeLists.txt file was not found.:

tcb> CMake Warning:
tcb>   Ignoring extra path from command line:
tcb>    ".."
tcb>
tcb> CMake Error: The source directory "/build/source" does not appear to contain CMakeLists.txt.
tcb> Specify --help for usage, or press the help button on the CMake GUI.

[ldv-alt]

As tcb can be built using a regular make, there is no cmake support.
By building with -DENABLE_SETFSUGID I mean compiling with ENABLE_SETFSUGID defined.

[Izorkin]

Should it be like this?

...
tcb> gcc -DENABLE_SETFSUGID  -I../include -Wall -Wextra -fPIC -c libtcb.c -o libtcb.o
tcb> gcc  -L../libs -shared -o libtcb.so.0.9.8 -Wl,-soname,libtcb.so.0 \
tcb>    -Wl,--version-script=libtcb.map libtcb.o
...

[ldv-alt]

Yes, I mean something like gcc -DENABLE_SETFSUGID.
Note that in the cited snippet tcb is being compiled without optimization, this looks unusual.

[Izorkin]

What parameters to add for optimization?

[ldv-alt]

On Sat, Dec 24, 2022 at 03:27:03PM -0800, Dmitry V. Levin wrote: Since nscd is a threaded process, the use of static variables in libnss_tcb via libtcb might be problematic. Anyway, please make sure you build tcb with -DENABLE_SETFSUGID, otherwise libtcb would use `setreuid` and `setregid` which affect the whole nscd process.

I've had a look, even with -DENABLE_SETFSUGID set both tcb_drop_priv_r() and tcb_gain_priv_r() are not quite thread-safe because of setgroups() calls which per POSIX affect all threads. This is probably not a big deal for nscd, though, as the worst thing that could happen to nscd is a permanent loss of supplementary groups.

[ldv-alt]

On Sun, Dec 25, 2022 at 02:34:02AM -0800, Yurii Izorkin wrote: What parameters to add for optimization?

Whatever you use for security sensitive software should be OK.

[Izorkin]

Compiled tcb with the flag -DENABLE_SETFSUGID flag. NSCD service still can't find library.
Most likely the problem is not with tcb.

[flokli]

@Izorkin i don't think this is still on-topic. Let's take this over to NixOS/nixpkgs#109457.

[Izorkin]

@solardiz found another error - unable to change password for normal user.

cat /etc/pam.d/passwd

# Account management.
account required /nix/store/0l6vx05sv179hs3kwbnc3wrkb6rsxqv3-tcb-1.2/lib/security/pam_tcb.so shadow fork helper=/run/wrappers/bin/tcb_chkpwd audit debug

# Authentication management.
auth sufficient /nix/store/0l6vx05sv179hs3kwbnc3wrkb6rsxqv3-tcb-1.2/lib/security/pam_tcb.so nullok shadow fork helper=/run/wrappers/bin/tcb_chkpwd audit debug
auth required pam_deny.so

# Password management.
password sufficient /nix/store/0l6vx05sv179hs3kwbnc3wrkb6rsxqv3-tcb-1.2/lib/security/pam_tcb.so shadow fork nullok write_to=tcb helper=/run/wrappers/bin/tcb_chkpwd audit debug

# Session management.
session required pam_env.so conffile=/etc/pam/environment readenv=0
session required /nix/store/0l6vx05sv179hs3kwbnc3wrkb6rsxqv3-tcb-1.2/lib/security/pam_tcb.so helper=/run/wrappers/bin/tcb_chkpwd audit debug

The use_authtok option is not used.

Error:

passwd[1190]: pam_tcb(passwd:chauthtok): Unable to find user in the selected database

[Izorkin]

Sorry, I disabled the password change in the NixOS configuration and forgot about it. It's working now.

[Izorkin]

Can these errors be ignored?

LD_DEBUG=all passwd
...
   1143647:     /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2: error: symbol lookup error: undefined symbol: _nss_tcb_initgroups_dyn (fatal)
   1143647:     symbol=_nss_tcb_netname2user;  lookup in file=/nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2 [0]
   1143647:     symbol=_nss_tcb_netname2user;  lookup in file=/nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libtcb.so.0 [0]
   1143647:     symbol=_nss_tcb_netname2user;  lookup in file=/nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/libc.so.6 [0]
   1143647:     symbol=_nss_tcb_netname2user;  lookup in file=/nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/ld-linux-x86-64.so.2 [0]
   1143647:     /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2: error: symbol lookup error: undefined symbol: _nss_tcb_netname2user (fatal)
   1143647:     symbol=_nss_tcb_setaliasent;  lookup in file=/nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2 [0]
   1143647:     symbol=_nss_tcb_setaliasent;  lookup in file=/nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libtcb.so.0 [0]
   1143647:     symbol=_nss_tcb_setaliasent;  lookup in file=/nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/libc.so.6 [0]
   1143647:     symbol=_nss_tcb_setaliasent;  lookup in file=/nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/ld-linux-x86-64.so.2 [0]
...

[solardiz]

Can these errors be ignored?

I'm surprised you're getting those, but yes I guess they can be ignored if everything works for you.

[solardiz]

even with -DENABLE_SETFSUGID set both tcb_drop_priv_r() and tcb_gain_priv_r() are not quite thread-safe because of setgroups() calls which per POSIX affect all threads.

@ldv-alt Should we possibly switch to using the setgroups() syscall instead of the glibc function? Should we make -DENABLE_SETFSUGID the default, or even make it the only implementation?

[ldv-alt]

even with -DENABLE_SETFSUGID set both tcb_drop_priv_r() and tcb_gain_priv_r() are not quite thread-safe because of setgroups() calls which per POSIX affect all threads.

@ldv-alt Should we possibly switch to using the setgroups() syscall instead of the glibc function? Should we make -DENABLE_SETFSUGID the default, or even make it the only implementation?

I'm inclined to agree. At least, I see no potential benefits in keeping the non-ENABLE_SETFSUGID variant,
and it definitely makes sense to use the setgroups() syscall instead of the glibc function.