(ssh) authentication fails for root with empty password on netboot image

[deliciouslytyped]

Update 1: su can be "fixed" by adding nullok to it's auth required pam_unix.so line. Why sshd is unhappy remains to be discovered.
Update 2: the same applies to sshd, I was using .ssh instead of .sshd by mistake.
=============
With the following module, and the default empty password:

{
  services.openssh = {
    enable = true;
    permitRootLogin = "yes";
    extraConfig = ''
      PermitEmptyPasswords yes
      '';
    };
  }

I can't access ssh, authentication fails with the following:

Apr 17 13:05:37 nixos sshd[1042]: Connection from 192.168.0.206 port 41760 on 192.168.0.32 port 22 rdomain ""
Apr 17 13:05:42 nixos sshd[1042]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.206  user=root
Apr 17 13:05:45 nixos sshd[1042]: Failed none for root from 192.168.0.206 port 41760 ssh2
Apr 17 13:05:47 nixos sshd[1042]: Postponed keyboard-interactive for root from 192.168.0.206 port 41760 ssh2 [preauth]
Apr 17 13:05:54 nixos sshd[1042]: error: PAM: Authentication failure for root from 192.168.0.206
Apr 17 13:05:54 nixos sshd[1042]: Failed keyboard-interactive/pam for root from 192.168.0.206 port 41760 ssh2
Apr 17 13:05:54 nixos sshd[1042]: Postponed keyboard-interactive for root from 192.168.0.206 port 41760 ssh2 [preauth

passwd reports the following:

root NP 01/02/1970 -1 -1 -1 -1

su root from another user also fails, but I didn't check for logs.

[deliciouslytyped]

It's not clear to me yet if there is any way to enable extended logging for pam somewhere, or if there are other debugging techniques for this. The pamtester tool is packaged but that basically only seems to tell you what is already visble in the sshd log. also only says "Authentication failure" as well.

[deliciouslytyped]

This doesn't seem to work even if UsePAM no is prepended to the sshd configuration.

[deliciouslytyped]

Some misc. notes:

So, I still don't know about how to make this more debuggable, but at least with respect to authentication on su and sshd with PAM, it should be observed that only the password lines have nullok set by default, not the auth lines, which are responsible for determining how the password is checked. (

nixpkgs/nixos/modules/security/pam.nix

Line 447 in 5a333c5

"auth sufficient pam_unix.so ${optionalString cfg.allowNullPassword "nullok"} ${optionalString cfg.nodelay "nodelay"} likeauth try_first_pass"}

)

nullok can be added to the auth section with security.pam.services.<name>.allowNullPassword = true;

This seems to exist since 5dfaf56 , and is enabled for a few services https://github.com/NixOS/nixpkgs/search?q=allownullpassword

• by what criteria is it enabled or not?
• What does Note that regardless of what the pam_unix2 documentation says, accounts with hashed empty passwords are always allowed to log in. mean?

Some orientational docs can be read at http://www.linux-pam.org/Linux-PAM-html/sag-configuration-file.html .

• however it seems it hasn't been updated since 2010 (this may just be the date not being updated - the source still seems to say 2010) as opposed to https://github.com/linux-pam/linux-pam/tree/master/doc

sshd can partly be debugged by passing -d or setting LogLevel DEBUG3 or such (per the man page), though that didn't help in the PAM case since sshd probably doesn't handle any of that. In the case of prepending UsePAM no to sshd_config, the debug output still didn't seem to help.

Here is an example of a pam file for su:

account required pam_unix.so






# Authentication management.

auth sufficient pam_rootok.so

auth required pam_faillock.so







auth sufficient pam_unix.so   likeauth try_first_pass




auth required pam_deny.so

# Password management.
password sufficient pam_unix.so nullok sha512







# Session management.
session required pam_env.so conffile=/nix/store/l0pbij6sknb3a14cwwlpqxp2hwhhz1js-pam-environment readenv=0

session required pam_unix.so

Asking the person that helped me notice the nullok issue about debugging pam, they said:

The last few times I've had to deal with pam configs, I went for a mix of staring at the configs real hard, adding debug flags to modules which support it, and sprinkling pam_debug invocations all over the place?

• https://www.ibm.com/docs/en/aix/7.2?topic=modules-enabling-pam-debug mentions something about /etc/pam_debug but is probably AIX only?

[deliciouslytyped]

Ok so I should have probably bit the small bullet earlier and looked into compiling pam with debugging enabled *, but since all the google results were centos or redhat I (incorrectly and unwarrantedly) assumed that it wasn't an upstream thing.

I'm still not sure if PAM is just horribly unverbose even when debugging (despite passing audit and debug to pam_unix.so in the confs), or something is wrong with my logging setup or how I'm using it. As part of my shotgun debugging, I tried enabling syslogd and that didn't seem to do anything useful. Various parts of the syslog module do state that systemd is supposed to be able to handle syslog already.

• confirm whether or not I was getting all the PAM messages

* So after digging a bit in the pam_unix.so code I discovered some aditional debug strings that seem to be gated by the D() macro, leading to this SO post: https://stackoverflow.com/questions/32741574/pam-standard-macros-and-logging-on-centos7/32741712#32741712

The macros can be found at https://github.com/linux-pam/linux-pam/blob/e65d93223bbcf9931f580963edeacbe3c2cf72cb/libpam/include/security/_pam_macros.h#L81 currently.

/*
 * This is for debugging purposes ONLY. DO NOT use on live systems !!!
 * You have been warned :-) - CG
 *
 * to get automated debugging to the log file, it must be created manually.
 * _PAM_LOGFILE must exist and be writable to the programs you debug.
 */

#ifndef _PAM_LOGFILE
#define _PAM_LOGFILE "/var/run/pam-debug.log"
#endif

#define D(x) do { \
    _pam_output_debug_info(__FILE__, __FUNCTION__, __LINE__); \
    _pam_output_debug x ; \
} while (0)

https://github.com/linux-pam/linux-pam/blob/e65d93223bbcf9931f580963edeacbe3c2cf72cb/configure.ac#L237

dnl lots of debugging information goes to /var/run/pam-debug.log
AC_ARG_ENABLE([debug],
    AS_HELP_STRING([--enable-debug],[specify you are building with debugging on]))

if test x"$enable_debug" = x"yes" ; then
   AC_DEFINE([PAM_DEBUG],,
		[lots of stuff gets written to /var/run/pam-debug.log])
fi

Meanwhile, various tidbits in https://serverfault.com/questions/249671/switch-on-pam-debugging-to-syslog that may or may not help at all.

[deliciouslytyped]

To quote myself from #16884, with regards to appling a function to override the PAM strings :

This solves a longstanding question I've had. It would be good to document this.

(Generally good to be thorough in that one documents creation, destruction, mutation..wait wasn't there an acronym for that? https://en.wikipedia.org/wiki/Create,_read,_update,_and_delete I guess - and do this for each orthogonal component.)

Here is an almost-minimal expression demonstrating this functionality:

#! /usr/bin/env nix-shell
#! nix-shell -i "nix-build -A etc" -p""
let
  modules = [ ({lib, ...}: {
    options.networking.firewall.enable = lib.mkOption {  
      apply = v: builtins.trace v v; # Should trace `false`
      };
    config.networking.firewall.enable = false;
    }) ];
in rec {
  result = import <nixpkgs/nixos/lib/eval-config.nix> { inherit modules; };
  inherit (result.config.system.build) etc;
  }

• But I don't know why this is in options.networking whereas with nbp's example it should be config.networking?

Combining this with a submodule (thanks @infinisil for the submodule example the other day), here is a demonstration adding some flags to https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/security/pam.nix#L389 (observe the content of /etc/pam.d/su):

#! /usr/bin/env nix-shell
#! nix-shell -i "nix-build -A etc" -p""
let
  module = {lib, ...}: with lib // lib.types; {
    options.security.pam.services = mkOption {
      type = attrsOf (submodule ({...}: {
        options.text = mkOption {
          apply = v: builtins.replaceStrings [ "pam_unix.so" ] [ "pam_unix.so audit debug" ] v; 
          };
        }));     
      };
    };
in rec {
  result = import <nixpkgs/nixos/lib/eval-config.nix> { modules = [ module ]; };
  etc = result.config.system.build.etc;
  }

Though I'm now left with some questions on how this is supposed to work... what happens if you do this multiple times? (priority system like always?)

[deliciouslytyped]

Well, the way I ended up solving this was noticing that when I was applying security.pam.services.ssh.allowNullPassword = true;, I did it to ssh and not sshd. The nullok settings we'ren't showing up in the file, and I was scratching my head as to why. To my defense, it's called services.openssh, not services.opensshd :P

[deliciouslytyped]

I'm not sure where my initial difficulties were, using services.openssh.extraConfig = mkOrder (-1) "UsePAM no\n" does result in this working for the non-PAM variant as well.

[nh2]

@deliciouslytyped Thanks for investigating this.

I lost half an hour trying to set up a NixOS VM test machine that allows SSH with empty password.

How to configure a NixOS machine to allow SSH for root with empty password

users.users.root.hashedPassword = ""; # "" means passwordless login
services.openssh.settings.PermitRootLogin = "yes";
services.openssh.settings.PermitEmptyPasswords = "yes";
security.pam.services.sshd.allowNullPassword = true;