gssapi-keyex failures blocking other authentication methods fallback

[athos-ribeiro]

Attempting a connection with kerberos authentication enabled in the server using the following authentication order,

ssh -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex $USER@$HOST -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes,

works as expected.

However, if there is no ticket for $USER in the client, the server throws an error:

monitor_read: unpermitted request 48

And the client gets the connection closed, instead of receiving an authorization error.

As a consequence, setting additional authentication fallbacks, such as PreferredAuthentications=gssapi-with-mic,gssapi-keyex,password results in the methods listed after gssapi-keyex not being tried at all.

Moreover, swapping the authentication order to gssapi-keyex,gssapi-with-mic results in the described issue not manifesting itself at all. i.e., using PreferredAuthentications=gssapi-keyex,gssapi-with-mic,password without a ticket for $USER in the client results in a password being requested.

The issue was originally reported against the Ubuntu package https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1938144. However, it is currently reproducible in Debian unstable and in Fedora rawhide.

Reproducers

The reproducer relies on OCI containers.

Before you run the containers, ensure you do have both the server and client scripts in a new directory (to be mounted in the containers). See the next section for getting the proper script for the right distro.

Set up the server container

docker run --rm -it -p 2222:2222 -p 8888:88 -v ${PWD}:/src -w /src $CONTAINER_IMAGE /bin/bash

The $ CONTAINER_IMAGE could be any of ubuntu:focal,ubuntu:impish,fedora:latest,fedora:rawhide,debian:unstable

Within the server container, run

./reproduce-server.sh

Note that this script will contain a few interactive bits. You can just press
enter in most of them, do make sure to create (and remember) passwords when prompted for those.

After the server setup is complete (the ssh server will be running), start the
client container setup:

docker run --rm -it -v ${PWD}:/src -w /src ubuntu:focal /bin/bash

Within the client container, run

./reproduce-client.sh

Note that this script will also contain a few interactive bits.

You should now see the issue reproduced as described above.

Fedora: reproduce-server.sh

#!/bin/bash

set -ex

if [ ! -f /tmp/setup-complete ]; then
  dnf install -y passwd hostname vim net-tools openssh-server krb5-libs krb5-server krb5-workstation
  CONTAINER_HOSTNAME=$(hostname)
  cat <<EOF > /etc/krb5.conf
[libdefaults]
    default_realm = EXAMPLE.COM
    forwardable = TRUE
    dns_canonicalize_hostname = true
[realms]
    EXAMPLE.COM = {
            kdc = ${CONTAINER_HOSTNAME}
            admin_server = ${CONTAINER_HOSTNAME}
    }
[domain_realm]
        ${CONTAINER_HOSTNAME} = EXAMPLE.COM
EOF
  kdb5_util create -s
  kadmin.local addprinc user
  kadmin.local addprinc -randkey host/${CONTAINER_HOSTNAME}
  kadmin.local ktadd -k /etc/krb5.keytab host/${CONTAINER_HOSTNAME}
  tail -n1 /etc/hosts > /src/hosts_tail
  adduser user -p test
  krb5kdc
  mkdir /run/sshd
  passwd root
  touch /tmp/setup-complete
fi
/usr/sbin/sshd -d -d -d -p 2222 -f /dev/null -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes

Debian/Ubuntu: reproduce-server.sh

#!/bin/bash

if [ ! -f /tmp/setup-complete ]; then
  apt update
  apt install -y vim net-tools inetutils-tools openssh-server krb5-kdc krb5-admin-server
  touch /etc/krb5kdc/kadm5.acl
  touch /etc/krb5kdc/kadm5.dict
  CONTAINER_HOSTNAME=$(hostname)
  cat <<EOF > /etc/krb5.conf
[libdefaults]
    default_realm = EXAMPLE.COM
    forwardable = TRUE
[realms]
    EXAMPLE.COM = {
            kdc = ${CONTAINER_HOSTNAME}
            admin_server = ${CONTAINER_HOSTNAME}
    }
[domain_realm]
        ${CONTAINER_HOSTNAME} = EXAMPLE.COM
EOF
  krb5_newrealm
  kadmin.local addprinc user
  kadmin.local addprinc -randkey host/${CONTAINER_HOSTNAME}
  kadmin.local ktadd -k /etc/krb5.keytab host/${CONTAINER_HOSTNAME}
  tail -n1 /etc/hosts > /src/hosts_tail
  adduser user
  mkdir /run/sshd
  touch /tmp/setup-complete
fi

/usr/sbin/sshd -d -p 2222 -f /dev/null -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes

reproduce-client.sh

#!/bin/bash

set -x
if [ ! -f /tmp/setup-complete ]; then
  SERVER_HOSTNAME=$(cut -d$'\t' -f2 hosts_tail)
  apt update
  apt install -y vim net-tools inetutils-tools openssh-client krb5-user
  cat /src/hosts_tail >> /etc/hosts
  cat <<EOF > /etc/krb5.conf
[libdefaults]
    default_realm = EXAMPLE.COM
    forwardable = TRUE
[realms]
    EXAMPLE.COM = {
            kdc = ${SERVER_HOSTNAME}
            admin_server = ${SERVER_HOSTNAME}
    }
[domain_realm]
        ${SERVER_HOSTNAME} = EXAMPLE.COM
EOF
  kinit user
  rm -f /src/hosts_tail
  touch /tmp/setup-complete
fi

ssh -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex root@${SERVER_HOSTNAME} -v -p 2222 -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes -F /dev/null

[Jakuje]

Heya. Thanks for a detailed reproducer and bug report. I am sorry for a delay, but I am somehow not receiving notifications for new issues in github.

At this moment, either @beldmit or @cjwatson could look into that as they are now working on the OpenSSH patches for this so I will probably defer to them.

[beldmit]

I think I've got the problem.

The request 48 MONITOR_REQ_GSSCHECKMIC is permitted in

openssh-gsskex/monitor.c

Lines 1886 to 1891 in 8571f49

	if (major == GSS_S_COMPLETE) {
	monitor_permit(mon_dispatch, MONITOR_REQ_GSSSTEP, 0);
	monitor_permit(mon_dispatch, MONITOR_REQ_GSSUSEROK, 1);
	monitor_permit(mon_dispatch, MONITOR_REQ_GSSCHECKMIC, 1);
	monitor_permit(mon_dispatch, MONITOR_REQ_GSSSIGN, 1);
	}

When we use the gssapi-with-mic authentication and it fails, the flags are cleaned up here

openssh-gsskex/monitor.c

Lines 542 to 547 in 8571f49

	/* The child may use this request only once, disable it */
	if (ent->flags & MON_ONCE) {
	debug2("%s: %d used once, disabling now", __func__,
	type);
	ent->flags &= ~MON_PERMIT;
	}

so we need restore the permission for the MONITOR_REQ_GSSCHECKMIC request for trying the other GSS Auth method.

I didn't find the proper place to restore it yet.