SSSD unable to retrieve secondary groups after upgrade of ipa-server

[krishna-pp]

Hello Community,
We recently updated ipa-server and a bunch of related packages from 4.6.8-5.el7.centos.11 to 4.6.8-5.el7.centos.12. This also updated the IPA data. After that, the clients are unable to retrieve group information. However, they can load SSH public keys and other user details fine. When I query the FreeIPA server using ipa and ldapsearch against a user, I see all group memberships. So, the data on the FreeIPA server seems fine, but only how SSSD talks to FreeIPA has changed.

On the clients, there were no changes, and I tried all combinations of ldap_schema (rfc2307, rfc2307bis, ipa) and ldap_group_member (memberUid, uniqueMember) every time, removing the cache and restarting SSSD. However, I don't see any change when I run id <username> or getent group <group>. They just return the user id and primary group; group and gid. I also tried to add initgroups sss files in /etc/nsswitch.conf, but that didn't make a difference.

I tried to revert the packages on the server, but it failed to say data schema is incompatible. So, the current status is, our users can SSH to the instances but can't sudo as group information is missing. Please let me know if you need any more details to help troubleshoot.

Thanks!

On client:

id
uid=1987401269(user_name) gid=1987401269(user_name) groups=1987401269(user_name) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

getent group sudo
sudo:*:27:

On FreeIPA server:

id
uid=1987401269(user_name) gid=1987401269(user_name) groups=1987401269(user_name),27(sudo),1987400000(group1),1987400473(group2),1987401284(group3), context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

ipa user-show --all --raw user_name 

 dn: uid=user_name,cn=users,cn=accounts,dc=REDACTED,dc=com

REDACTED 

  ipaSshPubKey: REDACTED
..
  memberof: cn=group1,cn=groups,cn=accounts,dc=REDACTED,dc=com
  memberof: cn=greoup2,cn=groups,cn=accounts,dc=REDACTED,dc=com
  memberof: cn=sudo,cn=groups,cn=accounts,dc=REDACTED,dc=com
  memberof: cn=group3,cn=groups,cn=accounts,dc=REDACTED,dc=com
..

 ldapsearch -Y GSSAPI -b 'uid=<user_name>,cn=users,cn=accounts,dc=REDACTED,dc=com'

Shows output similar to above.

I enabled debug logs(debug_level=6) on the SSSD client for all nss, pam and be calls to see if there are any issues, but I didn't find anything obvious. I thought it is not very useful to share it here, but I'm sharing the relevant commands SSSD initiates to the FreeIPA server.

(2022-11-18 10:00:29): [be[REDACTED.com]] [sdap_initgr_rfc2307_next_base] (0x0400): Searching for groups with base [dc=REDACTED,dc=com]
(2022-11-18 10:00:29): [be[REDACTED.com]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(memberUid=<user_name>)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))][dc=REDACTED,dc=com].
(2022-11-18 10:00:29): [be[REDACTED.com]] [sdap_get_generic_op_finished] (0x0400): Search result: Success(0), no errmsg set

and 

(2022-11-18 10:00:29): [be[REDACTED.com]] [sdap_get_groups_next_base] (0x0400): Searching for groups with base [dc=REDACTED,dc=com]
(2022-11-18 10:00:29): [be[REDACTED.com]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(gidNumber=<gid_name>)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))][dc=REDACTED,dc=com].
(2022-11-18 10:00:29): [be[REDACTED.com]] [sdap_get_generic_op_finished] (0x0400): Search result: Success(0), no errmsg set
(2022-11-18 10:00:29): [be[REDACTED.com]] [sdap_get_groups_process] (0x0400): Search for groups, returned 1 results.

More details below

FreeIPA server OS details

cat /etc/*release*
CentOS Linux release 7.9.2009 (Core)
Derived from Red Hat Enterprise Linux 7.9 (Source)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.9.2009 (Core)
CentOS Linux release 7.9.2009 (Core)
cpe:/o:centos:centos:7

Relevant upgrade logs on the FreeIPA server

---> Package ipa-client.x86_64 0:4.6.8-5.el7.centos.11 will be updated
---> Package ipa-client.x86_64 0:4.6.8-5.el7.centos.12 will be an update
---> Package ipa-client-common.noarch 0:4.6.8-5.el7.centos.11 will be updated
---> Package ipa-client-common.noarch 0:4.6.8-5.el7.centos.12 will be an update
---> Package ipa-common.noarch 0:4.6.8-5.el7.centos.11 will be updated
---> Package ipa-common.noarch 0:4.6.8-5.el7.centos.12 will be an update
---> Package ipa-python-compat.noarch 0:4.6.8-5.el7.centos.11 will be updated
---> Package ipa-python-compat.noarch 0:4.6.8-5.el7.centos.12 will be an update
---> Package ipa-server.x86_64 0:4.6.8-5.el7.centos.11 will be updated
---> Package ipa-server.x86_64 0:4.6.8-5.el7.centos.12 will be an update
---> Package ipa-server-common.noarch 0:4.6.8-5.el7.centos.11 will be updated
---> Package ipa-server-common.noarch 0:4.6.8-5.el7.centos.12 will be an update
---> Package python2-ipaclient.noarch 0:4.6.8-5.el7.centos.11 will be updated
---> Package python2-ipaclient.noarch 0:4.6.8-5.el7.centos.12 will be an update
---> Package python2-ipalib.noarch 0:4.6.8-5.el7.centos.11 will be updated
---> Package python2-ipalib.noarch 0:4.6.8-5.el7.centos.12 will be an update
---> Package python2-ipaserver.noarch 0:4.6.8-5.el7.centos.11 will be updated
---> Package python2-ipaserver.noarch 0:4.6.8-5.el7.centos.12 will be an update

Client OS and sssd versions

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
Amazon Linux release 2 (Karoo)
cpe:2.3:o:amazon:amazon_linux:2


yum list installed|grep sssd
python-sssdconfig.noarch            1.16.5-10.amzn2.10               @amzn2-core
sssd.x86_64                         1.16.5-10.amzn2.10               @amzn2-core
sssd-ad.x86_64                      1.16.5-10.amzn2.10               @amzn2-core
sssd-client.x86_64                  1.16.5-10.amzn2.10               @amzn2-core
sssd-common.x86_64                  1.16.5-10.amzn2.10               @amzn2-core
sssd-common-pac.x86_64              1.16.5-10.amzn2.10               @amzn2-core
sssd-ipa.x86_64                     1.16.5-10.amzn2.10               @amzn2-core
sssd-krb5.x86_64                    1.16.5-10.amzn2.10               @amzn2-core
sssd-krb5-common.x86_64             1.16.5-10.amzn2.10               @amzn2-core
sssd-ldap.x86_64                    1.16.5-10.amzn2.10               @amzn2-core
sssd-proxy.x86_64                   1.16.5-10.amzn2.10               @amzn2-core

sssd.conf on Client

[domain/REDACTED]
ldap_search_base = cn=users,cn=accounts,dc=REDACTED,dc=com
cache_credentials = true
id_provider = ldap
auth_provider = ldap
chpass_provider = ldap
ldap_uri = ldaps://freeipa.REDACTED.com,ldaps://ipa-slave.REDACTED.com
ldap_tls_cacert = /etc/openldap/cacerts/ca.crt
ldap_schema = rfc2307
ldap_user_ssh_public_key = ipaSshPubKey
ldap_group_search_base = dc=REDACTED,dc=com
ldap_page_size = 1900
group_name_attribute = cn
ldap_group_member = memberUid
group_class = posixGroup

[sssd]
config_file_version = 2
reconnection_retries = 3
sbus_timeout = 30
services = nss, pam, ssh, sudo
domains = REDACTED.com

[nss]
filter_groups = root
filter_users = root
reconnection_retries = 3
homedir_substring = /home

[pam]
reconnection_retries = 3
offline_credentials_expiration = 2
offline_failed_login_attempts = 3
offline_failed_login_delay = 5

[ssh]

[sumit-bose]

Hi,

with the given sssd.conf SSSD will connect anonymously to the LDAP server. Are the memberOf attributes shown if you call:

ldapsearch -x -b 'uid=<user_name>,cn=users,cn=accounts,dc=REDACTED,dc=com'

bye,
Sumit

[krishna-pp]

Hi,

with the given sssd.conf SSSD will connect anonymously to the LDAP server. Are the memberOf attributes shown if you call:

ldapsearch -x -b 'uid=<user_name>,cn=users,cn=accounts,dc=REDACTED,dc=com'

Hi Sumit, thanks for the quick reply. Appreciate it! It doesn't show the memberOf attribute. Interestingly, we have a production environment which hasn't been upgraded yet and working fine. However, it doesn't show the memberOf attribute as well.

ldapsearch -x -b 'uid=<user_name>,cn=users,cn=accounts,dc=REDACTED,dc=com'
# extended LDIF
#
# LDAPv3
# base <uid=user_name,cn=users,cn=accounts,dc=REDACTED,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# user_name, users, accounts, REDACTED.com
dn: uid=user_name,cn=users,cn=accounts,dc=REDACTED,dc=com
ipaSshPubKey: ..
displayName: <user name>
uid: <user_name>
objectClass: top
objectClass: person
objectClass: organizationalperson
objectClass: inetorgperson
objectClass: inetuser
objectClass: posixaccount
objectClass: krbprincipalaux
objectClass: krbticketpolicyaux
objectClass: ipaobject
objectClass: ipasshuser
objectClass: ipaSshGroupOfPubKeys
objectClass: mepOriginEntry
..skipping..

uidNumber: 1987401269
gidNumber: 1987401269

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

[sumit-bose]

Hi,

by default the memberOf and member attributes is not accessible for anonymous users. Is SSSD in the production environment using an authenticated bind by chance?

bye,
Sumit

[krishna-pp]

Hi,

by default the memberOf and member attributes is not accessible for anonymous users. Is SSSD in the production environment using an authenticated bind by chance?

No, it has a similar configuration. However, when I run the below command in production (I grabbed the command from sssd logs for the backend), it shows all the groups and all its members for each group (which is a huge output because it also shows memberUid for each group).

ldapsearch -x -b "dc=REDACTED,dc=sh" "(&(memberUid=user_name)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))"

# waf_admin, groups, compat, REDACTED.sh
dn: cn=waf_admin,cn=groups,cn=compat,dc=REDACTED,dc=sh
gidNumber: 1419400384
ipaAnchorUUID:: OklQQTppemwuc2g6OGJkMjk5OTItMTNkZC0xMWU3LWIwNzctMGFlMDlhOGYwMW
 Y1
memberUid: user_name1
memberUid: user_name2
memberUid: user_name3
memberUid: user_name4

The same ldapsearch command in the test doesn't return anything. Does it have any clue?

[sumit-bose]

# waf_admin, groups, compat, REDACTED.sh
dn: cn=waf_admin,cn=groups,cn=compat,dc=REDACTED,dc=sh

The results are coming from the compat-tree, is the compat-tree enabled in the test environment?

[krishna-pp]

The results are coming from the compat-tree, is the compat-tree enabled in the test environment?

Oh..seems like you found the issue! I've not heard before but a quick google search says it is some compatibility plugin designed for applications expecting the user/groups to follow RFC 2307. Is that correct?

However, I didn't find a way to enable it. I went to the IPA server in the test/prod and did a grep on compat in /etc/sssd and /etc/ipa. No results.

I'm not a FreeIPA expert. Is there a guide/howto?

[krishna-pp]

I think I found a utility to check/enable compact tree on the FreeIPA server, which is enabled in production. However, I don't have the Directory manager password for the test to check/enable the status. Looking for ways to reset the directory password. I'll let you know the status, and thanks a lot for your help so far!

 ipa-compat-manage status 
Directory Manager password: 

Plugin Enabled

[krishna-pp]

Hi @sumit-bose

I reset the Directory server password in the test and checked that the compat status is enabled already..Is there anything else that needs to be checked?

ipa-compat-manage status
Directory Manager password:

Plugin Enabled

[sumit-bose]

Hi,

in your sssd.conf there is ldap_search_base = cn=users,cn=accounts,dc=REDACTED,dc=com. Do you have the same in the "production" environment? I would suggest to try to remove this option because it would not cover the compat-tree.

bye,
Sumit

[krishna-pp]

Hi @sumit-bose ,

Yes, I have the same config in prod; removing it didn't help in the test. However, when I ran ldapsearch against compat tree, it worked fine in the test. So, I changed the ldap_group_search_base in sssd.conf:

From:
ldap_group_search_base = dc=REDACTED,dc=com

to:
ldap_group_search_base = cn=groups,cn=compat,dc=REDACTED,dc=com

And it started working!

Thanks a lot for all the help. Appreciate it.

[krishna-pp]

Closing the issue as it is resolved now. Here is to the power of community 🥇