LDAP ID provider corrupts db-cache and causes sssd to fail on restart #3918
Labels
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
Cloned from Pagure issue: https://pagure.io/SSSD/sssd/issue/2877
Problem surfaces when sudo_provider = ldap is enabled in sssd.conf. When commented out it works.
First run works, but when sssd is stopped and then restarted the process fails. And in the error logs there is a "[sssm_ldap_sudo_init] (0x0020): Cannot init LDAP ID provider [5]: Input/output error".
When I delete /var/lib/sss/db/cache_openforce.org.ldb I am able to start the service again, but once again when the service is restarted it breaks.
Comments
Comment from nandersson at 2015-11-19 17:58:46
Trace (-d 9) of the first run that works.
1-first-run-this-works
Comment from nandersson at 2015-11-19 17:59:18
2:nd run after the process is restarted. Trace with -d 9
2-second-run-after-sssd-restart-this-breaks
Comment from nandersson at 2015-11-19 17:59:51
3:rd run, after the cache is deleted. Now it works again.
3-deleted_cache_openforce.org.ldb-from-cache-now-it-works-again
Comment from nandersson at 2015-11-19 18:00:17
Finally 4:th trace where is again breaks.
4-second-restart-now-it-breaks-again
Comment from nandersson at 2015-11-19 18:02:53
The probably corrupt database file that includes the breaking issue.
cache_openforce.org.ldb
Comment from nandersson at 2015-11-19 18:15:50
This is the sssd.conf
_comment0: This is the sssd.conf
[sssd]
domains = openforce.org
config_file_version = 2
services = nss, pam, ssh, sudo
#reconnection_retries = 7
[ssh]
[sudo]
debug_level = 4
[pam]
offline_credentials_expiration = 60
pam_pwd_expiration_warning = 14
[nss]
#filter_groups = root
#filter_users = root
[domain/openforce.org]
id_provider = ad
#auth_provider = ad
#chpass_provider = ad
#access_provider = permit
sudo_provider = ldap
ignore_group_members = true
#enumerature = false
dyndns_update = false
use_fully_qualified_names = False
lookup_family_order = ipv4_only
cache_credentials = True
fallback_homedir = /home/%u
create_homedir = True
override_shell = /bin/bash
Sudo
ldap_uri = ldap://srv11.openforce.org
ldap_sudo_search_base = ou=SUDOers,dc=openforce,dc=org
ldap_default_bind_dn = cn=admin,dc=openforce,dc=org
ldap_default_authtok = Abc123!
#debug_level = 5
=> 1448015919523259
Comment from nandersson at 2015-11-19 18:16:51
sssd.conf
sssd.conf
Comment from nandersson at 2015-11-19 18:37:54
I had a check on the corrupt cache-file with tdbdump, and it is clear that the sudo-rules are fetched from the ldap-database and then stored in cache.
So, it looks like the problem might arise when the rules are then fetched from the cache on the 2:nd run, or with the sync between the cache and the ldap-storage.
Comment from jhrozek at 2015-11-20 11:38:19
It's not a database corruption, it's something with ID mapping. I was unable to reproduce the issue locally against AD 2012R2. You're using Samba, though, right?
Would using:
fix the issue for you? You'd drop the last section completely, then.
Alternatively, would it help if you added the domain SID with
ldap_idmap_default_domain_sidexplicitly?Comment from nandersson at 2015-11-20 14:03:30
I am using Samba4 instead of AD because it is easier to setup. In the corporate environment we have AD and I can't extend that schema to include Sudo-rules, so we must use a separate LDAP-database unfortunately.
I'll give ldap_idmap_default_domain_sid a look! Thanks!
Comment from jhrozek at 2015-11-20 14:08:59
Aaah, it's a separate LDAP tree. That might be the difference in my testing. Then I don't think it makes sense to add the default_domain_sid and I will run another test.
Comment from nandersson at 2015-11-20 15:14:22
That is correct.
When I look at the cache-file it seems to me that the Sudo-rules aren't correct. They don't point to the original ldap-source. Seems that they now point to some internal database? :-/
Edit: (Well, after posting I realized that that might be the purpose for the cache. I.e that they now reference to the internal cache-file instead of being fetched? I see there is a key DN=@Index:ORIGINALDN:CN=DEFAULTS,OU=SUDOERS,DC=OPENFORCE,DC=ORG that indeed is correct)
DN=@Index:DATAEXPIRETIMESTAMP:1447948439
DN=@Index:NAME:openforce.org
DN=NAME=candersson,CN=USERS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@modules
DN=@Index:NAME:defaults
DN=@Index:OBJECTCLASS:USER
DN=@Index:ORIGINALDN:CN=CANDERSSON,CN=USERS,DC=OPENFORCE,DC=ORG
DN=NAME=nandersson,CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:@idxone:
DN=@Index:SUDOUSER:nandersson
DN=@Index:NAMEALIAS:candersson
DN=@Index:ORIGINALDN:CN=NANDERSSON,OU=SUDOERS,DC=OPENFORCE,DC=ORG
DN=@ATTRIBUTES
DN=CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:CN:NANDERSSON
DN=@Index:CN:USERS
DN=@Index:NAME:candersson
DN=CN=RANGES,CN=SYSDB
DN=CN=USERS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:@idxone:CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:CN:SUDORULES
DN=@Index:CN:SYSDB
DN=@Index:GIDNUMBER:700400513
DN=@Index:UIDNUMBER:700401104
DN=@Index:CN:GROUPS
DN=@baseinfo
DN=@Index:@idxone:CN=USERS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:ORIGINALDN:CN=DEFAULTS,OU=SUDOERS,DC=OPENFORCE,DC=ORG
DN=@Index:OBJECTCLASS:ID_MAPPING
DN=@Index:SUDOUSER:candersson
DN=CN=GROUPS,CN=OPENFORCE.ORG,CN=SYSDB
DN=NAME=candersson,CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:ORIGINALDN:CN=CANDERSSON,OU=SUDOERS,DC=OPENFORCE,DC=ORG
DN=@Index:NAME:nandersson
DN=CN=SYSDB
DN=@Index:DATAEXPIRETIMESTAMP:1447948420
DN=@Index:@idxone:CN=ID_MAPPINGS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:LASTUPDATE:1447943039
DN=@Index:OBJECTCLASS:SUDORULE
DN=@Index:CN:RANGES
DN=@Index:@idxone:CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=NAME=defaults,CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:CN:OPENFORCE.ORG
DN=@Index:@idxone:CN=SYSDB
DN=@Index:CN:DEFAULTS
DN=@Index:CN:CANDERSSON
DN=@INDEXLIST
DN=CN=OPENFORCE.ORG,CN=SYSDB
DN=OBJECTSID=S-1-5-21-3065846811-936052125-3479462643,CN=ID_MAPPINGS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:@idxone:CN=OPENFORCE.ORG,CN=SYSDB
_comment0: That is correct.
When I look at the cache-file it seems to me that the Sudo-rules isn't correct. They don't point to the original ldap-source. Seems that they now point to some internal database? :-/
DN=@Index:DATAEXPIRETIMESTAMP:1447948439
DN=@Index:NAME:openforce.org
DN=NAME=candersson,CN=USERS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@modules
DN=@Index:NAME:defaults
DN=@Index:OBJECTCLASS:USER
DN=@Index:ORIGINALDN:CN=CANDERSSON,CN=USERS,DC=OPENFORCE,DC=ORG
DN=NAME=nandersson,CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:@idxone:
DN=@Index:SUDOUSER:nandersson
DN=@Index:NAMEALIAS:candersson
DN=@Index:ORIGINALDN:CN=NANDERSSON,OU=SUDOERS,DC=OPENFORCE,DC=ORG
DN=@ATTRIBUTES
DN=CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:CN:NANDERSSON
DN=@Index:CN:USERS
DN=@Index:NAME:candersson
DN=CN=RANGES,CN=SYSDB
DN=CN=USERS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:@idxone:CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:CN:SUDORULES
DN=@Index:CN:SYSDB
DN=@Index:GIDNUMBER:700400513
DN=@Index:UIDNUMBER:700401104
DN=@Index:CN:GROUPS
DN=@baseinfo
DN=@Index:@idxone:CN=USERS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:ORIGINALDN:CN=DEFAULTS,OU=SUDOERS,DC=OPENFORCE,DC=ORG
DN=@Index:OBJECTCLASS:ID_MAPPING
DN=@Index:SUDOUSER:candersson
DN=CN=GROUPS,CN=OPENFORCE.ORG,CN=SYSDB
DN=NAME=candersson,CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:ORIGINALDN:CN=CANDERSSON,OU=SUDOERS,DC=OPENFORCE,DC=ORG
DN=@Index:NAME:nandersson
DN=CN=SYSDB
DN=@Index:DATAEXPIRETIMESTAMP:1447948420
DN=@Index:@idxone:CN=ID_MAPPINGS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:LASTUPDATE:1447943039
DN=@Index:OBJECTCLASS:SUDORULE
DN=@Index:CN:RANGES
DN=@Index:@idxone:CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=NAME=defaults,CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:CN:OPENFORCE.ORG
DN=@Index:@idxone:CN=SYSDB
DN=@Index:CN:DEFAULTS
DN=@Index:CN:CANDERSSON
DN=@INDEXLIST
DN=CN=OPENFORCE.ORG,CN=SYSDB
DN=OBJECTSID=S-1-5-21-3065846811-936052125-3479462643,CN=ID_MAPPINGS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:@idxone:CN=OPENFORCE.ORG,CN=SYSDB
=> 1448028967295553
_comment1: That is correct.
When I look at the cache-file it seems to me that the Sudo-rules aren't correct. They don't point to the original ldap-source. Seems that they now point to some internal database? :-/
Edit: (Well, after posting I realized that that might be the purpose for the cache. I.e that they now reference to the internal cache-file instead of being fetched?)
DN=@Index:DATAEXPIRETIMESTAMP:1447948439
DN=@Index:NAME:openforce.org
DN=NAME=candersson,CN=USERS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@modules
DN=@Index:NAME:defaults
DN=@Index:OBJECTCLASS:USER
DN=@Index:ORIGINALDN:CN=CANDERSSON,CN=USERS,DC=OPENFORCE,DC=ORG
DN=NAME=nandersson,CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:@idxone:
DN=@Index:SUDOUSER:nandersson
DN=@Index:NAMEALIAS:candersson
DN=@Index:ORIGINALDN:CN=NANDERSSON,OU=SUDOERS,DC=OPENFORCE,DC=ORG
DN=@ATTRIBUTES
DN=CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:CN:NANDERSSON
DN=@Index:CN:USERS
DN=@Index:NAME:candersson
DN=CN=RANGES,CN=SYSDB
DN=CN=USERS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:@idxone:CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:CN:SUDORULES
DN=@Index:CN:SYSDB
DN=@Index:GIDNUMBER:700400513
DN=@Index:UIDNUMBER:700401104
DN=@Index:CN:GROUPS
DN=@baseinfo
DN=@Index:@idxone:CN=USERS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:ORIGINALDN:CN=DEFAULTS,OU=SUDOERS,DC=OPENFORCE,DC=ORG
DN=@Index:OBJECTCLASS:ID_MAPPING
DN=@Index:SUDOUSER:candersson
DN=CN=GROUPS,CN=OPENFORCE.ORG,CN=SYSDB
DN=NAME=candersson,CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:ORIGINALDN:CN=CANDERSSON,OU=SUDOERS,DC=OPENFORCE,DC=ORG
DN=@Index:NAME:nandersson
DN=CN=SYSDB
DN=@Index:DATAEXPIRETIMESTAMP:1447948420
DN=@Index:@idxone:CN=ID_MAPPINGS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:LASTUPDATE:1447943039
DN=@Index:OBJECTCLASS:SUDORULE
DN=@Index:CN:RANGES
DN=@Index:@idxone:CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=NAME=defaults,CN=SUDORULES,CN=CUSTOM,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:CN:OPENFORCE.ORG
DN=@Index:@idxone:CN=SYSDB
DN=@Index:CN:DEFAULTS
DN=@Index:CN:CANDERSSON
DN=@INDEXLIST
DN=CN=OPENFORCE.ORG,CN=SYSDB
DN=OBJECTSID=S-1-5-21-3065846811-936052125-3479462643,CN=ID_MAPPINGS,CN=OPENFORCE.ORG,CN=SYSDB
DN=@Index:@idxone:CN=OPENFORCE.ORG,CN=SYSDB
=> 1448029929813418
Comment from nandersson at 2015-11-20 15:33:57
Hi,
I just thought of, that the problem might be that I am having the same basedn dc=openforce,dc=org in both Samba4 and in the 2:nd OpenLDAP?
Perhaps I should try and rename the LDAP-one and see if that might be the issue here?
Comment from jhrozek at 2015-11-20 16:04:19
Maybe, worth a shot.
About the cache dump, it's more convenient to use ldbsearch from the ldb-tools package than tdbdump. Your dump shows a lot of internal data (indexes) and in general there is no guarantee that the ldb database would be powered by tdb, especially in the future.
Comment from nandersson at 2015-11-20 16:29:27
I changed the domain for the sudo server to a different TLD.
Same issue unfortunately :(
Comment from nandersson at 2015-11-20 16:37:52
These are error messages from sssd_sudo
(Fri Nov 20 16:27:33 2015) [sssd[sudo]] [confdb_get_domain_internal] (0x0100): Setting domain password expiration warning to 14 days
(Fri Nov 20 16:27:33 2015) [sssd[sudo]] [monitor_common_send_id] (0x0100): Sending ID: (sudo,1)
(Fri Nov 20 16:27:33 2015) [sssd[sudo]] [sss_names_init_from_args] (0x0100): Using re [(((?P[^\\]+)\(?P.+$))|((?P[^@]+)@(?P.+$))|(^(?P[^@\\]+)$))].
(Fri Nov 20 16:27:33 2015) [sssd[sudo]] [sss_fqnames_init] (0x0100): Using fq format [%1$s@%2$s].
(Fri Nov 20 16:27:33 2015) [sssd[sudo]] [dp_common_send_id] (0x0100): Sending ID to DP: (1,SUDO)
(Fri Nov 20 16:27:33 2015) [sssd[sudo]] [dp_id_callback] (0x0100): Got id ack and version (1) from DP
(Fri Nov 20 16:27:33 2015) [sssd[sudo]] [id_callback] (0x0100): Got id ack and version (1) from Monitor
(Fri Nov 20 16:27:44 2015) [sssd[sudo]] [confdb_get_domain_internal] (0x0100): Setting domain password expiration warning to 14 days
(Fri Nov 20 16:27:44 2015) [sssd[sudo]] [monitor_common_send_id] (0x0100): Sending ID: (sudo,1)
(Fri Nov 20 16:27:44 2015) [sssd[sudo]] [sss_names_init_from_args] (0x0100): Using re [(((?P[^\\]+)\(?P.+$))|((?P[^@]+)@(?P.+$))|(^(?P[^@\\]+)$))].
(Fri Nov 20 16:27:44 2015) [sssd[sudo]] [sss_fqnames_init] (0x0100): Using fq format [%1$s@%2$s].
(Fri Nov 20 16:27:44 2015) [sssd[sudo]] [sbus_client_init] (0x0020): check_file failed for [/var/lib/sss/pipes/private/sbus-dp_openforce.org].
(Fri Nov 20 16:27:44 2015) [sssd[sudo]] [sss_dp_init] (0x0010): Failed to connect to monitor services.
(Fri Nov 20 16:27:44 2015) [sssd[sudo]] [sss_process_init] (0x0010): fatal error setting up backend connector
(Fri Nov 20 16:27:44 2015) [sssd[sudo]] [sudo_process_init] (0x0010): sss_process_init() failed
Comment from nandersson at 2015-11-20 16:43:21
Seems to be a problem with /var/lib/sss/pipes/private/sbus-dp_openforce.org ?
It is a symbolic link that points to /var/lib/sss/pipes/private/sbus-dp_openforce.org.13800
ls -l /var/lib/sss/pipes/private/sbus-dp_openforce.org
lrwxrwxrwx 1 root root 54 nov 20 16:34 /var/lib/sss/pipes/private/sbus-dp_openforce.org -> /var/lib/sss/pipes/private/sbus-dp_openforce.org.13800
root@ows-d1b929e3:/var/log/sssd# ls -l /var/lib/sss/pipes/private/sbus-dp_openforce.org.13800
srw------- 1 root root 0 nov 20 16:34 /var/lib/sss/pipes/private/sbus-dp_openforce.org.13800
Comment from nandersson at 2015-11-21 00:42:17
When I do the failed run, both sssd_nss.log, sssd_pam.log, sssd_ssh.log and sssd_sudo.log reports:
sssd_openforce.org.log reports:
...and this message is repeated 4 times.
Comment from nandersson at 2015-11-21 01:23:47
Hi, I think I have tracked down the error to sdap_idmap_init.
This error occurs when the cache file is used. If it is not present, it is initialized correctly. It is also here where the /var/lib/sss/pipes/private/sbus-dp_openforce.org.14310 is teared down, and becuase if this the error in the other modules occurs (I think)
This is the log when it does not work:
Here is the same section when it does work: (with linenumbers)
Comment from nandersson at 2015-11-22 22:36:42
Here is the area in the log when it breaks by adding sudo_provider = ldap when the cache-file is present. I am positive :)
I don't know how to debug this further. If this could be debugged even further with gdb or something, just give me som guide lines and I'll try and track it down.
Comment from jhrozek at 2015-11-23 11:29:53
We resolved this ticket by explicitly adding:
to the
domainsection.resolution: => worksforme
status: new => closed
Comment from nandersson at 2017-02-24 14:40:27
Metadata Update from @nandersson:
The text was updated successfully, but these errors were encountered: