LDAPCP and securitytoken.svc timeout

[aerotodorr]

Hi Yvan

We are experiencing this issue where users inteminnently are getting HTTP error 500 with message "The server is busy now. Try again later".
This happens after user is authenticatet and redirected to https://site/_trust/

In ULS I can see that calls to securitytoken.svc time out after 60 sec. It happens on both frontends.
The servers are not under pressure, doesn't look like a hardware capacity issue. Restarting SecurityTokenServiceAppPool seem to help but not for a long period.

This farm is using LDAPCP, and I'm wondering if our LDAPCP's configuration somehow contribute to the issue, specially the augmentation part of it.

• LDAPCP the only thing that is unique comparing to other farms

• Users can be members of up to 500 AD groups

• The token issued by ADFS contains claim "Role" with users group membership, but this is ignored and LDAPCP augmentation of "Role" is used, getting the same memberships from identity providers directly using LDAP

• Can see LDAP errors occuring in ULS, LDAP timeouts or LDAP Unknown error (0x8000500c). Usually when it happens users get "This page is not shared with you..." message

Is there anything with this configuration and symptoms that can cause securitytoken.svc timeouts when converting external to internal token og should I search for a solution somewhere else?

:: Sharepoint configuration

• Sharepoint 2016 RTM on Windows Server 2016
• Using claims authentication
• Federation provider: ADFS
• Identity providers: 2 active directories AD.DOMAIN1.ORG and AD.DOMAIN2.ORG
• Claim provider: LDAPCP
• Identity claim: UPN

:: LDAPCP configuration

• LDAP connections:
  LDAP://OU=UsersOU1,OU=Accounts,DC=AD,DC=DOMAIN1,DC=ORG
  LDAP://OU=UsersOU2,OU=Accounts,DC=AD,DC=DOMAIN1,DC=ORG
  LDAP://OU=UsersOU3,OU=Accounts,DC=AD,DC=DOMAIN1,DC=ORG
  LDAP://OU=UsersOU4,OU=Accounts,DC=AD,DC=DOMAIN1,DC=ORG
  LDAP://OU=UsersOU5,OU=Accounts,DC=AD,DC=DOMAIN1,DC=ORG
  LDAP://OU=Groups,DC=AD,DC=DOMAIN1,DC=ORG (used for groups augmentation in DOMAIN1)
  LDAP://AD.DOMAIN2.ORG (used for groups augmentation in DOMAIN2)

• Augmentation: Enabled

• Augmented Group Claim Type: http://schemas.microsoft.com/ws/2008/06/identity/claims/role

• LDAP connections used for augmentation ("This is an AD server" selected):
  LDAP://OU=Groups,DC=AD,DC=DOMAIN1,DC=ORG
  LDAP://AD.DOMAIN2.ORG

• LDAP query timeout: increased to 30, does not seem to help

• Additional LDAP filter for user attributes: Defined and present

:: Full exception from ULS

Claims Saml Sign-In: Could not get local token for trusted third party token. Exception: 'System.TimeoutException: The request channel timed out while waiting for a reply after 00:01:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc' has exceeded the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) - -- End of inner exception stack trace --- at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason) at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) - -- End of inner exception stack trace --- Server stack trace: at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustContract.Issue(Message message) at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr) at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst) at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo, SPRequestSecurityTokenProperties properties) at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForOnBehalfOfContext(Uri context, SecurityToken onBehalfOf) at Microsoft.SharePoint.IdentityModel.SPFederationAuthenticationModule.ExchangeArgumentTrustedThirdPartySessionSecurityTokenForLocalToken(SecurityToken thirdPartyToken, SessionSecurityTokenCreatedEventArgs arguments)'. Stack: ' Server stack trace: at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustContract.Issue(Message message) at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr) at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst) at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo, SPRequestSecurityTokenProperties properties) at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForOnBehalfOfContext(Uri context, SecurityToken onBehalfOf) at Microsoft.SharePoint.IdentityModel.SPFederationAuthenticationModule.ExchangeArgumentTrustedThirdPartySessionSecurityTokenForLocalToken(SecurityToken thirdPartyToken, SessionSecurityTokenCreatedEventArgs arguments)'.

[Yvand]

Hi @aerotodorr
LDAPCP plays a role when user signs in, if augmentation if enabled.
Can you try to disable it and check if timeout issue disappears?

[aerotodorr]

Hi @Yvand

This farm is in production and it takes time to get this trough.
I'll let you know how it goes.

[aerotodorr]

Hi @Yvand

We tried to disable LDAPCP augmentation today, but then users who were given access to site via group memberships (Role claim) got access denied error "This site is not shared with you... "

Tried to figure out why, and it seems that under LDAPCPs config page "Claim mapping", for Role claim, there is defined "Prefix to display" with value "{fqdn}"
and all groups that were given access to this site by that point have it prepended.

On the other side Role claim that comes with ADFS tokens consist of group names only (samaccountname), without prefixes. This mismatch caused access denied error.

When the prefix is removed on the "Claims mapping" config page, and access to same group is given again, now shown without the prefix, uses are able to acces pages with augmentation disabled.

Now the issue is how to easy convert these prefixes so the site is usable without augmentation.
Is it possible ho hack it somehow on LDAPCP level or by modifying those claims on ADFS level?
Do you have any ides and is this expected behavior?

[Yvand]

Hi @aerotodorr, the only way is to migrate them, e.g.:

$oldlogin="c:0-.t|contoso.local|contoso.local\group1";
$newlogin="c:0-.t|contoso.local|group1";
[Microsoft.SharePoint.Administration.SPFarm]::Local.MigrateUserAccount($oldlogin, $newlogin, $false);

[aerotodorr]

Hi @Yvand Thank you, Is there any reason why SPFarm.MigrateUserAccount() should be used comparing to using Move-SPUser ?

…

On Fri, 20 Apr 2018, 15:04 Yvan Duhamel, ***@***.***> wrote: Hi @aerotodorr <https://github.com/aerotodorr>, the only way is to migrate them, e.g.: $oldlogin="c:0-.t|contoso.local|contoso.local\group1";$newlogin="c:0-.t|contoso.local|group1";[Microsoft.SharePoint.Administration.SPFarm]::Local.MigrateUserAccount($oldlogin, $newlogin, $false); — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#50 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AP6Pi8-LbMNFXkK4WSCSSqw4OrSJ6T43ks5tqdzwgaJpZM4S6Sxa> .

[Yvand]

Hi @aerotodorr
Sorry for the very late reply. I just find SPFarm.MigrateUserAccount() much more convenient to use than Move-SPUser since it just takes 2 string parameters...

[aerotodorr]

Thanks @Yvand
Just to let you know, disabling augmentation solved our issue.

[KNareshChand]

HI Yvand,

I am still facing the same issue even though the augmentation setting was not enabled. Restarting the SharePoint IIS is resolving the issue only. If I am not wrong, I went to the setting as : Central Admin->Security->LDAPCP Configuration->Global Configuration->Augmentation->Enable augmentation (checkbox unchecked). Please help.

[Yvand]

@KNareshChand What error exactly are you getting?

[KNareshChand]

Thanks Yvand for the quick help.

This is the below error I am getting from ULS log

Claims Authentication 8307 Critical An exception occurred in LDAPCP claim provider when calling SPClaimProvider.FillClaimsForEntity(): Retrieving the COM class factory for component with CLSID {BDEADF26-C265-11D0-BCED-00A0C90AB50F} failed due to the following error: 800703fa Illegal operation attempted on a registry key that has been marked for deletion. (Exception from HRESULT: 0x800703FA).. 56ff379e-e938-f067-f602-ad570a5df7a0
12/19/2017 02:59:07.70 w3wp.exe (0x12B0) 0x1448 SharePoint Foundation Claims Authentication af3zp Unexpected STS Call Claims Saml: Problem getting output claims identity. Exception: 'System.Runtime.InteropServices.COMException (0x800703FA): Retrieving the COM class factory for component with CLSID {BDEADF26-C265-11D0-BCED-00A0C90AB50F} failed due to the following error: 800703fa Illegal operation attempted on a registry key that has been marked for deletion. (Exception from HRESULT: 0x800703FA). at Microsoft.SharePoint.Library.SPRequest..ctor() at Microsoft.SharePoint.SPGlobal.CreateSPRequestAndSetIdentity(SPSite site, String name, Boolean bNotGlobalAdminCode, String strUrl, Boolean bNotAddToContext, Byte[] UserToken, SPAppPrincipalToken appPrincipalToken, String userName, Boolean bIgnoreTokenTimeout, Boolean bAsAnonymous) at Microsoft.SharePoint.SPRequestManager.GetContextRequest(SPR... 56ff379e-e938-f067-f602-ad570a5df7a0
12/19/2017 02:59:07.70* w3wp.exe (0x12B0) 0x1448 SharePoint Foundation Claims Authentication af3zp Unexpected ...equestAuthenticationMode authenticationMode) at Microsoft.SharePoint.Administration.SPFarm.get_RequestAny() at Microsoft.SharePoint.SPSecurity.GetCurrentUserTokenNoApplicationPrincipalDelegated(SPWebApplication webApp, Uri siteUrl) at Microsoft.SharePoint.SPSecurity.GetCurrentUserToken() at Microsoft.SharePoint.SPSecurity.EnsureOriginatingUserToken() at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode) at Microsoft.SharePoint.Administration.Claims.SPClaimProvider.GetClaimsForEntity(Uri context, SPClaim entity, SPClaimProviderContext claimProviderContext) at Microsoft.SharePoint.Administration.Claims.SPClaimProviderOperatio... 56ff379e-e938-f067-f602-ad570a5df7a0
12/19/2017 02:59:07.70* w3wp.exe (0x12B0) 0x1448 SharePoint Foundation Claims Authentication af3zp Unexpected ...ns.ClaimsForEntity(Uri context, SPClaimProviderOperationOptions mode, String[] providerNames, SPClaim entity, SPClaimProviderContext claimProviderContext) at Microsoft.SharePoint.IdentityModel.SPSecurityTokenService.AugmentClaimsIdentityWithClaimProviders(SPRequestInfo requestInfo, IClaimsIdentity identity, SPClaim identityClaim) at Microsoft.SharePoint.IdentityModel.SPSecurityTokenService.AugmentOutputIdentityForRequest(SPRequestInfo requestInfo, IClaimsIdentity outputIdentity) at Microsoft.SharePoint.IdentityModel.SPSecurityTokenService.GetOutputClaimsIdentity(IClaimsPrincipal principal, RequestSecurityToken request, Scope scope)'. 56ff379e-e938-f067-f602-ad570a5df7a0
12/19/2017 02:59:07.70 w3wp.exe

[KNareshChand]

I don't know the exact reason why this error is happening but there is pattern that the SharePoint site is used to work as usual but after some day(may be many users logged in to the site), the site stopped working and we see as usual yellow error screen and pointing the url to /_trust/default.aspx. After restarting the SharePoint server, the issue is gone and it used to work again.

[Yvand]

@KNareshChand I think you likely face issue described in kb https://support.microsoft.com/en-in/help/3114011/800703fa-illegal-operation-attempted-on-a-registry-key-that-has-been-m
Can you try the solutions and confirm if it fixes the issue?

[KNareshChand]

Thanks Yvand, I will try this solution and will update.

[NaveenAhuja-PD]

Hello All, I am facing the same issue in SharePoint 2019 farm. Could you please advise. I have configured/tried the below but none of them worked:

1. App pool is started and also tried to Recycle the security token service pool
2. Clearing SharePoint config cache
3. IISRESET
4. http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc URL can be browsed, shows a WCF page.
5. Set LDAPCP as Claim provider
6. Augmentation set to false
7. Restarted the servers
8. Reprovisioned SecureTokenServiceApplication
9. Removed LDAPCP completely following https://ldapcp.com/Fix-setup-issues.html and installed again
10. Central admin and timer services are started

Below is the error in ULS logs:
Claims Saml Sign-In: Could not get local token for trusted third party token. Exception: 'System.TimeoutException: The request channel timed out while waiting for a reply after 00:01:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc' has exceeded the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) --- End of inner exception stack trace --- at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason) at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)

[Yvand]

@NaveenAhuja-PD can you filter the SharePoint logs on the SharePoint STS process?
They should explain what is wrong

[NaveenAhuja-PD]

@Yvand , Sorry, I missed your response.
I tried to filter out but did not get any process that seemed relevant to STS. Can you let me know if there is any special keyword with which i should filter?

I tried to search with the STS URL and got some logs only like:
_The audience uri matches a trusted login provider provider realm. AudienceUri:urn:sharepoint:name
The absolute uri validation succeeded. AudienceUri:urn:sharepoint:name
STS Call Claims Saml: Successfully requested sign-in claim identity on behalf of user '05.t.....
AVF claim not present. Claim type: 'http://schemas.microsoft.com/sharepoint/online/2014/06/claims/authenticationvalidfromutc'.

I do notice one log as critical:
A certificate validation operation took 60036.021 milliseconds and has exceeded the execution time threshold. If this continues to occur, it may represent a configuration issue. Please see http://go.microsoft.com/fwlink/?LinkId=246987 for more details.

[Yvand]

The certificate validation timeout is very likely the cause of the issue in the SharePoint STS process.
To identify the SharePoint STS process, run this command as administrator:
cmd /r %systemroot%\system32\inetsrv\appcmd list wp

[NaveenAhuja-PD]

@Yvand , thanks a lot for your responses. Appreciate it. The timeout issue got fixed by allowing crl.microsoft.com site in firewall. The servers do not have access to internet.