Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent LDAP problems on v0.49.0 - "Timeout waiting for idle object" #2986

Closed
bghinkle opened this issue Dec 10, 2015 · 9 comments
Closed

Intermittent LDAP problems on v0.49.0 - "Timeout waiting for idle object" #2986

bghinkle opened this issue Dec 10, 2015 · 9 comments
Assignees
@hibooboo2
Labels
area/access-control kind/bug Issues that are defects reported by users or that we know have reached a real release

Comments

@bghinkle
Copy link

Since enabling authentication on Rancher v0.49.0, we have twice noticed an issue where LDAP authentication fails with the following error...the domain controller which we are targeting is healthy during the time of errors and successfully handling LDAP queries from other Linux systems. Restarting the Rancher container fixes the problem.

2015-12-10 20:39:43,410 ERROR [:] [] [] [] [p800074881-2356] [i.c.p.i.a.a.i.l.a.ADIdentityProvider] Failed to get service context for ldap. java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442) ~[commons-pool2-2.1.jar:2.1]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360) ~[commons-pool2-2.1.jar:2.1]
at io.cattle.platform.iaas.api.auth.integration.ldap.LDAPIdentityProvider.getServiceContext(LDAPIdentityProvider.java:316) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.integration.ldap.ad.ADIdentityProvider.userRecord(ADIdentityProvider.java:85) [cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.integration.ldap.ad.ADIdentityProvider.getIdentities(ADIdentityProvider.java:199) [cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.integration.ldap.ad.ADTokenCreator.getLdapToken(ADTokenCreator.java:43) [cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.integration.ldap.ad.ADTokenCreator.getToken(ADTokenCreator.java:60) [cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.identity.TokenResourceManager.createToken(TokenResourceManager.java:56) [cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.identity.TokenResourceManager.createInternal(TokenResourceManager.java:45) [cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.resource.impl.AbstractBaseResourceManager.create(AbstractBaseResourceManager.java:115) [cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.ResourceManagerRequestHandler.generate(ResourceManagerRequestHandler.java:39) [cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.AbstractResponseGenerator.handle(AbstractResponseGenerator.java:14) [cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.handle(DefaultReadWriteApiDelegate.java:27) [cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.write(DefaultReadWriteApiDelegate.java:22) [cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at sun.reflect.GeneratedMethodAccessor394.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_85]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) [spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) [spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96) [spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260) [spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) [spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) [spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) [spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at com.sun.proxy.$Proxy43.write(Unknown Source) [na:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.ReadWriteApiHandler.handle(ReadWriteApiHandler.java:19) [cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.servlet.ApiRequestFilterDelegate.doFilter(ApiRequestFilterDelegate.java:86) [cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.servlet.ApiRequestFilter$1.runInContext(ApiRequestFilter.java:84) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.servlet.ApiRequestFilter.doFilter(ApiRequestFilter.java:77) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) [jetty-security-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]
@deniseschannon deniseschannon added kind/bug Issues that are defects reported by users or that we know have reached a real release area/authentication labels Dec 10, 2015
@hibooboo2
Copy link

@bhinkle-ezesoft Does auth fail until rancher is restarted or just fail one request and the next one works?

@bghinkle
Copy link
Author

Keeps failing until restart.

Sent from my iPhone

On Dec 10, 2015, at 5:34 PM, James Jeffrey <notifications@github.commailto:notifications@github.com> wrote:

@bhinkle-ezesofthttps://github.com/bhinkle-ezesoft Does auth fail until rancher is restarted or just fail one request and the next one works?

Reply to this email directly or view it on GitHubhttps://github.com//issues/2986#issuecomment-163786294.

@hibooboo2
Copy link

@bhinkle-ezesoft About how long does it take to fail?

@bghinkle
Copy link
Author

I don't have exact numbers but less than an hour the first time, and less than 8 hours the second time.

@bghinkle
Copy link
Author

Just happened again this morning.

@bghinkle
Copy link
Author

Just happened again:

2015-12-17 15:26:13,933 ERROR [:] [] [] [] [cutorService-18] [i.c.p.i.a.a.i.l.a.ADIdentityProvider] Ldap Failure during group retrieval: Unable to borrow a service context from context pool. io.cattle.platform.iaas.api.auth.integration.ldap.ServiceContextRetrievalException: Unable to borrow a service context from context pool.

@hibooboo2 hibooboo2 mentioned this issue Dec 22, 2015
Merged
@hibooboo2
Copy link

@bhinkle-ezesoft Does the latest release of rancher still cause this issue for you?

@bghinkle
Copy link
Author

We only upgraded about 48 hours ago, but so far so good.

@hibooboo2
Copy link

@bhinkle-ezesoft That is great. I am going to close this issue for now then. If it comes up again We can reopen it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hibooboo2 hibooboo2

Labels
area/access-control kind/bug Issues that are defects reported by users or that we know have reached a real release
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@vincent99 @hibooboo2 @will-chan @deniseschannon @bghinkle