Failed to connect to the Ceph cluster

[pareshpp]

hi,
I have a single node ceph cluster on ubuntu where i have installed cosbench 0.4.2.c2.

I am getting error as "Failed to connect to the Ceph cluster" as shown below.

Workload file ==================================================================

<storage type="librados" config="accesskey=admin;secretkey=AQAKzJNWJa7tIRAApeMbgrIf3d2XwxNRrLGK0g==;endpoint=172.20.0.40"/>

</workflow>

cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = AQAKzJNWJa7tIRAApeMbgrIf3d2XwxNRrLGK0g==

=================================================system.log
2016-01-13 20:04:33,416 [INFO] [WorkloadProcessor] - begin to process workload w29
2016-01-13 20:04:33,416 [INFO] [WorkloadProcessor] - begin to run stage s1-prepare
2016-01-13 20:04:33,416 [INFO] [WorkloadProcessor] - ============================================
2016-01-13 20:04:33,416 [INFO] [WorkloadProcessor] - START WORK: prepare
2016-01-13 20:04:33,029 [INFO] [AbstractCommandTasklet] - time drift between controller and driver-driver1 is 29 mSec
2016-01-13 20:04:33,029 [INFO] [StageRunner] - successfully booted all tasks in stage s1-prepare
2016-01-13 20:09:33,123 [ERROR] [AbstractCommandHandler] - unexpected error
com.intel.cosbench.api.storage.StorageException: com.ceph.rados.exceptions.RadosTimeoutException: Failed to connect to the Ceph cluster; ETIMEDOUT: Connection timed out (-110)
at com.intel.cosbench.api.librados.LibradosStorage.init(LibradosStorage.java:96)
at com.intel.cosbench.driver.service.COSBStorageAPIService.getStorage(COSBStorageAPIService.java:54)
at com.intel.cosbench.driver.service.MissionHandler.createStorageApi(MissionHandler.java:210)
at com.intel.cosbench.driver.service.MissionHandler.createWorkerContext(MissionHandler.java:197)
at com.intel.cosbench.driver.service.MissionHandler.createWorkers(MissionHandler.java:185)
at com.intel.cosbench.driver.service.MissionHandler.init(MissionHandler.java:100)
at com.intel.cosbench.driver.service.COSBDriverService.createHandler(COSBDriverService.java:112)
at com.intel.cosbench.driver.service.COSBDriverService.submit(COSBDriverService.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:71)
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:54)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy20.submit(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:71)
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:54)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy5.submit(Unknown Source)
at com.intel.cosbench.driver.handler.SubmitHandler.process(SubmitHandler.java:38)
at com.intel.cosbench.driver.handler.AbstractCommandHandler.handleRequest(AbstractCommandHandler.java:76)
at com.intel.cosbench.driver.handler.SubmitHandler.handleRequest(SubmitHandler.java:1)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.ceph.rados.exceptions.RadosTimeoutException: Failed to connect to the Ceph cluster; ETIMEDOUT: Connection timed out (-110)
at com.ceph.rados.RadosBase.throwException(RadosBase.java:87)
at com.ceph.rados.RadosBase.handleReturnCode(RadosBase.java:50)
at com.ceph.rados.Rados.connect(Rados.java:150)
at com.intel.cosbench.api.librados.LibradosStorage.init(LibradosStorage.java:76)
... 62 more
2016-01-13 20:09:33,126 [ERROR] [AbstractCommandTasklet] - driver report error: HTTP 500 - com.ceph.rados.exceptions.RadosTimeoutException: Failed to connect to the Ceph cluster; ETIMEDOUT: Connection timed out (-110)
2016-01-13 20:09:33,127 [ERROR] [StageRunner] - detected tasks [t1] have encountered errors
2016-01-13 20:09:33,127 [INFO] [StageRunner] - begin to terminate stage s1-prepare
2016-01-13 20:09:33,132 [ERROR] [AbstractCommandTasklet] - driver report error: HTTP 400 - unrecognized request: org.apache.catalina.connector.RequestFacade@277c2b4e
2016-01-13 20:09:33,132 [ERROR] [Aborter] - fail to abort driver
com.intel.cosbench.controller.tasklet.TaskletException
at com.intel.cosbench.controller.tasklet.AbstractCommandTasklet.issueCommand(AbstractCommandTasklet.java:81)
at com.intel.cosbench.controller.tasklet.Aborter.executeAbort(Aborter.java:53)
at com.intel.cosbench.controller.tasklet.Aborter.execute(Aborter.java:42)
at com.intel.cosbench.controller.tasklet.AbstractTasklet.call(AbstractTasklet.java:47)
at com.intel.cosbench.controller.tasklet.AbstractTasklet.call(AbstractTasklet.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-01-13 20:09:33,133 [INFO] [StageRunner] - stage s1-prepare has been terminated
2016-01-13 20:09:33,133 [ERROR] [WorkloadProcessor] - detected stage s1-prepare encountered error
2016-01-13 20:09:33,133 [INFO] [WorkloadProcessor] - begin to terminate workload w29
2016-01-13 20:09:33,196 [INFO] [SimpleWorkloadArchiver] - workload w29 has been successfully archived
2016-01-13 20:09:33,196 [INFO] [WorkloadProcessor] - successfully terminated workload w29

[pareshpp]

I could resolve this issue by using public address in workload file. Thanks Anyways.

[zonArt]

Hi,

I've got the exact same issue using rados-java library. It's like it's willing to go out on a proxy (for my case) and I guess that's why you mention the public address (even if I don't really understand where you set it). Strangely this is working on one server I'm using it and on the other one not. If you could explain me the hypothesis on what's going on, I'd appreciate a lot. Thank you