Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate the hbase connection to reduce hbase region lock contention. #8691

Closed
emeroad opened this issue Mar 8, 2022 · 0 comments
Closed

Comments

@emeroad
Copy link
Member

emeroad commented Mar 8, 2022

Lock timeout occurs due to userRegionLock contention, separating the hbase connection to split the lock.

ERROR o.a.h.h.c.AsyncProcess                   : Failed to get region location
org.apache.hadoop.hbase.exceptions.LockTimeoutException: Failed to get user region lock in10000 ms.  for accessing meta region server.
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.takeUserRegionLock(ConnectionManager.java:1428) ~[hbase-shaded-client-1.7.1.jar!/:1.7.1]
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1315) ~[hbase-shaded-client-1.7.1.jar!/:1.7.1]
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1219) ~[hbase-shaded-client-1.7.1.jar!/:1.7.1]
        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:523) ~[hbase-shaded-client-1.7.1.jar!/:1.7.1]
        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:463) ~[hbase-shaded-client-1.7.1.jar!/:1.7.1]
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:299) ~[hbase-shaded-client-1.7.1.jar!/:1.7.1]
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:266) ~[hbase-shaded-client-1.7.1.jar!/:1.7.1]
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1477) ~[hbase-shaded-client-1.7.1.jar!/:1.7.1]
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1024) ~[hbase-shaded-client-1.7.1.jar!/:1.7.1]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate2$6.doInTable(HbaseTemplate2.java:290) ~[pinpoint-commons-hbase-2.4.0-SNAPSHOT.jar!/:2.4.0-SNAPSHOT]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate2.execute(HbaseTemplate2.java:751) ~[pinpoint-commons-hbase-2.4.0-SNAPSHOT.jar!/:2.4.0-SNAPSHOT]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate2.put(HbaseTemplate2.java:285) ~[pinpoint-commons-hbase-2.4.0-SNAPSHOT.jar!/:2.4.0-SNAPSHOT]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate2.put(HbaseTemplate2.java:280) ~[pinpoint-commons-hbase-2.4.0-SNAPSHOT.jar!/:2.4.0-SNAPSHOT]
        at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao.insert(HbaseAgentEventDao.java:80) ~[pinpoint-collector-2.4.0-SNAPSHOT.jar!/:2.4.0-SNAPSHOT]
        at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao$$FastClassBySpringCGLIB$$28cd1468.invoke(<generated>) ~[pinpoint-collector-2.4.0-SNAPSHOT.jar!/:2.4.0-SNAPSHOT]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao$$EnhancerBySpringCGLIB$$6c8c444e.insert(<generated>) ~[pinpoint-collector-2.4.0-SNAPSHOT.jar!/:2.4.0-SNAPSHOT]
        at com.navercorp.pinpoint.collector.service.AgentEventService.insert(AgentEventService.java:42) ~[pinpoint-collector-2.4.0-SNAPSHOT.jar!/:2.4.0-SNAPSHOT]
        at com.navercorp.pinpoint.collector.service.async.AgentEventAsyncTaskService.handleEvent(AgentEventAsyncTaskService.java:51) ~[pinpoint-collector-2.4.0-SNAPSHOT.jar!/:2.4.0-SNAPSHOT]
        at com.navercorp.pinpoint.collector.service.async.AgentEventAsyncTaskService$$FastClassBySpringCGLIB$$b2911b07.invoke(<generated>) ~[pinpoint-collector-2.4.0-SNAPSHOT.jar!/:2.4.0-SNAPSHOT]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at com.navercorp.pinpoint.collector.namespace.NameSpaceInfoPropagateInterceptor.aroundAdvice(NameSpaceInfoPropagateInterceptor.java:86) ~[classes!/:2.4.0-SNAPSHOT]
        at jdk.internal.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115) ~[spring-aop-5.3.13.jar!/:5.3.13]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
emeroad added a commit to emeroad/pinpoint that referenced this issue Mar 8, 2022
emeroad added a commit to emeroad/pinpoint that referenced this issue Mar 10, 2022
emeroad added a commit to emeroad/pinpoint that referenced this issue Mar 10, 2022
emeroad added a commit to emeroad/pinpoint that referenced this issue Mar 22, 2022
emeroad added a commit that referenced this issue Mar 22, 2022
@emeroad emeroad added this to the 2.4.0 milestone Apr 6, 2022
@emeroad emeroad closed this as completed Apr 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant