java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.

[lvcarlei]

Please answer these questions before submitting your issue.

• Why do you submit this issue?
• Question or discussion
• Bug
• Requirement
• Feature or performance improvement

---

Question

• What do you want to know?

---

Bug

• Which version of SkyWalking, OS, and JRE?
  SkyWalking version 8.3.0
  Alpine Linux v3.8
  java version "1.8.0_192"

• What happened?
  ERROR 2021-02-25 14:41:03:221 AnalysisSaveAsync-235 InstMethodsInter : class[class org.apache.hadoop.hbase.client.HTable] before method[getScanner] intercept failure
  java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
  at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:163)
  at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:143)
  at org.apache.skywalking.apm.agent.core.context.ContextManager.createExitSpan(ContextManager.java:112)
  at org.apache.skywalking.apm.plugin.hbase.HTableInterceptor.beforeMethod(HTableInterceptor.java:72)
  at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:76)
  at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java)
  at executor.aicoin.utils.HbaseUtil.scanLast(HbaseUtil.java:226)
  at executor.aicoin.service.AnalysisService.parseDataT(AnalysisService.java:57)
  at executor.aicoin.service.AnalysisService.parseData(AnalysisService.java:682)
  at executor.aicoin.service.AnalysisService$$FastClassBySpringCGLIB$$c21a2f26.invoke()
  at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
  at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
  at org.springframework.aop.interceptor.AsyncExecutionInterceptor$1.call(AsyncExecutionInterceptor.java:115)
  at org.apache.skywalking.apm.plugin.spring.async.SWCallable.call(SWCallable.java:47)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

[wu-sheng]

This should be a plugin bug, we need your help to locate this issue, and send a pull request. The key is, why we don't get the real peer(hbase server IPs).

[lvcarlei]

Is this also a plugin bug?

class[class org.elasticsearch.client.RestHighLevelClient] before method[search] intercept failure
java.lang.NullPointerException
at org.apache.skywalking.apm.plugin.elasticsearch.v6.interceptor.RestHighLevelClientSearchMethodsInterceptor.beforeMethod(RestHighLevelClientSearchMethodsInterceptor.java:45)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:76)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java)
at executor.aicoin.utils.hioes.EsUtil.getKlineRecordBytime(EsUtil.java:67)
at executor.aicoin.service.KlineAnalysisService.getFormulaKlineResultByTime(KlineAnalysisService.java:291)
at executor.aicoin.service.KlineAnalysisService.calKlineT(KlineAnalysisService.java:262)
at executor.aicoin.service.KlineAnalysisService.calKline(KlineAnalysisService.java:389)
at executor.aicoin.service.KlineAnalysisService$$FastClassBySpringCGLIB$$376ed08d.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor$1.call(AsyncExecutionInterceptor.java:115)
at org.apache.skywalking.apm.plugin.spring.async.SWCallable.call(SWCallable.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

[wu-sheng]

I don't know actually. Look like. Maybe version mismatch, usage mismatch. Same as previous comments, we need you to check locally with our existing plugin codes and test cases.

[nisiyong]

I don't know actually. Look like. Maybe version mismatch, usage mismatch. Same as previous comments, we need you to check locally with our existing plugin codes and test cases.

I met this exception on our production agent log, too. Today I debug the plugin and figure it out.
The hbase-1.x-plugin only suits the HBase client 1.x, cause after HBase 2.x, HTable has changed its constructor function. So when use hbase-client-2.x will meet this exception.

Environment

• SkyWalking Agent: 8.2.0
• HBase Client: 2.1.0 (1.x works fine)

Agent Log

ERROR 2021-03-17 18:31:53:844 http-nio-8080-exec-4 InstMethodsInter : class[class org.apache.hadoop.hbase.client.HTable] before method[delete] intercept failure
java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
	at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:163)
	at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:143)
	at org.apache.skywalking.apm.agent.core.context.ContextManager.createExitSpan(ContextManager.java:126)
	at org.apache.skywalking.apm.plugin.hbase.HTableInterceptor.beforeMethod(HTableInterceptor.java:72)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:76)
	at org.apache.hadoop.hbase.client.HTable.delete(HTable.java)
	at org.apache.skywalking.apm.testcase.hbase.controller.HBaseController.hbaseCase(HBaseController.java:91)

1.x HTable contructor (from version 1.0.0 to 1.6.0)

source code: HTable.java

@InterfaceAudience.Private
public HTable(TableName tableName, final ClusterConnection connection,
    final ConnectionConfiguration tableConfig,
    final RpcRetryingCallerFactory rpcCallerFactory,
    final RpcControllerFactory rpcControllerFactory,
    final ExecutorService pool) throws IOException {
    ...
}

2.x HTable contructor (from version 2.0.0 to 2.1.9)

source code: HTable.java

@InterfaceAudience.Private
protected HTable(final ClusterConnection connection,
  final TableBuilderBase builder,
  final RpcRetryingCallerFactory rpcCallerFactory,
  final RpcControllerFactory rpcControllerFactory,
  final ExecutorService pool) {
  ...
}

2.x HTable contructor (after version 2.2.0 include 2.2.0)

source code: HTable.java

@InterfaceAudience.Private
protected HTable(final ConnectionImplementation connection,
  final TableBuilderBase builder,
  final RpcRetryingCallerFactory rpcCallerFactory,
  final RpcControllerFactory rpcControllerFactory,
  final ExecutorService pool) {
  ...
}

The above difference cause org.apache.skywalking.apm.plugin.hbase.HTableInterceptor#onConstruct could not set the zk address in EnhancedInstance, and when invoke ContextManager.createExitSpan will pass the empty peer.

public void onConstruct(EnhancedInstance objInst, Object[] allArguments) throws Throwable {
    Configuration connection = ((ClusterConnection) allArguments[1]).getConfiguration();
    Field field = connection.getClass().getDeclaredField("overlay");
    field.setAccessible(true);
    Properties properties = (Properties) field.get(connection);
    for (Map.Entry entry : properties.entrySet()) {
        if ("hbase.zookeeper.quorum".equals(entry.getKey())) {
            objInst.setSkyWalkingDynamicField(entry.getValue().toString());
        }
    }
}

I have fixed the plugin now, but I need some help. The current plugin named hbase-1.x.-plugin, I want to add a new plugin hbase-2.x-plugin, but I was confused which plugin will be work at runtime because both of them enhance the same class org.apache.hadoop.hbase.client.HTable.

I want to know more about how does the plugin work on multi-version.

[wu-sheng]

There are 2 ways

1. Don't focus on the version match mechanism. You could intercept the new construct directly. But as you are sharing the same target class, you need to change the old HTableInstrumentation and add a ConstructorInterceptPoint. Then, the plugin should be renamed to hbase-1.x-2.x-plugin.(documents should be updated too)
2. If you insist to use version match mechanism to create a new plugin called hbase-2.x-plugin, then, read Set up witnessClasses and/or witnessMethods if the instrumentation should be activated in specific versions. section in the plug dev doc, https://skywalking.apache.org/docs/main/v8.4.0/en/guides/java-plugin-development-guide/. You need to find special class or method to identify the version in the runtime.

[nisiyong]

1. the plugin should be renamed to hbase-1.x-2.x-plugin

I prefer the first one since theHTable public methods still have no change, I think HBase Client Core API won't change often.
And if we need to change the plugin name, why not just name it hbase-plugin directly?

[nisiyong]

Maybe hbase-1.x-2.x-plugin will be more clearly, like tomcat-7.x-8.x-plugin.
After renaming the plugin, just use 2.x dependency in the pom.xml? No 1.x dependency anymore, but the plugin still suits 1.x.

[wu-sheng]

Maybe hbase-1.x-2.x-plugin will be more clearly, like tomcat-7.x-8.x-plugin.
After renaming the plugin, just use 2.x dependency in the pom.xml? No 1.x dependency anymore, but the plugin still suits 1.x.

Compiling dependency doesn't matter. Remember to add 2.0 into plugin tests(e2e). We need them to verify.

[wu-sheng]

@nisiyong You actually need another issue to describe the issue. This issue actually is related to ElasticSearch client plugin somehow.

[nisiyong]

Okay, I will do it tomorrow.