Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pinpoint - Flink 설정 문제(LocalStreamEnvironment) #10628

Closed
leon4652 opened this issue Jan 19, 2024 · 4 comments
Closed

Pinpoint - Flink 설정 문제(LocalStreamEnvironment) #10628

leon4652 opened this issue Jan 19, 2024 · 4 comments

Comments

@leon4652
Copy link

leon4652 commented Jan 19, 2024 •
edited
Loading

Prerequisites

Please check the FAQ, and search existing issues for similar questions before creating a new issue.YOU MAY DELETE THIS PREREQUISITES SECTION.

  • [v] I have checked the FAQ, and issues and found no answer.

환경 Linux Cent OS 7(x64), pinpoint 2.3.x, flink 1.8.2, Hbase 1.4.6
하나의 서버 인스턴스 안에 Hbase, Zookeeper, Web, Collector, flink 전부 존재.

유사 문제를 먼저 참조하였습니다.
#4178
#4618

image

기존 Web 모듈에서 Inspector 기능을 사용하고자 Flink 설치와 다음 공식 문서를 참조하였습니다. (https://pinpoint-apm.gitbook.io/pinpoint/documents/application-inspector#id-3)

  1. flink 설치
  2. 'https://github.com/pinpoint-apm/pinpoint/tree/2.3.x' 파일을 clone
  3. clone한 파일의 /pinpoint-2.3.x/pinpoint-2.3.x/flink/src/main/resources/profiles/release 내부의 properties 중 pinpoint-flink.properties 하단에 'flink.sourceFunction.Parallel=1' 옵션만 추가.
  4. 변경 이후 mvn clean package를 통해 .jar 파일 추출
  5. flink ./start-cluster 설정.
  6. 이후 './flink run pinpoint-flink-job-2.3.3.jar', './flink run pinpoint-flink-job-2.3.3.jar --spring.profiles.active release' 등으로 실행해도 계속 같은 에러가 발생하네요.

에러 로그와 properties를 추가적으로 참조합니다. 해결 방법을 문의드립니다.

추가적으로 파일 실행시 발생하는 flink-root-client-localhost log입니다.

2024-01-19 02:32:03,264 INFO  org.apache.flink.client.cli.CliFrontend                       - --------------------------------------------------------------------------------
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -  Starting Command Line Client (Version: 1.6.2, Rev:3456ad0, Date:17.10.2018 @ 18:46:46 GMT)
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -  OS current user: root
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -  Current Hadoop/Kerberos user: <no hadoop dependency found>
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -  JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.202-b08
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -  Maximum heap size: 1739 MiBytes
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -  JAVA_HOME: /home/jdk1.8.0_202
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -  No Hadoop Dependency available
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -  JVM Options:
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -     -Dlog.file=/home/flink-1.6.2/log/flink-root-client-localhost.localdomain.log
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -     -Dlog4j.configuration=file:/home/flink-1.6.2/conf/log4j-cli.properties
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -     -Dlogback.configurationFile=file:/home/flink-1.6.2/conf/logback.xml
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -  Program Arguments:
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -     run
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -     /home/pinpoint-2.3.1.x/flink/target/pinpoint-flink-job-2.3.3.jar
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -     --spring.profiles.active=release
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       -  Classpath: /home/flink-1.6.2/lib/flink-python_2.11-1.6.2.jar:/home/flink-1.6.2/lib/log4j-1.2.17.jar:/home/flink-1.6.2/lib/slf4j-log4j12-1.7.7.jar:/home/flink-1.6.2/lib/flink-dist_2.11-1.6.2.jar:::
2024-01-19 02:32:03,265 INFO  org.apache.flink.client.cli.CliFrontend                       - --------------------------------------------------------------------------------
2024-01-19 02:32:03,268 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2024-01-19 02:32:03,268 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2024-01-19 02:32:03,268 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m
2024-01-19 02:32:03,268 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 1024m
2024-01-19 02:32:03,268 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2024-01-19 02:32:03,268 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2024-01-19 02:32:03,269 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8082
2024-01-19 02:32:03,271 WARN  org.apache.flink.client.cli.CliFrontend                       - Could not load CLI class org.apache.flink.yarn.cli.FlinkYarnSessionCli.
java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1236)
	at org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1192)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1118)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 5 more
2024-01-19 02:32:03,278 INFO  org.apache.flink.core.fs.FileSystem                           - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2024-01-19 02:32:03,363 INFO  org.apache.flink.runtime.security.modules.HadoopModuleFactory  - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2024-01-19 02:32:03,373 INFO  org.apache.flink.runtime.security.SecurityUtils               - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2024-01-19 02:32:03,400 INFO  org.apache.flink.client.cli.CliFrontend                       - Running 'run' command.
2024-01-19 02:32:03,402 INFO  org.apache.flink.client.cli.CliFrontend                       - Building program from JAR file
2024-01-19 02:32:03,468 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'
2024-01-19 02:32:03,566 INFO  org.apache.flink.runtime.rest.RestClient                      - Rest client endpoint started.
2024-01-19 02:32:03,566 INFO  org.apache.flink.client.cli.CliFrontend                       - Starting execution of program
2024-01-19 02:32:03,567 INFO  org.apache.flink.client.program.rest.RestClusterClient        - Starting program in interactive mode (detached: false)
2024-01-19 02:32:03,569 INFO  com.navercorp.pinpoint.flink.StatStreamingVer2Job             - start Aggregation Stat Data job with job parameter. : {spring.profiles.active=release=__NO_VALUE_KEY}
2024-01-19 02:32:04,162 INFO  com.navercorp.pinpoint.common.server.util.PinpointThreadPoolExecutorFactoryBean  - Initializing ExecutorService 'hbaseThreadPool'
2024-01-19 02:32:04,422 INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper        - Process identifier=hconnection-0x7ea4d397 connecting to ZooKeeper ensemble=localhost:2181
2024-01-19 02:32:04,426 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2024-01-19 02:32:04,426 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:host.name=localhost
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.version=1.8.0_202
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.vendor=Oracle Corporation
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.home=/home/jdk1.8.0_202/jre
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.class.path=/home/flink-1.6.2/lib/flink-python_2.11-1.6.2.jar:/home/flink-1.6.2/lib/log4j-1.2.17.jar:/home/flink-1.6.2/lib/slf4j-log4j12-1.7.7.jar:/home/flink-1.6.2/lib/flink-dist_2.11-1.6.2.jar:::
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.io.tmpdir=/tmp
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.compiler=<NA>
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:os.name=Linux
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:os.arch=amd64
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:os.version=3.10.0-1160.el7.x86_64
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:user.name=root
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:user.home=/root
2024-01-19 02:32:04,427 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:user.dir=/home/flink-1.6.2/bin
2024-01-19 02:32:04,428 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@654c1a54
2024-01-19 02:32:04,437 WARN  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-3975952183753788761.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2024-01-19 02:32:04,448 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181
2024-01-19 02:32:04,452 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn  - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2024-01-19 02:32:04,461 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn  - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x18d1c0dac5a0021, negotiated timeout = 40000
2024-01-19 02:32:04,649 INFO  com.navercorp.pinpoint.collector.mapper.thrift.stat.ThriftAgentStatMapper  - ThriftStatMapper:ThriftDeadlockThreadCountBoMapper
2024-01-19 02:32:04,649 INFO  com.navercorp.pinpoint.collector.mapper.thrift.stat.ThriftAgentStatMapper  - ThriftStatMapper:ThriftFileDescriptorBoMapper
2024-01-19 02:32:04,649 INFO  com.navercorp.pinpoint.collector.mapper.thrift.stat.ThriftAgentStatMapper  - ThriftStatMapper:ThriftCpuLoadBoMapper
2024-01-19 02:32:04,649 INFO  com.navercorp.pinpoint.collector.mapper.thrift.stat.ThriftAgentStatMapper  - ThriftStatMapper:ThriftTransactionBoMapper
2024-01-19 02:32:04,649 INFO  com.navercorp.pinpoint.collector.mapper.thrift.stat.ThriftAgentStatMapper  - ThriftStatMapper:ThriftActiveTraceBoMapper
2024-01-19 02:32:04,649 INFO  com.navercorp.pinpoint.collector.mapper.thrift.stat.ThriftAgentStatMapper  - ThriftStatMapper:ThriftResponseTimeBoMapper
2024-01-19 02:32:04,649 INFO  com.navercorp.pinpoint.collector.mapper.thrift.stat.ThriftAgentStatMapper  - ThriftStatMapper:ThriftJvmGcDetailedBoMapper
2024-01-19 02:32:04,649 INFO  com.navercorp.pinpoint.collector.mapper.thrift.stat.ThriftAgentStatMapper  - ThriftStatMapper:ThriftLoadedClassBoMapper
2024-01-19 02:32:04,649 INFO  com.navercorp.pinpoint.collector.mapper.thrift.stat.ThriftAgentStatMapper  - ThriftStatMapper:ThriftJvmGcBoMapper
2024-01-19 02:32:04,649 INFO  com.navercorp.pinpoint.collector.mapper.thrift.stat.ThriftAgentStatMapper  - ThriftStatMapper:ThriftDataSourceBoMapper
2024-01-19 02:32:04,649 INFO  com.navercorp.pinpoint.collector.mapper.thrift.stat.ThriftAgentStatMapper  - ThriftStatMapper:ThriftTotalThreadCountBoMapper
2024-01-19 02:32:04,649 INFO  com.navercorp.pinpoint.collector.mapper.thrift.stat.ThriftAgentStatMapper  - ThriftStatMapper:ThriftDirectBufferBoMapper
2024-01-19 02:32:04,740 INFO  com.navercorp.pinpoint.flink.config.FlinkConfiguration        - org.slf4j.impl.Log4jLoggerAdapter(com.navercorp.pinpoint.flink.config.FlinkConfiguration)
2024-01-19 02:32:04,742 INFO  com.navercorp.pinpoint.collector.thrift.config.AgentBaseDataReceiverConfiguration  - AgentBaseDataReceiverConfiguration{bindIp='0.0.0.0', bindPort=19994, workerThreadSize=8, workerQueueSize=1024, workerMonitorEnable=true}
2024-01-19 02:32:04,881 INFO  com.navercorp.pinpoint.collector.receiver.ExecutorFactoryBean  - Initializing ExecutorService 'flinkWorker'
2024-01-19 02:32:04,927 INFO  com.navercorp.pinpoint.flink.Bootstrap                        - Bootstrap initialization. : job parameter {spring.profiles.active=release=__NO_VALUE_KEY}
2024-01-19 02:32:04,934 INFO  org.apache.flink.runtime.rest.RestClient                      - Shutting down rest endpoint.
2024-01-19 02:32:04,938 INFO  org.apache.flink.runtime.rest.RestClient                      - Rest endpoint shutdown complete.
2024-01-19 02:32:04,938 ERROR org.apache.flink.client.cli.CliFrontend                       - Error while running the command.
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
	at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:426)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:816)
	at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:290)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:216)
	at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1053)
	at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1129)
	at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1129)
Caused by: org.apache.flink.api.common.InvalidProgramException: The LocalStreamEnvironment cannot be used when submitting a program through a client, or running in a TestEnvironment context.
	at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.<init>(LocalStreamEnvironment.java:66)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.createLocalEnvironment(StreamExecutionEnvironment.java:1657)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.createLocalEnvironment(StreamExecutionEnvironment.java:1638)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.createLocalEnvironment(StreamExecutionEnvironment.java:1624)
	at com.navercorp.pinpoint.flink.Bootstrap.createStreamExecutionEnvironment(Bootstrap.java:171)
	at com.navercorp.pinpoint.flink.StatStreamingVer2Job.start(StatStreamingVer2Job.java:57)
	at com.navercorp.pinpoint.flink.StatStreamingVer2Job.main(StatStreamingVer2Job.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
	... 9 more

log

 The program finished with the following exception:

The LocalStreamEnvironment cannot be used when submitting a program through a client, or running in a TestEnvironment context.
        org.apache.flink.streaming.api.environment.LocalStreamEnvironment.<init>(LocalStreamEnvironment.java:66)
        org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.createLocalEnvironment(StreamExecutionEnvironment.java:1654)
        org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.createLocalEnvironment(StreamExecutionEnvironment.java:1636)
        org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.createLocalEnvironment(StreamExecutionEnvironment.java:1622)
        com.navercorp.pinpoint.flink.Bootstrap.createStreamExecutionEnvironment(Bootstrap.java:171)
        com.navercorp.pinpoint.flink.StatStreamingVer2Job.start(StatStreamingVer2Job.java:57)
        com.navercorp.pinpoint.flink.StatStreamingVer2Job.main(StatStreamingVer2Job.java:48)
[root@localhost bin]# ./flink run pinpoint-flink-job-2.3.3.jar --spring.profiles.active=server
Starting execution of program
2024-01-19 00:01:06,616 INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper        - Process identifier=hconnection-0x6974a715 connecting to ZooKeeper ensemble=localhost:2181
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:host.name=localhost
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.version=1.8.0_202
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.vendor=Oracle Corporation
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.home=/home/jdk1.8.0_202/jre
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.class.path=/home/flink/flink-1.8.2/lib/log4j-1.2.17.jar:/home/flink/flink-1.8.2/lib/slf4j-log4j12-1.7.15.jar:/home/flink/flink-1.8.2/lib/flink-dist_2.11-1.8.2.jar:::
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.io.tmpdir=/tmp
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:java.compiler=<NA>
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:os.name=Linux
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:os.arch=amd64
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:os.version=3.10.0-1160.el7.x86_64
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:user.name=root
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:user.home=/root
2024-01-19 00:01:06,620 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Client environment:user.dir=/home/flink/flink-1.8.2/bin
2024-01-19 00:01:06,621 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper  - Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@75699e35
2024-01-19 00:01:06,626 WARN  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-8294328163375005107.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2024-01-19 00:01:06,627 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server localhost/127.0.0.1:2181
2024-01-19 00:01:06,631 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn  - Socket connection established to localhost/127.0.0.1:2181, initiating session
2024-01-19 00:01:06,640 INFO  org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn  - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x18d1c0dac5a0014, negotiated timeout = 40000

------------------------------------------------------------
 The program finished with the following exception:

The LocalStreamEnvironment cannot be used when submitting a program through a client, or running in a TestEnvironment context.
        org.apache.flink.streaming.api.environment.LocalStreamEnvironment.<init>(LocalStreamEnvironment.java:66)
        org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.createLocalEnvironment(StreamExecutionEnvironment.java:1654)
        org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.createLocalEnvironment(StreamExecutionEnvironment.java:1636)
        org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.createLocalEnvironment(StreamExecutionEnvironment.java:1622)
        com.navercorp.pinpoint.flink.Bootstrap.createStreamExecutionEnvironment(Bootstrap.java:171)
        com.navercorp.pinpoint.flink.StatStreamingVer2Job.start(StatStreamingVer2Job.java:57)
        com.navercorp.pinpoint.flink.StatStreamingVer2Job.main(StatStreamingVer2Job.java:48)

properties

# configure l4 ip address to ignore health check logs
collector.l4.ip=

# base data receiver config  ---------------------------------------------------------------------
collector.receiver.base.ip=0.0.0.0
collector.receiver.base.port=19994

# number of tcp worker threads
collector.receiver.base.worker.threadSize=8
# capacity of tcp worker queue
collector.receiver.base.worker.queueSize=1024
# monitoring for tcp worker
collector.receiver.base.worker.monitor=true

# change OS level read/write socket buffer size (for linux)
#sudo sysctl -w net.core.rmem_max=
#sudo sysctl -w net.core.wmem_max=
# check current values using:
#$ /sbin/sysctl -a | grep -e rmem -e wmem

# number of agent event worker threads
collector.agentEventWorker.threadSize=4
# capacity of agent event worker queue
collector.agentEventWorker.queueSize=1024

#collector.admin.password=
#collector.admin.api.rest.active=
#collector.admin.api.jmx.active=

collector.spanEvent.sequence.limit=10000

# flink cluster
flink.cluster.enable=true
flink.cluster.zookeeper.address=localhost
flink.cluster.zookeeper.znode_root=/pinpoint-cluster
flink.cluster.zookeeper.sessiontimeout=3000
flink.cluster.zookeeper.retry.interval=5000
flink.cluster.tcp.port=19994

# flink env init
flink.StreamExecutionEnvironment=server
flink.sourceFunction.Parallel=1
@smilu97
Copy link
Contributor

smilu97 commented Jan 22, 2024

It seems flink.StreamExecutionEnvironment=server property is not applied well.

Could you try mvn clean install and test it again?

@leon4652
Copy link
Author

It seems flink.StreamExecutionEnvironment=server property is not applied well.

Could you try mvn clean install and test it again?

안녕하세요. 좋은 저녁입니다.
말씀하신 대로 'flink.StreamExecutionEnvironment=server' 문제 및 (배포로 나온 .jar의 local folder 내부 properties 수정)과 flink version 문제였습니다. (flink 1.16 -> 1.6.3) 답변에 감사드립니다.

하지만 여전히 inspector 기능을 작성할 수 없는데, 혹시 이 부분에 대해 추가로 답변해주실 수 있는지 질의드립니다..
image
hbase 로그를 살펴보니 AgentStatV2에 데이터가 저장되는 것으로 보아 세부적인 데이터들이 쌓이는 것 같긴 한데, 여전히 Web UI에서는 Heap 메모리 등 세부 정보를 확인 할 수 없는 이슈가 있습니다. (Graph가 공백으로 나타납니다.)

이 이슈에 관해 조언을 들을 수 있을까요?

@smilu97
Copy link
Contributor

smilu97 commented Jan 22, 2024

Do you see any error logs from flink job manager or collector? I think we should check which component is the root of cause.

As we have experienced, most common reasons of malfunctions in inspector were two: the collector does not find worker in flink cluster, or flink cannot write in hbase.

@leon4652
Copy link
Author

leon4652 commented Jan 23, 2024 •
edited
Loading

Do you see any error logs from flink job manager or collector? I think we should check which component is the root of cause.

As we have experienced, most common reasons of malfunctions in inspector were two: the collector does not find worker in flink cluster, or flink cannot write in hbase.

안녕하세요. 좋은 오후입니다.
말씀해주신 것처럼 Collector 자체의 log나, Web의 Log를 확인하고 원인을 분석해보았습니다.
결론적으로는 boot 과정의 문제였는데, shell script를 통한 boot 과정(hbase -> flink cluster, flink job -> web / collector 부트 )과정에서 충분한 boot 시간이 주어지지 않아 벌어진 문제 같네요.
여유 시간을 두고 부트하니 성공적으로 로그가 출력되는 것을 확인할 수 있었습니다. 친절한 답변에 감사드립니다. :)

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@smilu97 @leon4652