Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pinpoint-web startup error ,help! #3058

Closed
jack161641 opened this issue Jun 20, 2017 · 42 comments
Closed

pinpoint-web startup error ,help! #3058

jack161641 opened this issue Jun 20, 2017 · 42 comments
Assignees
Labels

Comments

@jack161641
Copy link

Description

catalina.out

2017-06-20 19:16:37 ERROR from localhost-startStop-1-SendThread(localhost:2181)
java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.hadoop.hbase.shaded.org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at java.lang.Throwable.initCause(Throwable.java:457)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1316)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1196)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1157)
at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1128)
Caused by: java.lang.ClassNotFoundException
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315)

Environment

pinpoint-web 1.6.2
jdk "1.8.0_101"
hbase-1.2.6

@vivekbny
Copy link

I am also facing the same issue in starting up the pinpoint web. Getting the same erros as above.

Kindly help

@vivekbny
Copy link

@denzelsN Please help in resolving the webui issue. Thanks

@jack161641
Copy link
Author

jack161641 commented Jun 23, 2017

@xbblq7n I had resolve this case, copy jar file hbase-shaded-server-1.1.1.jar to tomcat lib directory, download url http://snacktrace.com/artifacts/org.apache.hbase/hbase-shaded-server/1.1.1/org.apache.hadoop.hbase.shaded.org.apache.zookeeper.server.ZooTrace,
but I meet another issue now,pinpoint web and pinpoint collect had both same errors in starting up,follow:

org.apache.catalina.startup.ContextConfig.processServletContainerInitializers Failed to detect ServletContainerInitiali
zers for context with name [/pinpoint-collector]
java.io.IOException: java.lang.ClassNotFoundException: com.sun.jersey.server.impl.container.servlet.JerseyServletContainerInitializer
at org.apache.catalina.startup.WebappServiceLoader.loadServices(WebappServiceLoader.java:192)
at org.apache.catalina.startup.WebappServiceLoader.load(WebappServiceLoader.java:159)
at org.apache.catalina.startup.ContextConfig.processServletContainerInitializers(ContextConfig.java:1624)
at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1135)
at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:775)
at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5095)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1107)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1841)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1019)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:409)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1551)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:284)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:424)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:367)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:976)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:872)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1439)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1429)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)

@vivekbny
Copy link

@jack161641 The error is not occuring in the logs now. But still the web-ui is not coming up for me. Can you please help on this

@vivekbny
Copy link

Hi,

Can you help on this please?

@Xylus
Copy link
Contributor

Xylus commented Jun 26, 2017

@xbblq7n What kind of error are you getting? Could you explain the issue with more detail?

@jack161641
Copy link
Author

@xbblq7n I recompile and build war packages from the GitHub,resolved this error

@vivekbny
Copy link

@Xylus i am not able to get the pinpoint ui when deploying the ui war file,

Can you please help on that

@minwoo-jung
Copy link
Member

minwoo-jung commented Jun 27, 2017

Show me the all log of pinpoint web. Could explain the issue more detail?
ex) how does it look in web browser? Response is 404 status or you can see main page.

How did you get pinpoint web war package? did compile yourself? or did you download war file?
If you compile pinpoint yourself, recompile using 'mvn clean package'

@jack161641
Copy link
Author

@minwoo-jung can I use collect pull date from agent?

@vivekbny
Copy link

vivekbny commented Jun 27, 2017

weblogs.txt

@minwoo-jung Attached the logs. I am getting the 404 status.

I downloaded the war file and ran it in tomcat.

@vivekbny
Copy link

vivekbny commented Jun 27, 2017

@minwoo-jung @Xylus Steps i have done to setup the pinpoint. Please let me know if i am wrong some where.

HBASE

HBase is the Hadoop database, a distributed, scalable, big data store. Download the HBase binary in to the linux machine

Before installing the HBase, Please make sure the JAVA is installed on that machine. Now, extract and install the HBase by using the below commands
gunzip -d hbase-1.3.1-bin.tar.gz
tar -xvf hbase-1.3.1-bin.tar
2. Once installed, Set the JAVA_PATH
export JAVA_HOME=/usr/java/jdk1.8.0_74
PATH=$PATH:$HOME/bin
PATH=$PATH:$JAVA_HOME/bin
PATH=$PATH:(HBase directory)/bin
3. Now start the HBase by invoking the startup script placed in the bin directory
./start-hbase.sh
HBase is now started and it is up and running. Create the tables inside the Hbase

Whenever HBase is started up, it will store all the data under the /tmp directory. To change the location of the data written add the below in the conf/hbase-site.xml

Here i have chosen to write or create the table in the root/xbblq7n directory

hbase.rootdir
file:///root/xbblq7n/hbase


hbase.zookeeper.property.dataDir
/root/xbblq7n/zookeeper

4. To create the tables, we need to download the pinpoint.zip and then unzip it. Once it is unzipped, Execute the below command to create the tables.
/root/hbase-1.3.1/bin/hbase shell /root/pinpoint-master/hbase/scripts/hbase-create.hbase
(HBase directory)/bin/hbase shell (PinPoint directory)/hbase/scripts/hbase-create.hbase

Now the tables are created and can be viewed in the following URL:
http://localhost:16010/master-status

Pinpoint Collector

Step 1: Download the tomcat 8.5
Step 2: Go to the bin directory and invoke ./startup.sh to start the tomcat process
Step 3: Download the collector war file in to the webapps folder of tomcat. Since the tomcat is up and running, war file will deployed automatically\

Pinpoint Web

Step 1: Download the Pinpoint web war file in to the webapps folder of tomcat. Since the tomcat is up and running, war file will deployed automatically.
Step 2: Rename the pinpoint-web-1.6.1.war as ROOT.war.
Now you will be able to see the Pinpoint UI in the default tomcat port(8080).

@jack161641
Copy link
Author

jack161641 commented Jun 27, 2017

@xbblq7n export JAVA_HOME=/usr/java/jdk1.8.0_74 ,not use jdk 1.8,use jdk 1.7 home

@minwoo-jung
Copy link
Member

@xbblq7n You should see error log in weblogs.txt file.

@minwoo-jung
Copy link
Member

@jack161641 I do not know what you mean.

@minwoo-jung can I use collect pull date from agent?

@minwoo-jung minwoo-jung self-assigned this Jun 27, 2017
@jack161641
Copy link
Author

@minwoo-jung What I mean is how to use collect to get agent data, from the collect connection to the agent

@minwoo-jung
Copy link
Member

@jack161641 If you want to see agent data without pinpoint web, you can use and see agent data by connecting hbase.

@jack161641
Copy link
Author

@minwoo-jung I mean because of the reasons for the firewall, my agent can not access collect, but collect can access agent, the system default data is pushed from the agent to collect, but I hope to collect from the collect agent data

@minwoo-jung
Copy link
Member

minwoo-jung commented Jun 27, 2017

@jack161641 We don't call 'collect' with name of pinpoint collector.
Pinpoint don't provide function for your question. We don't recommend that collector pull agent data for various reasons.

@GilbertDai
Copy link

@minwoo-jung for what reasons ,Pinpoint do not support collector pull agent data?
( we deploy collector in our internal network , and agent is on the cloud, but collector service port is blocked by firewall , so we consider whether collector can pull agent data) thanks。

@vivekbny
Copy link

@xbblq7n You should see error log in weblogs.txt file.

@minwoo-jung i am not getting the above comment you mentioned. Can you please elaborate

@minwoo-jung
Copy link
Member

@xbblq7n
Exception message in weblogs.txt file.

2017-06-27 02:51:35 [ERROR](o.s.w.c.ContextLoader              ) Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'activeTraceService' defined in file [/root/apache-tomcat-8.0.44/webapps/pinpoint-web-1.6.2/WEB-INF/classes/com/navercorp/pinpoint/web/service/stat/ActiveTraceService.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [com.navercorp.pinpoint.web.dao.stat.ActiveTraceDao]: : Error creating bean with name 'activeTraceDaoFactory': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalStateException: AgentStat, AgentStatV2 tables do not exist; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'activeTraceDaoFactory': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalStateException: AgentStat, AgentStatV2 tables do not exist
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:185)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1137)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1040)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:434)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:729)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:940)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1816)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'activeTraceDaoFactory': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalStateException: AgentStat, AgentStatV2 tables do not exist
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1525)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:251)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1120)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1044)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:942)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:813)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
        ... 28 more
Caused by: java.lang.IllegalStateException: AgentStat, AgentStatV2 tables do not exist
        at com.navercorp.pinpoint.web.dao.hbase.stat.AgentStatDaoFactory.getDao(AgentStatDaoFactory.java:88)
        at com.navercorp.pinpoint.web.dao.hbase.stat.AgentStatDaoFactory$ActiveTraceDaoFactory.getObject(AgentStatDaoFactory.java:248)
        at com.navercorp.pinpoint.web.dao.hbase.stat.AgentStatDaoFactory$ActiveTraceDaoFactory.getObject(AgentStatDaoFactory.java:233)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
        ... 37 more`

did you create hbase table?

@minwoo-jung
Copy link
Member

@GilbertDai

Because Collector server process data for too many agent server, It is difficult to pull data of many agent.
If Collector pull data of agent, collector need to check many status connection and transmission status of data.
For example, WAS(web application server) listen connection from agent and accept client data.

Could change port of collector server to permitted at firewall?
You can change listen Port of collector server.

pinpoint-collector.properties

collector.tcpListenPort=9994
collector.udpStatListenPort=9995
collector.udpSpanListenPort=9996

@GilbertDai
Copy link

@minwoo-jung thanks for your answer, by the way,what the detail function of port 9994, 9995,9996

@minwoo-jung
Copy link
Member

minwoo-jung commented Jun 28, 2017

@GilbertDai
We use tcp to send meta data and agent info and use udp to send stat and transaction data.

@GilbertDai
Copy link

@minwoo-jung we have a app-A ,A is deployed on cloud,A will invoke app-B, B is deployed in our internal network , now collector and hbase is in our internal network, we use tomcat and suppose A
deal with 10,000 transactions per day. consider the internal network bandwidth(suppose 200M), If the agent of A push data to collector ,is ok? thanks.

@vivekbny
Copy link

@minwoo-jung I have the Hbase up and running.

@minwoo-jung
Copy link
Member

@GilbertDai
You could see for yourself. Network utilization depend on metadata count size, method call count in one transaction etc.
For reference
Try to raise sampling rate little by little.

pinpoint.config file in agent package

# 1 out of n transactions will be sampled where n is the rate. (20: 5%) 
37 profiler.sampling.rate=20 

@minwoo-jung
Copy link
Member

@xbblq7n

This exception message.
Caused by: java.lang.IllegalStateException: AgentStat, AgentStatV2 tables do not exist

did you check create table in your hbase server?

@vivekbny
Copy link

@minwoo-jung yes i have the tables created in the Hbase.

@minwoo-jung
Copy link
Member

@xbblq7n
check hbase config(ip, port) in pinpoint web.

@vivekbny
Copy link

@minwoo-jung
Below is my config
hbase.client.host=localhost
hbase.client.port=2181

hbase default:/hbase

hbase.zookeeper.znode.parent=/hbase

hbase timeout option==================================================================================

hbase default:true

hbase.ipc.client.tcpnodelay=true

hbase default:60000

hbase.rpc.timeout=10000

hbase default:Integer.MAX_VALUE

hbase.client.operation.timeout=10000

hbase socket read timeout. default: 200000

hbase.ipc.client.socket.timeout.read=20000

socket write timeout. hbase default: 600000

hbase.ipc.client.socket.timeout.write=30000

#==================================================================================

hbase client thread pool option

hbase.client.thread.max=64
hbase.client.threadPool.queueSize=5120

prestartAllCoreThreads

hbase.client.threadPool.prestart=false

#==================================================================================

hbase parallel scan options

hbase.client.parallel.scan.enable=true
hbase.client.parallel.scan.maxthreads=64
hbase.client.parallel.scan.maxthreadsperscan=16

@vivekbny
Copy link

@minwoo-jung

Also one more thing, today i download the code from the github and the i used the below command to run the pinpoint as docker
Command:
docker-compose up

sql started to download but after that hbase was not installed. below error is thrown

[root@r01052n0v pinpoint-master]# docker-compose up
Pulling pinpoint-mysql (mysql:5.7)...
5.7: Pulling from library/mysql
9f0706ba7422: Pull complete
2290e155d2d0: Pull complete
547981b8269f: Pull complete
2c9d42ed2f48: Pull complete
55e3122f1297: Pull complete
abc10bd84060: Pull complete
c0a5ce64f2b0: Pull complete
c4595eab8e90: Pull complete
098988cead35: Pull complete
300ca5fa5eea: Pull complete
43fdc4e3e690: Pull complete
Digest: sha256:d178dffba8d81afedc251498e227607934636e06228ac63d58b72f9e9ec271a6
Status: Downloaded newer image for mysql:5.7
Pulling pinpoint-hbase (naver/pinpoint-hbase:1.7.0-SNAPSHOT)...
ERROR: repository naver/pinpoint-hbase not found: does not exist or no pull access

@vivekbny
Copy link

@minwoo-jung Can you please help me on this?

@minwoo-jung
Copy link
Member

minwoo-jung commented Jun 29, 2017

@xbblq7n
Don't use SNAPSHOT version. I reecommend to use 1.6.2 version.

See hbase url and port configuration in pinpoint web. Check created table AgentStat, AgentStatV2 yourself using hbase shell.

And use create table script file 1.6.2 version.

@vivekbny
Copy link

vivekbny commented Jun 29, 2017

@minwoo-jung

Below is hbase url and port configuration in pinpoint web
hbase.client.host=localhost
hbase.client.port=2181

Checked the table AgentStat, AgentStatV2 is available via Hbase shell.

Also i see the Hbase is up and running in URL http://*******:16010/master-status

Also to run the Pinpoint as a docker. do we have any official images?

@vivekbny
Copy link

WebLogs.txt

@minwoo-jung

Attaching the logs of the Web. Still the UI shows 404 error. I am not able view it. Please advise

@minwoo-jung
Copy link
Member

@xbblq7n
I think that you need to underatand tomcat and how to deploy war. Please search web or read tomcat manual.

@vivekbny
Copy link

vivekbny commented Jul 4, 2017

@minwoo-jung I will look in to it.

Can you please tell me if we have pinpoint images to run as docker container.

@vivekbny
Copy link

vivekbny commented Jul 5, 2017

@minwoo-jung Can you please help on this. Do we have pinpoint images to run as docker container?

@vivekbny
Copy link

vivekbny commented Jul 7, 2017

@minwoo-jung Can you please update on this?

@minwoo-jung
Copy link
Member

minwoo-jung commented Jul 13, 2017

@xbblq7n
I'm sorry it has taken so long to reply to your question.
We don't provide docker image. But We guide using scipt( docker-compose.yml ).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants