Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

您好，我在练习"分析网站日志，打造访问地图"项目时遇到了如下错误，麻烦指导下，谢谢！ #4

Closed
bitian opened this issue May 5, 2017 · 9 comments
Closed

您好，我在练习"分析网站日志，打造访问地图"项目时遇到了如下错误，麻烦指导下，谢谢！ #4

bitian opened this issue May 5, 2017 · 9 comments

Comments

@bitian
Copy link

bitian commented May 5, 2017 •
edited by phodal

我在centos上安装了项目要求的一些工具，在运行下面这段代码时，最后一行会报错，将最后一行如果注释掉直接dump A就没有错误了，不知道什么原因。

image
报错记录如下：

2017-05-05 11:17:54,861 [Thread-20] WARN  org.apache.hadoop.mapred.LocalJobRunner - job_local265525317_0001
java.lang.Exception: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/protocol/ProtocolSocketFactory
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549)
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/protocol/ProtocolSocketFactory
	at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransportFactory.create(CommonsHttpTransportFactory.java:39)
	at org.elasticsearch.hadoop.rest.NetworkClient.selectNextNode(NetworkClient.java:99)
	at org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:82)
	at org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:59)
	at org.elasticsearch.hadoop.rest.RestClient.<init>(RestClient.java:92)
	at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:240)
	at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:546)
	at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:173)
	at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:149)
	at org.elasticsearch.hadoop.pig.EsStorage.putNext(EsStorage.java:192)
	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:75)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:144)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:275)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:65)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.protocol.ProtocolSocketFactory
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 29 more
2017-05-05 11:17:55,351 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2017-05-05 11:17:55,357 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local265525317_0001 has failed! Stop running all dependent jobs
2017-05-05 11:17:55,357 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2017-05-05 11:17:55,416 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2017-05-05 11:17:55,470 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2017-05-05 11:17:55,487 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!
2017-05-05 11:17:55,515 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: 

HadoopVersion	PigVersion	UserId	StartedAt	FinishedAt	Features
2.8.0	0.16.0	root	2017-05-05 11:17:41	2017-05-05 11:17:55	UNKNOWN

Failed!

Failed Jobs:
JobId	Alias	Feature	Message	Outputs
job_local265525317_0001	A,LOGS_BASE,RAW_LOGS	MAP_ONLY	Message: Job failed!	home/data_program/log,

Input(s):
Failed to read data from "/home/data_program/access2.log"

Output(s):
Failed to produce result in "home/data_program/log"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_local265525317_0001


2017-05-05 11:17:55,516 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2017-05-05 11:17:55,599 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job job_local265525317_0001 failed, hadoop does not return any error message
Details at logfile: /home/data_program/pig_1494008237155.log
2017-05-05 11:17:55,809 [main] INFO  org.apache.pig.Main - Pig script completed in 50 seconds and 751 milliseconds (50751 ms)

希望能得到你的帮助，谢谢！
@phodal
Copy link
Owner

phodal commented May 5, 2017

看看这个 elastic/elasticsearch-hadoop#586

你有没有加入相应的 jar 包？

@bitian
Copy link
Author

bitian commented May 6, 2017

jar包我都加入了，安装的pig0.16.0，将elasticsearch-hadoop下面的jar全部放在pig目录下了。

@phodal
Copy link
Owner

phodal commented May 6, 2017

@bitian

按照 http://stackoverflow.com/questions/17973970/how-to-solve-java-lang-noclassdeffounderror 上的解释，仍然是有可能缺少相应的 jar 包。

@yunzhongshu
Copy link

java.lang.ClassNotFoundException: org.apache.commons.httpclient.protocol.ProtocolSocketFactory ---缺少commons-httpclient-3.0.1的jar

@bitian
Copy link
Author

bitian commented May 8, 2017

你好，这个jar包应当放在那里，我下载下来也放到pig目录下了。

@bitian
Copy link
Author

bitian commented May 8, 2017

我把jar包加错地方了，现在加上没有这个错误了，但是又出现下面的错误：
2017-05-08 07:13:39,172 [LocalJobRunner Map Task Executor #0] INFO org.elasticsearch.hadoop.util.Version - Elasticsearch Hadoop v5.3.2 [d4a72b6494]
2017-05-08 07:13:40,010 [LocalJobRunner Map Task Executor #0] INFO org.apache.commons.httpclient.HttpMethodDirector - I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused)
2017-05-08 07:13:40,012 [LocalJobRunner Map Task Executor #0] INFO org.apache.commons.httpclient.HttpMethodDirector - Retrying request
2017-05-08 07:13:40,018 [LocalJobRunner Map Task Executor #0] INFO org.apache.commons.httpclient.HttpMethodDirector - I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused)
2017-05-08 07:13:40,018 [LocalJobRunner Map Task Executor #0] INFO org.apache.commons.httpclient.HttpMethodDirector - Retrying request
2017-05-08 07:13:40,019 [LocalJobRunner Map Task Executor #0] INFO org.apache.commons.httpclient.HttpMethodDirector - I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused)
2017-05-08 07:13:40,019 [LocalJobRunner Map Task Executor #0] INFO org.apache.commons.httpclient.HttpMethodDirector - Retrying request
2017-05-08 07:13:40,023 [LocalJobRunner Map Task Executor #0] ERROR org.elasticsearch.hadoop.rest.NetworkClient - Node [127.0.0.1:9200] failed (Connection refused (Connection refused)); no other nodes left - aborting...
2017-05-08 07:13:40,051 [Thread-20] INFO org.apache.hadoop.mapred.LocalJobRunner - map task executor complete.
2017-05-08 07:13:40,106 [Thread-20] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local1316418085_0001
java.lang.Exception: java.io.IOException: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549)
Caused by: java.io.IOException: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:83)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:144)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:275)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:65)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:250)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:546)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:173)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:149)
at org.elasticsearch.hadoop.pig.EsStorage.putNext(EsStorage.java:192)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:75)
... 18 more
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[127.0.0.1:9200]]
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:150)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:461)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:425)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:429)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:155)
at org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:627)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:243)
... 23 more
2017-05-08 07:13:40,286 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2017-05-08 07:13:40,287 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local1316418085_0001 has failed! Stop running all dependent jobs
2017-05-08 07:13:40,292 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2017-05-08 07:13:40,341 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2017-05-08 07:13:40,402 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2017-05-08 07:13:40,411 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!
2017-05-08 07:13:40,438 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.8.0 0.16.0 root 2017-05-08 07:13:28 2017-05-08 07:13:40 UNKNOWN

Failed!

Failed Jobs:
JobId Alias Feature Message Outputs
job_local1316418085_0001 A,LOGS_BASE,RAW_LOGS MAP_ONLY Message: Job failed! home/data_program/log,

Input(s):
Failed to read data from "/home/data_program/access2.log"

Output(s):
Failed to produce result in "home/data_program/log"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_local1316418085_0001

2017-05-08 07:13:40,439 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2017-05-08 07:13:40,506 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job job_local1316418085_0001 failed, hadoop does not return any error message
Details at logfile: /home/data_program/pig_1494252791860.log
2017-05-08 07:13:40,905 [main] INFO org.apache.pig.Main - Pig script completed in 35 seconds and 91 milliseconds (35091 ms)

@phodal
Copy link
Owner

phodal commented May 8, 2017

@bitian 日志上写着：

Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'

@bitian
Copy link
Author

bitian commented May 8, 2017

谢谢，我解决了这些问题。

@phodal phodal closed this as completed May 8, 2017
@bitian
Copy link
Author

bitian commented May 10, 2017

您好，最后还遇到一个问题，是关于country字段没有fielddata属性。如下所示：
image
应当怎么设置这个属性值，谢谢。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@phodal @yunzhongshu @bitian