Compilation #6
Comments
|
Hi, Sincerely, |
|
Hi,
produced the same results. Maybe because my hadoop version is 2.6.0 instead of 1.2.1? I suppose I have to find a way to compile it myself... |
|
Hi
Could you paste the error message or log here for debugging. Because It
works fine on my Hadoop 1 cluster. Thank you.
Antony
…On Apr 30, 2017 11:09 PM, "Stelios Barberakis" ***@***.***> wrote:
Hi,
Thank you for taking the time to rebuild with java7.
Unfortunately however, it didn't work. Running as
hadoop jar CloudBrush.jar -reads Ec10k -asm Ec10k_Brush -k 21 -readlen 36
-javaopts -Djava.util.Arrays.useLegacyMergeSort=True
produced the same results. Maybe because my hadoop version is 2.6.0
instead of 1.2.1? I suppose I have to find a way to compile it myself...
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#6 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAEPVWkGS79wh6BQYIYoMOvKEaFQI181ks5r1KQMgaJpZM4NLkpV>
.
|
|
In addition, could you give me more information about the "same result" means? Is it from environment, program, or configuration? Then please attach the log from CloudBrush and on the screen (stdout) if possible. It seems the stdout message may not be logged into the detail log file produced by Brush. Thank you. |
|
Hi, The environment info is:
Any logs I could found are presented below:
|
|
Hi Stelios
I have a comment that it may sounds weird: Could you please try to make the
"True" to be "true"?
I am afraid that it is due to the parameter type of Java which can be
accepted is boolean.
If it works, I need to apologize about that because of the typo in my first
comment.
Antony
…On May 1, 2017 1:03 AM, "Stelios Barberakis" ***@***.***> wrote:
Hi,
The environment info is:
uname -a
Linux clu04 3.2.0-120-generic #163-Ubuntu SMP Tue Dec 20 15:12:28 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux
java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
hadoop version
Hadoop 2.6.0.2.2.4.2-2
Subversion ***@***.***:hortonworks/hadoop.git -r
22a563ebe448969d07902aed869ac13c652b2872
Compiled by jenkins on 2015-03-31T19:40Z
Compiled with protoc 2.5.0
From source with checksum b3481c2cdbe2d181f2621331926e267
This command was run using /usr/hdp/2.2.4.2-2/hadoop/hado
op-common-2.6.0.2.2.4.2-2.jar
Any logs I could found are presented below:
- hadoop log:
Log Type: stderr
Log Upload Time: Sun Apr 30 19:32:18 +0300 2017
Log Length: 222
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.ipc.Server).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
Log Type: stdout
Log Upload Time: Sun Apr 30 19:32:18 +0300 2017
Log Length: 0
Log Type: syslog
Log Upload Time: Sun Apr 30 19:32:18 +0300 2017
Log Length: 1201661
Showing 4096 bytes of 1201661 total. Click here for the full log.
5 INFO [Thread-311] org.apache.hadoop.hdfs.DFSClient: Excluding datanode
:50010
2017-04-30 19:32:11,323 WARN [eventHandlingThread]
org.apache.hadoop.hdfs.DFSClient: Slow waitForAckedSeqno took 63298ms
(threshold=30000ms)
2017-04-30 19:32:11,365 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to
done location: hdfs:///mr-history/tmp/my_username/job_1486050547626_0068-
1493569802880-my_username-VerifyOverlap+Ec10k_Brush.tmp%
2F00%2Dpreprocess.prefix-1493569867402-55-34-FAILED-
default-1493569808376.jhist_tmp
2017-04-30 19:32:11,375 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying
hdfs:///user/my_username/.staging/job_1486050547626_0068/
job_1486050547626_0068_1_conf.xml to hdfs://:8020/mr-history/tmp/my
_username/job_1486050547626_0068_conf.xml_tmp
2017-04-30 19:32:11,428 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to
done location: hdfs://:8020/mr-history/tmp/my
_username/job_1486050547626_0068_conf.xml_tmp
2017-04-30 19:32:11,447 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp
to done: hdfs://:8020/mr-history/tmp/my_username/job_1486050547626_0068.summary_tmp
to hdfs://:8020/mr-history/tmp/my_username/job_1486050547626_0068.summary
2017-04-30 19:32:11,457 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp
to done: hdfs://:8020/mr-history/tmp/my_username/job_1486050547626_0068_conf.xml_tmp
to hdfs://:8020/mr-history/tmp/my_username/job_1486050547626_0068_conf.xml
2017-04-30 19:32:11,465 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp
to done: hdfs://:8020/mr-history/tmp/my_username/job_1486050547626_00
68-1493569802880-my_username-VerifyOverlap+Ec10k_Brush.tmp%
2F00%2Dpreprocess.prefix-1493569867402-55-34-FAILED-
default-1493569808376.jhist_tmp to hdfs:///mr-history/tmp/my_user
name/job_1486050547626_0068-1493569802880-my_username-
VerifyOverlap+Ec10k_Brush.tmp%2F00%2Dpreprocess.prefix-
1493569867402-55-34-FAILED-default-1493569808376.jhist
2017-04-30 19:32:11,475 INFO [Thread-308] org.apache.hadoop.mapreduce.jo
bhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler.
super.stop()
2017-04-30 19:32:11,478 INFO [Thread-308] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Setting job diagnostics to Task failed task_1486050547626_0068_r_000035
Job failed as tasks failed. failedMaps:0 failedReduces:1
2017-04-30 19:32:11,482 INFO [Thread-308] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
History url is http://:19888/jobhistory/job/job_1486050547626_0068
2017-04-30 19:32:11,505 INFO [Thread-308] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Waiting for application to be successfully unregistered.
2017-04-30 19:32:12,506 INFO [Thread-308] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0
AssignedReds:0 CompletedMaps:55 CompletedReds:35 ContAlloc:155 ContRel:31
HostLocal:44 RackLocal:11
2017-04-30 19:32:12,508 INFO [Thread-308] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Deleting staging directory hdfs:// /user/my_username/.staging/job
_1486050547626_0068
2017-04-30 19:32:12,520 INFO [Thread-308] org.apache.hadoop.ipc.Server:
Stopping server on 42201
2017-04-30 19:32:12,522 INFO [IPC Server listener on 42201]
org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 42201
2017-04-30 19:32:12,524 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2017-04-30 19:32:12,524 INFO [TaskHeartbeatHandler PingChecker]
org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler:
TaskHeartbeatHandler thread interrupted
- stdout:
============================================================
======================
Input: Ec10k
Workdir: Ec10k_Brush/
localBasePath: work/
HADOOP_MAPPERS = 50
HADOOP_REDUCERS = 50
HADOOP_JAVA_OPTS = -Djava.util.Arrays.useLegacyMergeSort=True
HADOOP_TIMEOUT = 0
HADOOP_LOCALNODES = 1000
READLENGTH = 36
K = 21
KMERCOV = 30.0
PWM MAJORITY = 0.6
PWM N = 0.1
KMER UPPER BOUND = 2000
KMER LOW BOUND = 1
TIPLENGTH = 360
MAXBUBBLELEN = 101
BUBBLEEDITRATE = 0.05
LOW_COV_THRESH = 1.0
MAX_LOW_COV_LEN = 72
RUN_STATS = null
N50_TARGET = -1
CONVERT_FA = null
== Starting time 2017-04-30 19:09:10
PreProcess:
Generate nonContained Reads: job_1486050547626_0064 221 s 20000 nodes
[20000 (100.00%) good reads, 720000 bp] 3493 contained_reads
Remove Redundant Reads: job_1486050547626_0065 91 s 3493 redundants 16507
nodes
Build High Frequency Kmer List: job_1486050547626_0066 373 s H_kmer: 0
Build Overlap Graph:
Match Prefix: job_1486050547626_0067 563 s 0 HKmer_skip
Verify Overlap:
- stderr:
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeForceCollapse(TimSort.java:426)
at java.util.TimSort.sort(TimSort.java:223)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeForceCollapse(TimSort.java:426)
at java.util.TimSort.sort(TimSort.java:223)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeForceCollapse(TimSort.java:426)
at java.util.TimSort.sort(TimSort.java:223)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException: Comparison method violates its
general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:873)
at Brush.VerifyOverlap.run(VerifyOverlap.java:381)
at Brush.BrushAssembler.buildOverlap(BrushAssembler.java:326)
at Brush.BrushAssembler.run(BrushAssembler.java:838)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at Brush.BrushAssembler.main(BrushAssembler.java:913)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
ssorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
thodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
- Attaching brush.details.log:
brush.details.log.txt
<https://github.com/ice91/CloudBrush/files/966845/brush.details.log.txt>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#6 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAEPVeqYcCGBWQGlFUkd13QqxhoRSWvQks5r1L7igaJpZM4NLkpV>
.
|
|
Hi Antony, Good point passing true as java boolean but unfortunately it didn't work. Logs are attached below (although they are probably identical to the previous ones) |
|
Hi @chefarov Sorry for inconvenient. After documentation survey, I doubt that the issue may from 1) the deprecated configuration parameter changed in Hadoop YARN; 2) or the default configuration of the parameters set on your host is configured as "final" (cannot be changed).
I've add the workaround that try to fix this issue and rebuild the jar file in my repository: Please take a look or try to run it if possible. We are very sorry for your incontinent. Antony |
|
This worked fine! Thank you so much for your efforts. |
Hi,
I have been trying to compile the project, doing things like:
Edit build/Brush/MANIFEST.MF to make it look like:
Then
jar cfm newBrush.jar build/Brush/MANIFEST.MF build/Brush/*.classRunning
hadoop jar newBrush.jarI get:
Could you provide us with your compilation process steps in your environment, or give us some insight on how to resolve this issue?
Thank you in advance
The text was updated successfully, but these errors were encountered: