Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

咨询一下提交的参数 #12

Closed
ace7 opened this issue Dec 31, 2016 · 8 comments
Closed

咨询一下提交的参数 #12

ace7 opened this issue Dec 31, 2016 · 8 comments

Comments

@ace7
Copy link

ace7 commented Dec 31, 2016

我的数据集有300万条数据，feature维度为1亿左右，每条数据非零值个数约200，用下面的参数提交，总是报143的错误（内存问题），想请教下你们训练 1 billion x 1 billion 的提交参数，谢谢！

--conf spark.driver.maxResultSize=12g
--conf spark.network.timeout=1200s
--conf spark.executor.heartbeatInterval=1200s
--master yarn-client
--num-executors 1000
--executor-cores 2
--executor-memory 12g
--driver-memory 10g
@takun2s
Copy link
Collaborator

takun2s commented Jan 2, 2017

可以把详细的错粘出来吗？

@ace7
Copy link
Author

ace7 commented Jan 3, 2017

主要代码是：
val (, trainData) = LibSvmReader.read(sc, trainDataPath, numFeatures)
val (, testData) = LibSvmReader.read(sc, testDataPath, numFeatures)
val model = LogisticRegression.run(trainData)
val prediction = model.classPredict(testData)
val auc = AreaUnderRoc.of(prediction.map { case ((x, l), (p, c)) => p -> l }.collect())
println(s"AUC: $auc")

运行日志如下：
[Stage 0:> (0 + 0) / 7000]
[Stage 0:> (0 + 62) / 7000]
[Stage 0:> (0 + 63) / 7000]
[Stage 0:> (0 + 75) / 7000]
[Stage 0:> (0 + 91) / 7000]
[Stage 0:> (0 + 103) / 7000]
[Stage 0:> (0 + 123) / 7000]
[Stage 0:> (0 + 155) / 7000]
[Stage 0:> (0 + 198) / 7000]
[Stage 0:> (0 + 234) / 7000]
[Stage 0:> (0 + 265) / 7000]
[Stage 0:> (0 + 300) / 7000]
[Stage 0:> (0 + 329) / 7000]
[Stage 0:> (0 + 357) / 7000]
[Stage 0:> (0 + 369) / 7000]
[Stage 0:> (0 + 393) / 7000]
[Stage 0:> (0 + 418) / 7000]
[Stage 0:> (0 + 442) / 7000]
[Stage 0:> (0 + 464) / 7000]
[Stage 0:> (0 + 485) / 7000]
[Stage 0:> (0 + 506) / 7000]
[Stage 0:> (0 + 524) / 7000]
[Stage 0:> (0 + 541) / 7000]
[Stage 0:> (0 + 558) / 7000]
[Stage 0:> (0 + 574) / 7000]
[Stage 0:> (0 + 590) / 7000]
[Stage 0:> (0 + 605) / 7000]
[Stage 0:> (0 + 621) / 7000]
[Stage 0:> (0 + 635) / 7000]
[Stage 0:> (0 + 647) / 7000]
[Stage 0:> (0 + 660) / 7000]
[Stage 0:> (0 + 670) / 7000]
[Stage 0:> (0 + 683) / 7000]
[Stage 0:> (0 + 695) / 7000]
[Stage 0:> (0 + 707) / 7000]
[Stage 0:> (0 + 716) / 7000]
[Stage 0:> (18 + 713) / 7000]
[Stage 0:> (29 + 715) / 7000]
[Stage 0:> (66 + 713) / 7000]
[Stage 0:> (77 + 713) / 7000]
[Stage 0:> (78 + 713) / 7000]
[Stage 0:> (80 + 713) / 7000]
[Stage 0:> (82 + 715) / 7000]
[Stage 0:> (88 + 715) / 7000]
[Stage 0:> (96 + 715) / 7000]
[Stage 0:> (123 + 712) / 7000]
[Stage 0:=> (143 + 710) / 7000]
[Stage 0:=> (178 + 710) / 7000]
[Stage 0:=> (206 + 708) / 7000]
[Stage 0:=> (222 + 707) / 7000]
[Stage 0:=> (228 + 707) / 7000]
[Stage 0:=> (239 + 707) / 7000]
[Stage 0:=> (249 + 715) / 7000]
[Stage 0:=> (259 + 715) / 7000]
[Stage 0:=> (267 + 715) / 7000]
[Stage 0:==> (274 + 714) / 7000]
[Stage 0:==> (278 + 714) / 7000]
[Stage 0:==> (290 + 714) / 7000]
[Stage 0:==> (307 + 714) / 7000]
[Stage 0:==> (308 + 714) / 7000]
[Stage 0:==> (311 + 714) / 7000]
[Stage 0:==> (314 + 714) / 7000]
[Stage 0:==> (316 + 715) / 7000]
[Stage 0:==> (318 + 715) / 7000]
[Stage 0:==> (333 + 714) / 7000]
[Stage 0:==> (362 + 712) / 7000]
[Stage 0:==> (398 + 711) / 7000]
[Stage 0:===> (438 + 712) / 7000]
[Stage 0:===> (471 + 711) / 7000]
[Stage 0:===> (491 + 710) / 7000]
[Stage 0:===> (506 + 710) / 7000]
[Stage 0:===> (530 + 709) / 7000]
[Stage 0:====> (551 + 713) / 7000]
[Stage 0:====> (578 + 709) / 7000]
[Stage 0:====> (599 + 709) / 7000]
[Stage 0:====> (609 + 708) / 7000]
[Stage 0:====> (615 + 708) / 7000]
[Stage 0:====> (633 + 714) / 7000]
[Stage 0:====> (645 + 714) / 7000]
[Stage 0:====> (647 + 714) / 7000]
[Stage 0:====> (650 + 714) / 7000]
[Stage 0:====> (659 + 714) / 7000]
[Stage 0:=====> (676 + 715) / 7000]
[Stage 0:=====> (688 + 715) / 7000]
[Stage 0:=====> (713 + 710) / 7000]
[Stage 0:=====> (742 + 708) / 7000]
[Stage 0:=====> (779 + 703) / 7000]
[Stage 0:======> (822 + 709) / 7000]
[Stage 0:======> (848 + 709) / 7000]
[Stage 0:======> (866 + 708) / 7000]
[Stage 0:======> (882 + 708) / 7000]
[Stage 0:======> (901 + 707) / 7000]
[Stage 0:======> (923 + 712) / 7000]
[Stage 0:======> (942 + 709) / 7000]
[Stage 0:=======> (953 + 709) / 7000]
[Stage 0:=======> (957 + 709) / 7000]
[Stage 0:=======> (971 + 709) / 7000]
[Stage 0:=======> (996 + 714) / 7000]
[Stage 0:=======> (1008 + 712) / 7000]
[Stage 0:=======> (1018 + 711) / 7000]
[Stage 0:=======> (1023 + 711) / 7000]
[Stage 0:=======> (1030 + 711) / 7000]
[Stage 0:=======> (1038 + 715) / 7000]
[Stage 0:=======> (1052 + 715) / 7000]
[Stage 0:=======> (1073 + 713) / 7000]
[Stage 0:=======> (1095 + 713) / 7000]
[Stage 0:========> (1135 + 707) / 7000]
[Stage 0:========> (1166 + 712) / 7000]
[Stage 0:========> (1193 + 710) / 7000]
[Stage 0:========> (1210 + 709) / 7000]
[Stage 0:========> (1231 + 708) / 7000]
[Stage 0:=========> (1257 + 705) / 7000]
[Stage 0:=========> (1287 + 707) / 7000]
[Stage 0:=========> (1314 + 703) / 7000]
[Stage 0:=========> (1334 + 702) / 7000]
[Stage 0:=========> (1343 + 700) / 7000]
[Stage 0:=========> (1364 + 698) / 7000]
[Stage 0:==========> (1386 + 705) / 7000]
[Stage 0:==========> (1395 + 705) / 7000]
[Stage 0:==========> (1397 + 705) / 7000]
[Stage 0:==========> (1403 + 705) / 7000]
[Stage 0:==========> (1412 + 704) / 7000]
[Stage 0:==========> (1428 + 709) / 7000]
[Stage 0:==========> (1453 + 709) / 7000]
[Stage 0:==========> (1496 + 707) / 7000]
[Stage 0:===========> (1531 + 704) / 7000]
[Stage 0:===========> (1582 + 697) / 7000]
[Stage 0:===========> (1620 + 703) / 7000]
[Stage 0:===========> (1641 + 702) / 7000]
[Stage 0:============> (1692 + 697) / 7000]
[Stage 0:============> (1728 + 693) / 7000]
[Stage 0:============> (1765 + 687) / 7000]
[Stage 0:=============> (1804 + 704) / 7000]
[Stage 0:=============> (1839 + 701) / 7000]
[Stage 0:=============> (1857 + 701) / 7000]
[Stage 0:=============> (1885 + 697) / 7000]
[Stage 0:==============> (1924 + 693) / 7000]
[Stage 0:==============> (1946 + 704) / 7000]
[Stage 0:==============> (1955 + 703) / 7000]
[Stage 0:==============> (1961 + 703) / 7000]
[Stage 0:==============> (1968 + 701) / 7000]
[Stage 0:==============> (1980 + 700) / 7000]
[Stage 0:==============> (2003 + 703) / 7000]
[Stage 0:==============> (2037 + 699) / 7000]
[Stage 0:===============> (2069 + 696) / 7000]
[Stage 0:===============> (2101 + 694) / 7000]
[Stage 0:===============> (2143 + 688) / 7000]
[Stage 0:===============> (2160 + 698) / 7000]
[Stage 0:===============> (2187 + 693) / 7000]
[Stage 0:================> (2211 + 691) / 7000]
[Stage 0:================> (2237 + 689) / 7000]
[Stage 0:================> (2271 + 685) / 7000]
[Stage 0:================> (2300 + 690) / 7000]
[Stage 0:================> (2326 + 689) / 7000]
[Stage 0:=================> (2337 + 688) / 7000]
[Stage 0:=================> (2363 + 687) / 7000]
[Stage 0:=================> (2396 + 682) / 7000]
[Stage 0:=================> (2417 + 691) / 7000]
[Stage 0:=================> (2439 + 689) / 7000]
[Stage 0:=================> (2452 + 688) / 7000]
[Stage 0:=================> (2459 + 687) / 7000]
[Stage 0:==================> (2488 + 684) / 7000]
[Stage 0:==================> (2521 + 684) / 7000]
[Stage 0:==================> (2569 + 675) / 7000]
[Stage 0:==================> (2598 + 674) / 7000]
[Stage 0:===================> (2636 + 671) / 7000]
[Stage 0:===================> (2678 + 665) / 7000]
[Stage 0:===================> (2705 + 681) / 7000]
[Stage 0:===================> (2739 + 678) / 7000]
[Stage 0:====================> (2764 + 676) / 7000]
[Stage 0:====================> (2801 + 674) / 7000]
[Stage 0:====================> (2852 + 664) / 7000]
[Stage 0:=====================> (2886 + 678) / 7000]
[Stage 0:=====================> (2917 + 678) / 7000]
[Stage 0:=====================> (2950 + 676) / 7000]
[Stage 0:=====================> (2986 + 672) / 7000]
[Stage 0:======================> (3021 + 667) / 7000]
[Stage 0:======================> (3039 + 679) / 7000]
[Stage 0:======================> (3051 + 678) / 7000]
[Stage 0:======================> (3064 + 679) / 7000]
[Stage 0:======================> (3069 + 678) / 7000]
[Stage 0:======================> (3093 + 674) / 7000]
[Stage 0:======================> (3116 + 674) / 7000]
[Stage 0:======================> (3142 + 671) / 7000]
[Stage 0:=======================> (3169 + 671) / 7000]
[Stage 0:=======================> (3202 + 671) / 7000]
[Stage 0:=======================> (3230 + 669) / 7000]
[Stage 0:=======================> (3256 + 671) / 7000]
[Stage 0:=======================> (3278 + 667) / 7000]
[Stage 0:========================> (3306 + 661) / 7000]
[Stage 0:========================> (3325 + 660) / 7000]
[Stage 0:========================> (3362 + 657) / 7000]
[Stage 0:========================> (3393 + 666) / 7000]
[Stage 0:========================> (3414 + 666) / 7000]
[Stage 0:=========================> (3434 + 664) / 7000]
[Stage 0:=========================> (3459 + 660) / 7000]
[Stage 0:=========================> (3488 + 660) / 7000]
[Stage 0:=========================> (3505 + 662) / 7000]
[Stage 0:=========================> (3520 + 661) / 7000]
[Stage 0:=========================> (3527 + 661) / 7000]
[Stage 0:=========================> (3548 + 655) / 7000]
[Stage 0:==========================> (3574 + 653) / 7000]
[Stage 0:==========================> (3609 + 657) / 7000]
[Stage 0:==========================> (3641 + 653) / 7000]
[Stage 0:==========================> (3683 + 649) / 7000]
[Stage 0:===========================> (3717 + 644) / 7000]
[Stage 0:===========================> (3753 + 642) / 7000]
[Stage 0:===========================> (3790 + 644) / 7000]
[Stage 0:===========================> (3808 + 642) / 7000]
[Stage 0:===========================> (3835 + 635) / 7000]
[Stage 0:============================> (3878 + 626) / 7000]
[Stage 0:============================> (3923 + 622) / 7000]
[Stage 0:============================> (3953 + 628) / 7000]
[Stage 0:============================> (3980 + 627) / 7000]
[Stage 0:=============================> (4012 + 621) / 7000]
[Stage 0:=============================> (4042 + 617) / 7000]
[Stage 0:=============================> (4068 + 614) / 7000]
[Stage 0:=============================> (4090 + 619) / 7000]
[Stage 0:=============================> (4107 + 614) / 7000]
[Stage 0:=============================> (4115 + 611) / 7000]
[Stage 0:==============================> (4131 + 607) / 7000]
[Stage 0:==============================> (4153 + 602) / 7000]
[Stage 0:==============================> (4178 + 595) / 7000]
[Stage 0:==============================> (4195 + 586) / 7000]
[Stage 0:==============================> (4225 + 579) / 7000]
[Stage 0:==============================> (4249 + 573) / 7000]
[Stage 0:===============================> (4272 + 567) / 7000]
[Stage 0:===============================> (4291 + 707) / 7000]
[Stage 0:===============================> (4303 + 706) / 7000]
[Stage 0:===============================> (4318 + 703) / 7000]
[Stage 0:===============================> (4344 + 696) / 7000]
[Stage 0:===============================> (4379 + 688) / 7000]
[Stage 0:================================> (4395 + 686) / 7000]
[Stage 0:================================> (4413 + 684) / 7000]
[Stage 0:================================> (4437 + 678) / 7000]
[Stage 0:================================> (4460 + 675) / 7000]
[Stage 0:================================> (4474 + 672) / 7000]
[Stage 0:================================> (4488 + 675) / 7000]
[Stage 0:================================> (4498 + 675) / 7000]
[Stage 0:================================> (4509 + 674) / 7000]
[Stage 0:================================> (4523 + 671) / 7000]
[Stage 0:=================================> (4552 + 668) / 7000]
[Stage 0:=================================> (4605 + 639) / 7000]
[Stage 0:=================================> (4632 + 630) / 7000]
[Stage 0:=================================> (4660 + 628) / 7000]
[Stage 0:==================================> (4691 + 624) / 7000]
[Stage 0:===================================> (4809 + 528) / 7000]
[Stage 0:===================================> (4838 + 518) / 7000]
[Stage 0:===================================> (4860 + 518) / 7000]
[Stage 0:===================================> (4881 + 516) / 7000]
[Stage 0:===================================> (4913 + 509) / 7000]
[Stage 0:===================================> (4939 + 509) / 7000]
[Stage 0:====================================> (4963 + 513) / 7000]
[Stage 0:====================================> (4985 + 510) / 7000]
[Stage 0:====================================> (5012 + 510) / 7000]
[Stage 0:====================================> (5036 + 508) / 7000]
[Stage 0:====================================> (5057 + 505) / 7000]
[Stage 0:====================================> (5067 + 509) / 7000]
[Stage 0:====================================> (5075 + 508) / 7000]
[Stage 0:=====================================> (5080 + 508) / 7000]
[Stage 0:=====================================> (5095 + 508) / 7000]
[Stage 0:=====================================> (5115 + 505) / 7000]
[Stage 0:=====================================> (5126 + 506) / 7000]
[Stage 0:=====================================> (5144 + 505) / 7000]
[Stage 0:=====================================> (5160 + 505) / 7000]
[Stage 0:=====================================> (5172 + 505) / 7000]
[Stage 0:=====================================> (5190 + 503) / 7000]
[Stage 0:=====================================> (5202 + 502) / 7000]
[Stage 0:=====================================> (5213 + 501) / 7000]
[Stage 0:======================================> (5226 + 501) / 7000]
[Stage 0:======================================> (5248 + 499) / 7000]
[Stage 0:======================================> (5270 + 498) / 7000]
[Stage 0:======================================> (5285 + 497) / 7000]
[Stage 0:======================================> (5299 + 495) / 7000]
[Stage 0:======================================> (5314 + 494) / 7000]
[Stage 0:======================================> (5335 + 491) / 7000]
[Stage 0:======================================> (5347 + 490) / 7000]
[Stage 0:=======================================> (5359 + 490) / 7000]
[Stage 0:=======================================> (5371 + 488) / 7000]
[Stage 0:=======================================> (5379 + 488) / 7000]
[Stage 0:=======================================> (5391 + 486) / 7000]
[Stage 0:=======================================> (5420 + 481) / 7000]
[Stage 0:=======================================> (5442 + 480) / 7000]
[Stage 0:=======================================> (5463 + 479) / 7000]
[Stage 0:=======================================> (5489 + 476) / 7000]
[Stage 0:========================================> (5520 + 473) / 7000]
[Stage 0:========================================> (5534 + 472) / 7000]
[Stage 0:========================================> (5553 + 476) / 7000]
[Stage 0:========================================> (5572 + 473) / 7000]
[Stage 0:========================================> (5601 + 465) / 7000]
[Stage 0:========================================> (5620 + 462) / 7000]
[Stage 0:=========================================> (5647 + 459) / 7000]
[Stage 0:=========================================> (5666 + 459) / 7000]
[Stage 0:=========================================> (5691 + 456) / 7000]
[Stage 0:=========================================> (5714 + 451) / 7000]
[Stage 0:=========================================> (5731 + 450) / 7000]
[Stage 0:=========================================> (5745 + 449) / 7000]
[Stage 0:=========================================> (5751 + 453) / 7000]
[Stage 0:=========================================> (5759 + 453) / 7000]
[Stage 0:==========================================> (5767 + 453) / 7000]
[Stage 0:==========================================> (5785 + 449) / 7000]
[Stage 0:==========================================> (5800 + 447) / 7000]
[Stage 0:==========================================> (5821 + 713) / 7000]
[Stage 0:==========================================> (5831 + 712) / 7000]
[Stage 0:==========================================> (5844 + 711) / 7000]
[Stage 0:==========================================> (5861 + 708) / 7000]
[Stage 0:==========================================> (5873 + 707) / 7000]
[Stage 0:==========================================> (5882 + 708) / 7000]
[Stage 0:==========================================> (5895 + 708) / 7000]
[Stage 0:===========================================> (5911 + 708) / 7000]
[Stage 0:===========================================> (5928 + 704) / 7000]
[Stage 0:===========================================> (5944 + 702) / 7000]
[Stage 0:===========================================> (5956 + 713) / 7000]
[Stage 0:===========================================> (5965 + 712) / 7000]
[Stage 0:===========================================> (5982 + 709) / 7000]
[Stage 0:===========================================> (5997 + 704) / 7000]
[Stage 0:===========================================> (6003 + 704) / 7000]
[Stage 0:============================================> (6056 + 681) / 7000]
[Stage 0:============================================> (6083 + 669) / 7000]
[Stage 0:============================================> (6100 + 664) / 7000]
[Stage 0:============================================> (6125 + 658) / 7000]
[Stage 0:=============================================> (6280 + 532) / 7000]
[Stage 0:==============================================> (6320 + 680) / 7000]
[Stage 0:==============================================> (6339 + 661) / 7000]
[Stage 0:==============================================> (6364 + 636) / 7000]
[Stage 0:==============================================> (6383 + 617) / 7000]
[Stage 0:==============================================> (6398 + 602) / 7000]
[Stage 0:==============================================> (6427 + 573) / 7000]
[Stage 0:==============================================> (6447 + 553) / 7000]
[Stage 0:===============================================> (6458 + 542) / 7000]
[Stage 0:===============================================> (6478 + 522) / 7000]
[Stage 0:===============================================> (6505 + 495) / 7000]
[Stage 0:===============================================> (6544 + 456) / 7000]
[Stage 0:===============================================> (6570 + 430) / 7000]
[Stage 0:================================================> (6589 + 411) / 7000]
[Stage 0:================================================> (6604 + 396) / 7000]
[Stage 0:================================================> (6612 + 388) / 7000]
[Stage 0:================================================> (6640 + 360) / 7000]
[Stage 0:================================================> (6659 + 341) / 7000]
[Stage 0:================================================> (6676 + 324) / 7000]
[Stage 0:================================================> (6706 + 294) / 7000]
[Stage 0:==================================================>(6866 + 134) / 7000]
[Stage 0:==================================================>(6888 + 112) / 7000]
[Stage 0:==================================================>(6892 + 108) / 7000]
[Stage 0:==================================================>(6893 + 107) / 7000]
[Stage 0:==================================================>(6899 + 101) / 7000]
[Stage 0:===================================================>(6901 + 99) / 7000]
[Stage 0:===================================================>(6904 + 96) / 7000]
[Stage 0:===================================================>(6908 + 92) / 7000]
[Stage 0:===================================================>(6915 + 85) / 7000]
[Stage 0:===================================================>(6921 + 79) / 7000]
[Stage 0:===================================================>(6928 + 72) / 7000]
[Stage 0:===================================================>(6934 + 66) / 7000]
[Stage 0:===================================================>(6935 + 65) / 7000]
[Stage 0:===================================================>(6944 + 56) / 7000]
[Stage 0:===================================================>(6948 + 52) / 7000]
[Stage 0:===================================================>(6954 + 46) / 7000]
[Stage 0:===================================================>(6959 + 41) / 7000]
[Stage 0:===================================================>(6965 + 35) / 7000]
[Stage 0:===================================================>(6968 + 32) / 7000]
[Stage 0:===================================================>(6981 + 19) / 7000]
[Stage 0:====================================================>(6992 + 8) / 7000]
[Stage 0:====================================================>(6997 + 3) / 7000]
[Stage 0:====================================================>(6999 + 1) / 7000]
[Stage 1:> (0 + 83) / 83]
[Stage 1:> (0 + 83) / 83]
[Stage 1:> (0 + 83) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:53:07 ERROR YarnScheduler: Lost executor 171 on bigdata-serv1722.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000172 on host: bigdata-serv1722.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:53:09 ERROR YarnScheduler: Lost executor 100 on bigdata-serv1733.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000101 on host: bigdata-serv1733.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:53:13 ERROR YarnScheduler: Lost executor 194 on bigdata-serv1776.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000195 on host: bigdata-serv1776.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:53:13 ERROR YarnScheduler: Lost executor 669 on bigdata-serv1775.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000670 on host: bigdata-serv1775.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 81) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:53:18 ERROR YarnScheduler: Lost executor 571 on bigdata-serv1725.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000572 on host: bigdata-serv1725.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:53:25 ERROR YarnScheduler: Lost executor 327 on bigdata-serv1709.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000328 on host: bigdata-serv1709.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:53:26 ERROR YarnScheduler: Lost executor 688 on bigdata-serv1732.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000689 on host: bigdata-serv1732.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:53:29 ERROR YarnScheduler: Lost executor 502 on bigdata-serv1598.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000503 on host: bigdata-serv1598.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:53:33 ERROR YarnScheduler: Lost executor 377 on bigdata-serv1601.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000378 on host: bigdata-serv1601.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:53:39 ERROR YarnScheduler: Lost executor 647 on bigdata-serv1616.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000648 on host: bigdata-serv1616.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:53:42 ERROR YarnScheduler: Lost executor 563 on bigdata-serv1716.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000564 on host: bigdata-serv1716.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:53:48 ERROR YarnScheduler: Lost executor 184 on bigdata-serv1682.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000185 on host: bigdata-serv1682.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:54:02 ERROR YarnScheduler: Lost executor 174 on bigdata-serv812.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000175 on host: bigdata-serv812.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:54:02 ERROR YarnScheduler: Lost executor 700 on bigdata-serv85.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000701 on host: bigdata-serv85.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:54:04 ERROR YarnScheduler: Lost executor 197 on bigdata-serv1767.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000198 on host: bigdata-serv1767.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]17/01/01 22:54:04 ERROR YarnScheduler: Lost executor 201 on bigdata-serv379.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000202 on host: bigdata-serv379.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:04 ERROR YarnScheduler: Lost executor 340 on bigdata-serv119.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000341 on host: bigdata-serv119.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 83) / 83]17/01/01 22:54:05 ERROR YarnScheduler: Lost executor 380 on bigdata-serv06.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000381 on host: bigdata-serv06.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:05 ERROR YarnScheduler: Lost executor 67 on bigdata-serv981.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000068 on host: bigdata-serv981.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:05 ERROR YarnScheduler: Lost executor 409 on bigdata-serv241.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000410 on host: bigdata-serv241.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:05 ERROR YarnScheduler: Lost executor 530 on bigdata-serv905.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000531 on host: bigdata-serv905.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:05 ERROR YarnScheduler: Lost executor 16 on bigdata-serv19.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000017 on host: bigdata-serv19.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:05 ERROR YarnScheduler: Lost executor 574 on bigdata-serv01.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000575 on host: bigdata-serv01.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:05 ERROR YarnScheduler: Lost executor 653 on bigdata-serv114.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000654 on host: bigdata-serv114.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 76) / 83]17/01/01 22:54:05 ERROR YarnScheduler: Lost executor 335 on bigdata-serv311.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000336 on host: bigdata-serv311.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 75) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:54:07 ERROR YarnScheduler: Lost executor 362 on bigdata-serv889.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000363 on host: bigdata-serv889.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:07 ERROR YarnScheduler: Lost executor 687 on bigdata-serv738.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000688 on host: bigdata-serv738.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:07 ERROR YarnScheduler: Lost executor 306 on bigdata-serv625.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000307 on host: bigdata-serv625.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:07 ERROR YarnScheduler: Lost executor 435 on bigdata-serv496.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000436 on host: bigdata-serv496.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:07 ERROR YarnScheduler: Lost executor 26 on bigdata-serv518.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000027 on host: bigdata-serv518.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 78) / 83]17/01/01 22:54:07 ERROR YarnScheduler: Lost executor 575 on bigdata-serv678.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000576 on host: bigdata-serv678.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 83) / 83]17/01/01 22:54:07 ERROR YarnScheduler: Lost executor 358 on bigdata-serv05.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000359 on host: bigdata-serv05.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:54:08 ERROR YarnScheduler: Lost executor 70 on bigdata-serv680.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000071 on host: bigdata-serv680.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:08 ERROR YarnScheduler: Lost executor 301 on bigdata-serv866.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000302 on host: bigdata-serv866.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:08 ERROR YarnScheduler: Lost executor 87 on bigdata-serv185.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000088 on host: bigdata-serv185.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:08 ERROR YarnScheduler: Lost executor 289 on bigdata-serv487.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000290 on host: bigdata-serv487.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 79) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:54:10 ERROR YarnScheduler: Lost executor 344 on bigdata-serv964.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000345 on host: bigdata-serv964.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:10 ERROR YarnScheduler: Lost executor 370 on bigdata-serv498.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000371 on host: bigdata-serv498.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:10 ERROR YarnScheduler: Lost executor 24 on bigdata-serv61.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000025 on host: bigdata-serv61.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:10 ERROR YarnScheduler: Lost executor 57 on bigdata-serv221.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000058 on host: bigdata-serv221.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:10 ERROR YarnScheduler: Lost executor 467 on bigdata-serv1781.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000468 on host: bigdata-serv1781.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:54:13 ERROR YarnScheduler: Lost executor 321 on bigdata-serv403.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000322 on host: bigdata-serv403.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:54:19 ERROR YarnScheduler: Lost executor 640 on bigdata-serv39.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000641 on host: bigdata-serv39.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:54:21 ERROR YarnScheduler: Lost executor 277 on bigdata-serv108.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000278 on host: bigdata-serv108.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:54:28 ERROR YarnScheduler: Lost executor 710 on bigdata-serv1028.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000711 on host: bigdata-serv1028.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]
[Stage 1:> (0 + 83) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:57:11 ERROR YarnScheduler: Lost executor 60 on bigdata-serv1615.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000061 on host: bigdata-serv1615.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:57:20 ERROR YarnScheduler: Lost executor 136 on bigdata-serv1689.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000137 on host: bigdata-serv1689.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:57:44 ERROR YarnScheduler: Lost executor 705 on bigdata-serv465.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000706 on host: bigdata-serv465.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:57:52 ERROR YarnScheduler: Lost executor 438 on bigdata-serv03.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000439 on host: bigdata-serv03.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:57:53 ERROR YarnScheduler: Lost executor 304 on bigdata-serv683.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000305 on host: bigdata-serv683.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:57:56 ERROR YarnScheduler: Lost executor 515 on bigdata-serv932.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000516 on host: bigdata-serv932.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:58:25 ERROR YarnScheduler: Lost executor 585 on bigdata-serv933.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000586 on host: bigdata-serv933.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:58:29 ERROR YarnScheduler: Lost executor 241 on bigdata-serv173.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000242 on host: bigdata-serv173.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:58:31 ERROR YarnScheduler: Lost executor 44 on bigdata-serv164.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000045 on host: bigdata-serv164.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:34 ERROR YarnScheduler: Lost executor 25 on bigdata-serv1462.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000026 on host: bigdata-serv1462.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:34 ERROR YarnScheduler: Lost executor 260 on bigdata-serv186.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000261 on host: bigdata-serv186.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 81) / 83]17/01/01 22:58:34 ERROR YarnScheduler: Lost executor 145 on bigdata-serv310.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000146 on host: bigdata-serv310.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:58:36 ERROR YarnScheduler: Lost executor 264 on bigdata-serv368.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000265 on host: bigdata-serv368.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:37 ERROR YarnScheduler: Lost executor 147 on bigdata-serv60.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000148 on host: bigdata-serv60.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:37 ERROR YarnScheduler: Lost executor 457 on bigdata-serv479.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000458 on host: bigdata-serv479.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:37 ERROR YarnScheduler: Lost executor 661 on bigdata-serv585.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000662 on host: bigdata-serv585.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:37 ERROR YarnScheduler: Lost executor 76 on bigdata-serv1199.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000077 on host: bigdata-serv1199.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:38 ERROR YarnScheduler: Lost executor 66 on bigdata-serv1526.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000067 on host: bigdata-serv1526.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:38 ERROR YarnScheduler: Lost executor 425 on bigdata-serv238.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000426 on host: bigdata-serv238.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:38 ERROR YarnScheduler: Lost executor 181 on bigdata-serv1455.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000182 on host: bigdata-serv1455.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]17/01/01 22:58:39 ERROR YarnScheduler: Lost executor 678 on bigdata-serv576.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000679 on host: bigdata-serv576.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:39 ERROR YarnScheduler: Lost executor 605 on bigdata-serv2083.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000606 on host: bigdata-serv2083.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 80) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:58:40 ERROR YarnScheduler: Lost executor 216 on bigdata-serv1923.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000217 on host: bigdata-serv1923.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:40 ERROR YarnScheduler: Lost executor 99 on bigdata-serv514.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000100 on host: bigdata-serv514.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:58:41 ERROR YarnScheduler: Lost executor 664 on bigdata-serv754.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000665 on host: bigdata-serv754.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:41 ERROR YarnScheduler: Lost executor 441 on bigdata-serv1446.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000442 on host: bigdata-serv1446.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 81) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:58:42 ERROR YarnScheduler: Lost executor 37 on bigdata-serv1833.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000038 on host: bigdata-serv1833.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:58:43 ERROR YarnScheduler: Lost executor 215 on bigdata-serv1343.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000216 on host: bigdata-serv1343.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:45 ERROR YarnScheduler: Lost executor 620 on bigdata-serv1305.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000621 on host: bigdata-serv1305.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:45 ERROR YarnScheduler: Lost executor 116 on bigdata-serv1266.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000117 on host: bigdata-serv1266.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:45 ERROR YarnScheduler: Lost executor 691 on bigdata-serv693.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000692 on host: bigdata-serv693.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:45 ERROR YarnScheduler: Lost executor 355 on bigdata-serv1509.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000356 on host: bigdata-serv1509.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:45 ERROR YarnScheduler: Lost executor 357 on bigdata-serv1415.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000358 on host: bigdata-serv1415.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:46 ERROR YarnScheduler: Lost executor 693 on bigdata-serv344.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000694 on host: bigdata-serv344.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:46 ERROR YarnScheduler: Lost executor 353 on bigdata-serv1473.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000354 on host: bigdata-serv1473.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]17/01/01 22:58:46 ERROR YarnScheduler: Lost executor 168 on bigdata-serv1811.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000169 on host: bigdata-serv1811.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 83) / 83]17/01/01 22:58:47 ERROR YarnScheduler: Lost executor 654 on bigdata-serv834.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000655 on host: bigdata-serv834.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:48 ERROR YarnScheduler: Lost executor 630 on bigdata-serv1817.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000631 on host: bigdata-serv1817.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:48 ERROR YarnScheduler: Lost executor 114 on bigdata-serv81.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000115 on host: bigdata-serv81.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:49 ERROR YarnScheduler: Lost executor 503 on bigdata-serv1410.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000504 on host: bigdata-serv1410.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:49 ERROR YarnScheduler: Lost executor 497 on bigdata-serv1150.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000498 on host: bigdata-serv1150.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:49 ERROR YarnScheduler: Lost executor 754 on bigdata-serv221.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000755 on host: bigdata-serv221.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:49 ERROR YarnScheduler: Lost executor 150 on bigdata-serv1210.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000151 on host: bigdata-serv1210.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:50 ERROR YarnScheduler: Lost executor 569 on bigdata-serv2123.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000570 on host: bigdata-serv2123.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]17/01/01 22:58:51 ERROR YarnScheduler: Lost executor 238 on bigdata-serv1461.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000239 on host: bigdata-serv1461.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:51 ERROR YarnScheduler: Lost executor 112 on bigdata-serv2263.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000113 on host: bigdata-serv2263.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 83) / 83]17/01/01 22:58:52 ERROR YarnScheduler: Lost executor 419 on bigdata-serv2189.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000420 on host: bigdata-serv2189.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:52 ERROR YarnScheduler: Lost executor 637 on bigdata-serv1093.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000638 on host: bigdata-serv1093.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:52 ERROR YarnScheduler: Lost executor 220 on bigdata-serv2029.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000221 on host: bigdata-serv2029.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:54 ERROR YarnScheduler: Lost executor 383 on bigdata-serv1512.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000384 on host: bigdata-serv1512.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:54 ERROR YarnScheduler: Lost executor 586 on bigdata-serv1507.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000587 on host: bigdata-serv1507.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:57 ERROR YarnScheduler: Lost executor 97 on bigdata-serv1822.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000098 on host: bigdata-serv1822.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:58 ERROR YarnScheduler: Lost executor 322 on bigdata-serv1847.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000323 on host: bigdata-serv1847.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:58 ERROR YarnScheduler: Lost executor 420 on bigdata-serv1978.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000421 on host: bigdata-serv1978.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:58:58 ERROR YarnScheduler: Lost executor 196 on bigdata-serv2149.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000197 on host: bigdata-serv2149.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:59:01 ERROR YarnScheduler: Lost executor 246 on bigdata-serv1922.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000247 on host: bigdata-serv1922.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:59:01 ERROR YarnScheduler: Lost executor 517 on bigdata-serv2032.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000518 on host: bigdata-serv2032.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 81) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:59:06 ERROR YarnScheduler: Lost executor 625 on bigdata-serv1301.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000626 on host: bigdata-serv1301.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 22:59:07 ERROR YarnScheduler: Lost executor 432 on bigdata-serv1539.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000433 on host: bigdata-serv1539.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:59:10 ERROR YarnScheduler: Lost executor 729 on bigdata-serv85.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000730 on host: bigdata-serv85.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:59:12 ERROR YarnScheduler: Lost executor 498 on bigdata-serv2037.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000499 on host: bigdata-serv2037.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:59:13 ERROR YarnScheduler: Lost executor 611 on bigdata-serv1182.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000612 on host: bigdata-serv1182.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 22:59:47 ERROR YarnScheduler: Lost executor 286 on bigdata-serv1370.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000287 on host: bigdata-serv1370.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]
[Stage 1:> (0 + 83) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:01:51 ERROR YarnScheduler: Lost executor 714 on bigdata-serv1594.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000715 on host: bigdata-serv1594.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:02:03 ERROR YarnScheduler: Lost executor 312 on bigdata-serv2078.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000313 on host: bigdata-serv2078.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:02:16 ERROR YarnScheduler: Lost executor 631 on bigdata-serv2031.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000632 on host: bigdata-serv2031.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:02:23 ERROR YarnScheduler: Lost executor 178 on bigdata-serv1633.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000179 on host: bigdata-serv1633.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:02:23 ERROR YarnScheduler: Lost executor 755 on bigdata-serv61.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000756 on host: bigdata-serv61.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:02:23 ERROR YarnScheduler: Lost executor 36 on bigdata-serv1359.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000037 on host: bigdata-serv1359.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:02:23 ERROR YarnScheduler: Lost executor 473 on bigdata-serv697.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000474 on host: bigdata-serv697.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 79) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:02:30 ERROR YarnScheduler: Lost executor 365 on bigdata-serv1607.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000366 on host: bigdata-serv1607.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:02:35 ERROR YarnScheduler: Lost executor 723 on bigdata-serv1598.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000724 on host: bigdata-serv1598.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:02:40 ERROR YarnScheduler: Lost executor 674 on bigdata-serv1629.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000675 on host: bigdata-serv1629.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:02:45 ERROR YarnScheduler: Lost executor 667 on bigdata-serv1686.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000668 on host: bigdata-serv1686.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:02:50 ERROR YarnScheduler: Lost executor 256 on bigdata-serv1366.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000257 on host: bigdata-serv1366.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:02:53 ERROR YarnScheduler: Lost executor 69 on bigdata-serv1166.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000070 on host: bigdata-serv1166.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:02:55 ERROR YarnScheduler: Lost executor 243 on bigdata-serv162.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000244 on host: bigdata-serv162.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:00 ERROR YarnScheduler: Lost executor 363 on bigdata-serv692.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000364 on host: bigdata-serv692.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:03 ERROR YarnScheduler: Lost executor 200 on bigdata-serv969.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000201 on host: bigdata-serv969.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:04 ERROR YarnScheduler: Lost executor 394 on bigdata-serv130.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000395 on host: bigdata-serv130.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:05 ERROR YarnScheduler: Lost executor 202 on bigdata-serv249.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000203 on host: bigdata-serv249.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:05 ERROR YarnScheduler: Lost executor 572 on bigdata-serv120.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000573 on host: bigdata-serv120.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 81) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:07 ERROR YarnScheduler: Lost executor 369 on bigdata-serv795.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000370 on host: bigdata-serv795.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:09 ERROR YarnScheduler: Lost executor 633 on bigdata-serv1074.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000634 on host: bigdata-serv1074.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:09 ERROR YarnScheduler: Lost executor 474 on bigdata-serv1005.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000475 on host: bigdata-serv1005.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:09 ERROR YarnScheduler: Lost executor 371 on bigdata-serv910.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000372 on host: bigdata-serv910.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 80) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:10 ERROR YarnScheduler: Lost executor 50 on bigdata-serv1566.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000051 on host: bigdata-serv1566.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:10 ERROR YarnScheduler: Lost executor 141 on bigdata-serv2308.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000142 on host: bigdata-serv2308.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 81) / 83]17/01/01 23:03:10 ERROR YarnScheduler: Lost executor 96 on bigdata-serv1029.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000097 on host: bigdata-serv1029.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 80) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:12 ERROR YarnScheduler: Lost executor 125 on bigdata-serv1110.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000126 on host: bigdata-serv1110.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:12 ERROR YarnScheduler: Lost executor 681 on bigdata-serv453.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000682 on host: bigdata-serv453.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]17/01/01 23:03:13 ERROR YarnScheduler: Lost executor 143 on bigdata-serv2126.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000144 on host: bigdata-serv2126.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:13 ERROR YarnScheduler: Lost executor 368 on bigdata-serv33.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000369 on host: bigdata-serv33.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 83) / 83]17/01/01 23:03:15 ERROR YarnScheduler: Lost executor 5 on bigdata-serv755.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000006 on host: bigdata-serv755.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:17 ERROR YarnScheduler: Lost executor 771 on bigdata-serv186.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000772 on host: bigdata-serv186.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:17 ERROR YarnScheduler: Lost executor 695 on bigdata-serv168.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000696 on host: bigdata-serv168.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 81) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:18 ERROR YarnScheduler: Lost executor 675 on bigdata-serv852.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000676 on host: bigdata-serv852.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:18 ERROR YarnScheduler: Lost executor 54 on bigdata-serv410.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000055 on host: bigdata-serv410.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:18 ERROR YarnScheduler: Lost executor 292 on bigdata-serv464.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000293 on host: bigdata-serv464.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:18 ERROR YarnScheduler: Lost executor 84 on bigdata-serv460.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000085 on host: bigdata-serv460.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 80) / 83]17/01/01 23:03:18 ERROR YarnScheduler: Lost executor 206 on bigdata-serv1393.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000207 on host: bigdata-serv1393.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]17/01/01 23:03:19 ERROR YarnScheduler: Lost executor 326 on bigdata-serv1758.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000327 on host: bigdata-serv1758.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 81) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:20 ERROR YarnScheduler: Lost executor 318 on bigdata-serv2167.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000319 on host: bigdata-serv2167.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:20 ERROR YarnScheduler: Lost executor 746 on bigdata-serv678.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000747 on host: bigdata-serv678.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:20 ERROR YarnScheduler: Lost executor 348 on bigdata-serv875.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000349 on host: bigdata-serv875.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 80) / 83]17/01/01 23:03:20 ERROR YarnScheduler: Lost executor 240 on bigdata-serv2074.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000241 on host: bigdata-serv2074.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 79) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:21 ERROR YarnScheduler: Lost executor 774 on bigdata-serv479.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000775 on host: bigdata-serv479.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:21 ERROR YarnScheduler: Lost executor 142 on bigdata-serv440.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000143 on host: bigdata-serv440.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:21 ERROR YarnScheduler: Lost executor 426 on bigdata-serv2104.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000427 on host: bigdata-serv2104.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 80) / 83]17/01/01 23:03:21 ERROR YarnScheduler: Lost executor 592 on bigdata-serv450.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000593 on host: bigdata-serv450.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 79) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:23 ERROR YarnScheduler: Lost executor 164 on bigdata-serv2274.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000165 on host: bigdata-serv2274.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:24 ERROR YarnScheduler: Lost executor 266 on bigdata-serv83.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000267 on host: bigdata-serv83.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:24 ERROR YarnScheduler: Lost executor 786 on bigdata-serv754.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000787 on host: bigdata-serv754.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 81) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:26 ERROR YarnScheduler: Lost executor 279 on bigdata-serv163.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000280 on host: bigdata-serv163.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:32 ERROR YarnScheduler: Lost executor 212 on bigdata-serv572.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000213 on host: bigdata-serv572.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:32 ERROR YarnScheduler: Lost executor 622 on bigdata-serv1368.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000623 on host: bigdata-serv1368.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:32 ERROR YarnScheduler: Lost executor 598 on bigdata-serv2122.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000599 on host: bigdata-serv2122.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 80) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:33 ERROR YarnScheduler: Lost executor 83 on bigdata-serv562.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000084 on host: bigdata-serv562.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:35 ERROR YarnScheduler: Lost executor 176 on bigdata-serv688.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000177 on host: bigdata-serv688.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:03:35 ERROR YarnScheduler: Lost executor 182 on bigdata-serv243.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000183 on host: bigdata-serv243.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 81) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:38 ERROR YarnScheduler: Lost executor 274 on bigdata-serv2296.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000275 on host: bigdata-serv2296.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:03:50 ERROR YarnScheduler: Lost executor 522 on bigdata-serv920.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000523 on host: bigdata-serv920.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]
[Stage 1:> (0 + 83) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:06:08 ERROR YarnScheduler: Lost executor 721 on bigdata-serv1709.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000722 on host: bigdata-serv1709.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

[Stage 1:> (0 + 82) / 83]
[Stage 1:> (0 + 83) / 83]17/01/01 23:06:14 ERROR YarnScheduler: Lost executor 126 on bigdata-serv1444.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000127 on host: bigdata-serv1444.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:06:18 ERROR YarnScheduler: Lost executor 389 on bigdata-serv602.mycorp.com: Container marked as failed: container_e50_1481285758114_1471903_01_000390 on host: bigdata-serv602.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

17/01/01 23:06:18 ERROR TaskSetManager: Task 59 in stage 1.0 failed 4 times; aborting job
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 59 in stage 1.0 failed 4 times, most recent failure: Lost task 59.3 in stage 1.0 (TID 7191, bigdata-serv602.mycorp.com): ExecutorLostFailure (executor 389 exited caused by one of the running tasks) Reason: Container marked as failed: container_e50_1481285758114_1471903_01_000390 on host: bigdata-serv602.mycorp.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1443)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1431)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1430)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1430)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:810)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:810)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:810)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1652)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1611)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1600)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1874)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1994)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1025)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.RDD.reduce(RDD.scala:1007)
at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1.apply(RDD.scala:1150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.RDD.treeAggregate(RDD.scala:1127)
at org.apache.spark.rdd.RDD$$anonfun$treeReduce$1.apply(RDD.scala:1058)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.RDD.treeReduce(RDD.scala:1036)
at fregata.spark.model.SparkTrainer.run(SparkTrainer.scala:28)
at fregata.spark.model.SparkTrainer$$anonfun$run$1.apply$mcVI$sp(SparkTrainer.scala:15)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at fregata.spark.model.SparkTrainer.run(SparkTrainer.scala:13)
at fregata.spark.model.classification.LogisticRegression$.run(LogisticRegression.scala:29)
at com.mycorp.bigdata.datamining.exe.FregataTest$.main(FregataTest.scala:21)
at com.mycorp.bigdata.datamining.exe.FregataTest.main(FregataTest.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/01/01 23:06:18 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler or it has been stopped.
at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:161)
at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:131)
at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:578)
at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:170)
at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:104)
at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:104)
at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@115a20a8 rejected from java.util.concurrent.ThreadPoolExecutor@84a2d14[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 340]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:122)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.processBatch$1(Future.scala:643)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply$mcV$sp(Future.scala:658)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:635)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:635)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at scala.concurrent.Future$InternalCallbackExecutor$Batch.run(Future.scala:634)
at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:685)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
at org.apache.spark.rpc.netty.NettyRpcEnv.org$apache$spark$rpc$netty$NettyRpcEnv$$onFailure$1(NettyRpcEnv.scala:208)
at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$2.apply(NettyRpcEnv.scala:230)
at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$2.apply(NettyRpcEnv.scala:230)
at org.apache.spark.rpc.netty.RpcOutboxMessage.onFailure(Outbox.scala:71)
at org.apache.spark.network.client.TransportResponseHandler.failOutstandingRequests(TransportResponseHandler.java:110)
at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:124)
at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:94)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:739)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:659)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:328)
at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:627)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:362)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@5141f23b rejected from java.util.concurrent.ThreadPoolExecutor@84a2d14[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 340]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:122)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.processBatch$1(Future.scala:643)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply$mcV$sp(Future.scala:658)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:635)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:635)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at scala.concurrent.Future$InternalCallbackExecutor$Batch.run(Future.scala:634)
at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:685)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
at org.apache.spark.rpc.netty.NettyRpcEnv.org$apache$spark$rpc$netty$NettyRpcEnv$$onFailure$1(NettyRpcEnv.scala:208)
at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$2.apply(NettyRpcEnv.scala:230)
at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$2.apply(NettyRpcEnv.scala:230)
at org.apache.spark.rpc.netty.RpcOutboxMessage.onFailure(Outbox.scala:71)
at org.apache.spark.network.client.TransportResponseHandler.failOutstandingRequests(TransportResponseHandler.java:110)
at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:124)
at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:94)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:739)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:659)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:328)
at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:627)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:362)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)

@takun2s
Copy link
Collaborator

takun2s commented Jan 3, 2017

可以试一下 --executor-cores 1
testData 预测完后，如果量比较大，collect也有可能有问题，有一个对RDD算AUC的。

@ace7
Copy link
Author

ace7 commented Jan 3, 2017

试过了，也不行
--num-executors 1000
--executor-cores 1
--executor-memory 12g
--driver-memory 20g

@takun2s
Copy link
Collaborator

takun2s commented Jan 3, 2017

300万条数据，原始partition个数有 7000 吗？
把partition数变少试一下？

@ace7
Copy link
Author

ace7 commented Jan 3, 2017

对，原始的partition个数是7000，repartition到1000, 500, 100等数目都不行，还是报同样的错误

@takun2s
Copy link
Collaborator

takun2s commented Jan 3, 2017

加下我QQ详细看一下吧。404667676

@ace7
Copy link
Author

ace7 commented Jan 3, 2017

把partition数目再减少后可以了，多谢 @takun2s

@ace7 ace7 closed this as completed Jan 3, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ace7 @takun2s