/
CHANGES.txt
6172 lines (4020 loc) · 229 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Hadoop YARN Change Log
Trunk - Unreleased
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
YARN-2438. yarn-env.sh cleanup (aw)
YARN-2472. yarn-daemons.sh should jsut call yarn directly (Masatake Iwasaki
via aw)
YARN-2437. start-yarn.sh/stop-yarn should give info (Varun Saxena via aw)
YARN-2796. deprecate sbin/yarn-daemon.sh (aw)
YARN-2980. Move health check script related functionality to hadoop-common
(Varun Saxena via aw)
YARN-3168. Convert site documentation from apt to markdown (Gururaj Shetty
via aw)
YARN-3199. Fair Scheduler documentation improvements (Rohit Agarwal via
aw)
YARN-2280. Resource manager web service fields are not accessible
(Krisztian Horvath via aw)
YARN-3261. rewrite resourcemanager restart doc to remove roadmap bits (Gururaj Shetty via aw)
OPTIMIZATIONS
BUG FIXES
YARN-524 TestYarnVersionInfo failing if generated properties doesn't
include an SVN URL. (stevel)
YARN-1471. The SLS simulator is not running the preemption policy
for CapacityScheduler (Carlo Curino via cdouglas)
YARN-2436. [post-HADOOP-9902] yarn application help doesn't work (aw)
YARN-2525. yarn logs command gives error on trunk (Akira AJISAKA via aw)
YARN-3002. YARN documentation needs updating post-shell rewrite (aw)
YARN-2428. LCE default banned user list should have yarn (Varun
Saxena via aw)
YARN-2355. MAX_APP_ATTEMPTS_ENV may no longer be a useful env var
for a container (Darrell Taylor via aw)
YARN-3915. scmadmin help message correction (Bibin A Chundatt via aw)
Release 2.8.0 - UNRELEASED
INCOMPATIBLE CHANGES
YARN-2336. Fair scheduler's REST API returns a missing '[' bracket JSON for
deep queue tree. (Kenji Kikushima and Akira Ajisaka via ozawa)
NEW FEATURES
YARN-3360. Add JMX metrics to TimelineDataManager (Jason Lowe via jeagles)
YARN-3345. Add non-exclusive node label API. (Wangda Tan via jianhe)
YARN-3365. Enhanced NodeManager to support using the 'tc' tool via
container-executor for outbound network traffic control. (Sidharta Seethana
via vinodkv)
YARN-1376. NM need to notify the log aggregation status to RM through
heartbeat. (Xuan Gong via junping_du)
YARN-3348. Add a 'yarn top' tool to help understand cluster usage. (Varun
Vasudev via jianhe)
YARN-3347. Improve YARN log command to get AMContainer logs as well as
running containers logs. (Xuan Gong via junping_du)
YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support
for new resource types on the NM. (Sidharta Seethana via junping_du)
YARN-3361. CapacityScheduler side changes to support non-exclusive node
labels. (Wangda Tan via jianhe)
YARN-3318. Create Initial OrderingPolicy Framework and FifoOrderingPolicy.
(Craig Welch via wangda)
YARN-3326. Support RESTful API for getLabelsToNodes. (Naganarasimha G R
via ozawa)
YARN-3354. Add node label expression in ContainerTokenIdentifier to support
RM recovery. (Wangda Tan via jianhe)
YARN-1402. Update related Web UI and CLI with exposing client API to check
log aggregation status. (Xuan Gong via junping_du)
YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler.
(Craig Welch via wangda)
YARN-3410. YARN admin should be able to remove individual application
records from RMStateStore. (Rohith Sharmaks via wangda)
YARN-3225. New parameter of CLI for decommissioning node gracefully in
RMAdmin CLI. (Devaraj K via junping_du)
YARN-3366. Enhanced NodeManager to support classifying/shaping outgoing
network bandwidth traffic originating from YARN containers (Sidharta Seethana
via vinodkv)
YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda)
YARN-2498. Respect labels in preemption policy of capacity scheduler for
inter-queue preemption. (Wangda Tan via jianhe)
YARN-2619. Added NodeManager support for disk io isolation through cgroups.
(Varun Vasudev and Wei Yan via vinodkv)
YARN-3448. Added a rolling time-to-live LevelDB timeline store implementation.
(Jonathan Eagles via zjshen)
YARN-3505. Node's Log Aggregation Report with SUCCEED should not cached in
RMApps. (Xuan Gong via junping_du)
YARN-3541. Add version info on timeline service / generic history web UI
and REST API. (Zhijie Shen via xgong)
YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from
underlying OS when configured to do so. (Varun Vasudev via vinodkv)
YARN-41. The RM should handle the graceful shutdown of the NM. (Devaraj K via
junping_du)
YARN-1012. Report NM aggregated container resource utilization in heartbeat.
(Inigo Goiri via kasha)
YARN-3116. RM notifies NM whether a container is an AM container or normal
task container. (Giovanni Matteo Fumarola via zjshen)
YARN-2003. Support for Application priority : Changes in RM and Capacity
Scheduler. (Sunil G via wangda)
YARN-3656. LowCost: A Cost-Based Placement Agent for YARN Reservations.
(Jonathan Yaniv and Ishai Menache via curino)
YARN-3852. Add docker container support to container-executor
(Abin Shahab via vvasudev)
YARN-3853. Add docker container runtime support to LinuxContainterExecutor.
(Sidharta Seethana via vvasudev)
YARN-3736. Add RMStateStore apis to store and load accepted reservations for
failover (adhoot via asuresh)
YARN-3948. Display Application Priority in RM Web UI.(Sunil G via rohithsharmaks)
YARN-3873. PendingApplications in LeafQueue should also use OrderingPolicy.
(Sunil G via wangda)
YARN-3887. Support changing Application priority during runtime. (Sunil G
via jianhe)
YARN-4023. Publish Application Priority to TimelineServer. (Sunil G
via rohithsharmaks)
YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha)
YARN-4055. Report node resource utilization in heartbeat.
(Inigo Goiri via kasha)
YARN-2923. Support configuration based NodeLabelsProvider Service in Distributed
Node Label Configuration Setup. (Naganarasimha G R)
YARN-221. NM should provide a way for AM to tell it not to aggregate logs.
(Ming Ma via xgong)
YARN-4014. Support user cli interface in for Application Priority.
(Rohith Sharma K S via jianhe)
YARN-3250. Support admin cli interface in for Application Priority.
(Rohith Sharma K S via jianhe)
YARN-3970. Add REST api support for Application Priority.
(Naganarasimha G R via vvasudev)
YARN-2884. Added a proxy service in NM to proxy the the communication
between AM and RM. (Kishore Chaliparambil via jianhe)
YARN-313. Add Admin API for supporting node resource configuration in
command line. (Inigo Goiri, Kenji Kikushima and Junping Du
via junping_du)
YARN-4034. Render cluster Max Priority in scheduler metrics in RM web
UI. (Rohith Sharma K S via jianhe)
YARN-3212. RMNode State Transition Update with DECOMMISSIONING state.
(Junping Du via wangda)
YARN-3866. AM-RM protocol changes to support container resizing. (Meng Ding
via jianhe)
YARN-1449. AM-NM protocol changes to support container resizing.
(Meng Ding & Wangda Tan via jianhe)
YARN-1645. ContainerManager implementation to support container resizing.
(Meng Ding & Wangda Tan via jianhe)
YARN-3867. ContainerImpl changes to support container resizing. (Meng Ding
via jianhe)
YARN-1643. Make ContainersMonitor support changing monitoring size of an
allocated container. (Meng Ding and Wangda Tan)
YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to
support container resizing. (Meng Ding via jianhe)
YARN-3868. Recovery support for container resizing. (Meng Ding via jianhe)
YARN-1651. CapacityScheduler side changes to support container resize.
(Wangda Tan via jianhe)
YARN-1897. CLI and core support for signal container functionality.
(Ming Ma via xgong)
YARN-261. Ability to fail AM attempts (Andrey Klochkov and
Rohith Sharma K S via jlowe)
YARN-3964. Support NodeLabelsProvider at Resource Manager side.
(Dian Fu via devaraj)
YARN-2556. Tool to measure the performance of the timeline server (Chang Li
via sjlee)
YARN-4262. Allow whitelisted users to run privileged docker containers.
(Sidharta Seethana via vvasudev)
IMPROVEMENTS
YARN-644. Basic null check is not performed on passed in arguments before
using them in ContainerManagerImpl.startContainer (Varun Saxena via bobby)
YARN-1880. Cleanup TestApplicationClientProtocolOnHA
(ozawa via harsh)
YARN-3357. Move TestFifoScheduler to FIFO package. (Rohith Sharmaks
via devaraj)
YARN-3356. Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to
track used-resources-by-label. (Wangda Tan via jianhe)
YARN-3350. YARN RackResolver spams logs with messages at info level.
(Wilfred Spiegelenburg via junping_du)
YARN-2868. FairScheduler: Metric for latency to allocate first container
for an application. (Ray Chiang via kasha)
YARN-3397. yarn rmadmin should skip -failover. (J.Andreina via kasha)
YARN-3288. Document and fix indentation in the DockerContainerExecutor code
YARN-2495. Allow admin specify labels from each NM (Distributed
configuration for node label). (Naganarasimha G R via wangda)
YARN-3258. FairScheduler: Need to add more logging to investigate
allocations. (Anubhav Dhoot via ozawa)
YARN-3428. Debug log resources to be localized for a container. (kasha)
YARN-3424. Change logs for ContainerMonitorImpl's resourse monitoring
from info to debug. (Anubhav Dhoot via ozawa)
YARN-2901. Add errors and warning metrics page to RM, NM web UI.
(Varun Vasudev via wangda)
YARN-3294. Allow dumping of Capacity Scheduler debug logs via
web UI for a fixed time period. (Varun Vasudev via xgong)
YARN-3293. Track and display capacity scheduler health metrics
in web UI. (Varun Vasudev via xgong)
YARN-3394. Enrich WebApplication proxy documentation. (Naganarasimha G R
via jianhe)
YARN-3404. Display queue name on application page. (Ryu Kobayashi via jianhe)
YARN-2696. Queue sorting in CapacityScheduler should consider node label.
(Wangda Tan via jianhe)
YARN-3451. Display attempt start time and elapsed time on the web UI.
(Rohith Sharmaks via jianhe)
YARN-3494. Expose AM resource limit and usage in CS QueueMetrics. (Rohith
Sharmaks via jianhe)
YARN-3503. Expose disk utilization percentage and bad local and log dir
counts in NM metrics. (Varun Vasudev via jianhe)
YARN-3511. Add errors and warnings page to ATS. (Varun Vasudev via xgong)
YARN-3406. Display count of running containers in the RM's Web UI.
(Ryu Kobayashi via ozawa)
YARN-3363. add localization and container launch time to ContainerMetrics
at NM to show these timing information for each active container.
(zxu via rkanter)
YARN-3396. Handle URISyntaxException in ResourceLocalizationService.
(Brahma Reddy Battula via junping_du)
YARN-3491. PublicLocalizer#addResource is too slow. (zxu via rkanter)
YARN-3523. Cleanup ResourceManagerAdministrationProtocol interface audience.
(Naganarasimha G R via junping_du)
YARN-3169. Drop YARN's overview document. (Brahma Reddy Battula via ozawa)
YARN-2784. Make POM project names consistent. (Rohith via devaraj)
YARN-20. More information for "yarn.resourcemanager.webapp.address" in
yarn-default.xml (Bartosz Ługowski vai tgraves)
YARN-3593. Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page.
(Naganarasimha G R via wangda)
YARN-3018. Unified the default value for the configuration property
yarn.scheduler.capacity.node-locality-delay in code and default xml file.
(Nijel SF via vinodkv)
YARN-2331. Distinguish shutdown during supervision vs. shutdown for
rolling upgrade. (Jason Lowe via xgong)
YARN-1050. Document the Fair Scheduler REST API.
(Kenji Kikushima and Roman Shaposhnik via kasha)
YARN-3271. FairScheduler: Move tests related to max-runnable-apps from
TestFairScheduler to TestAppRunnability. (nijel via kasha)
YARN-1912. ResourceLocalizer started without any jvm memory control.
(Masatake Iwasaki via xgong)
YARN-1287. Consolidate MockClocks.
(Sebastian Wong and Anubhav Dhoot via kasha)
YARN-3395. FairScheduler: Trim whitespaces when using username for
queuename. (Zhihai Xu via kasha)
YARN-3587. Fix the javadoc of DelegationTokenSecretManager in yarn, etc.
projects. (Gabor Liptak via junping_du)
YARN-3513. Remove unused variables in ContainersMonitorImpl and add debug
log for overall resource usage by all containers. (Naganarasimha G R via devaraj)
YARN-3613. TestContainerManagerSecurity should init and start Yarn cluster in
setup instead of individual methods. (nijel via kasha)
YARN-3579. CommonNodeLabelsManager should support NodeLabel instead of string
label name when getting node-to-label/label-to-label mappings. (Sunil G via wangda)
YARN-3521. Support return structured NodeLabel objects in REST API (Sunil G via wangda)
YARN-3362. Add node label usage in RM CapacityScheduler web UI.
(Naganarasimha G R via wangda)
YARN-3565. NodeHeartbeatRequest/RegisterNodeManagerRequest should use
NodeLabel object instead of String. (Naganarasimha G R via wangda)
YARN-3583. Support of NodeLabel object instead of plain String
in YarnClient side. (Sunil G via wangda)
YARN-3684. Changed ContainerExecutor's primary lifecycle methods to use a more
extensible mechanism of context objects. (Sidharta Seethana via vinodkv)
YARN-3594. WintuilsProcessStubExecutor.startStreamReader leaks streams.
(Lars Francke via junping_du)
YARN-3647. RMWebServices api's should use updated api from CommonNodeLabelsManager
to get NodeLabel object. (Sunil G via wangda)
YARN-3581. Deprecate -directlyAccessNodeLabelStore in RMAdminCLI.
(Naganarasimha G R via wangda)
YARN-3722. Merge multiple TestWebAppUtils into o.a.h.yarn.webapp.util.TestWebAppUtils.
(Masatake Iwasaki via devaraj)
YARN-3713. Remove duplicate function call storeContainerDiagnostics in
ContainerDiagnosticsUpdateTransition (zxu via rkanter)
YARN-3467. Expose allocatedMB, allocatedVCores, and runningContainers metrics on
running Applications in RM Web UI. (Anubhav Dhoot via kasha)
YARN-2392. Add more diags about app retry limits on AM failures. (Steve
Loughran via jianhe)
YARN-1462. AHS API and other AHS changes to handle tags for completed MR jobs. (xgong)
YARN-2716. Refactor ZKRMStateStore retry code with Apache Curator.
(Karthik Kambatla via jianhe)
YARN-3786. Document yarn class path options.
(Brahma Reddy Battula via cnauroth)
YARN-3787. Allowed generic history service to load a number of applications whose
started time is within the given range. (Xuan Gong via zjshen)
YARN-3785. Support for Resource as an argument during submitApp call in MockRM
test class. (Sunil G via xgong)
YARN-3789. Improve logs for LeafQueue#activateApplications().
(Bibin A Chundatt via devaraj)
YARN-3148. Allow CORS related headers to passthrough in WebAppProxyServlet.
(Varun Saxena via devaraj)
YARN-3834. Scrub debug logging of tokens during resource localization.
(Chris Nauroth via xgong)
YARN-3827. Migrate YARN native build to new CMake framework (Alan Burlison
via Colin P. McCabe)
YARN-3800. Reduce storage footprint for ReservationAllocation. (Anubhav Dhoot
via curino)
YARN-3069. Document missing properties in yarn-default.xml.
(Ray Chiang via aajisaka)
YARN-3844. Make hadoop-yarn-project Native code -Wall-clean (Alan Burlison
via Colin P. McCabe)
YARN-3026. Move application-specific container allocation logic from
LeafQueue to FiCaSchedulerApp. (Wangda Tan via jianhe)
YARN-3950. Add unique SHELL_ID environment variable to DistributedShell
(Robert Kanter via jlowe)
YARN-3965. Add startup timestamp to nodemanager UI (Hong Zhiguo via jlowe)
YARN-3543. ApplicationReport should be able to tell whether the Application
is AM managed or not. (Rohith Sharma K S via xgong)
YARN-4004. container-executor should print output of docker logs if the docker
container exits with non-0 exit status. (Varun Vasudev via xgong)
YARN-3961. Expose pending, running and reserved containers of a queue in REST
api and yarn top (adhoot via asuresh)
YARN-4019. Add JvmPauseMonitor to ResourceManager and NodeManager. (Robert Kanter
via junping_du)
YARN-3974. Refactor the reservation system test cases to use parameterized
base test. (subru via curino)
YARN-3966. Fix excessive loggings in CapacityScheduler. (Jian He via wangda)
YARN-4026. Refactored ContainerAllocator to accept a list of priorites
rather than a single priority. (Wangda Tan via jianhe)
YARN-4031. Add JvmPauseMonitor to ApplicationHistoryServer and
WebAppProxyServer (djp via rkanter)
YARN-4057. If ContainersMonitor is not enabled, only print
related log info one time. (Jun Gong via zxu)
YARN-1556. NPE getting application report with a null appId. (Weiwei Yang via
junping_du)
YARN-4121. Fix typos in capacity scheduler documentation.
(Kai Sasaki via vvasudev)
YARN-4086. Allow Aggregated Log readers to handle HAR files (rkanter)
YARN-4145. Make RMHATestBase abstract so its not run when running all
tests under that namespace (adhoot via rkanter)
YARN-2005. Blacklisting support for scheduling AMs. (Anubhav Dhoot via kasha)
YARN-3717. Expose app/am/queue's node-label-expression to RM web UI /
CLI / REST-API. (Naganarasimha G R via wangda)
YARN-4149. yarn logs -am should provide an option to fetch all the log files
(Varun Vasudev via xgong)
YARN-2597. MiniYARNCluster should propagate reason for AHS not starting.
(stevel)
MAPREDUCE-6478. Add an option to skip cleanupJob stage or ignore cleanup
failure during commitJob. (Junping Du via wangda)
YARN-3920. FairScheduler container reservation on a node should be
configurable to limit it to large containers (adhoot via asuresh)
HADOOP-12428. Fix inconsistency between log-level guards and statements.
(Jagadesh Kiran N and Jackie Chang via ozawa)
YARN-4095. Avoid sharing AllocatorPerContext object in LocalDirAllocator
between ShuffleHandler and LocalDirsHandlerService. (Zhihai Xu via jlowe)
YARN-4176. Resync NM nodelabels with RM periodically for distributed nodelabels.
(Bibin A Chundatt via wangda)
YARN-4215. RMNodeLabels Manager Need to verify and replace node labels for the
only modified Node Label Mappings in the request. (Naganarasimha G R via wangda)
YARN-3943. Use separate threshold configurations for disk-full detection
and disk-not-full detection. (Zhihai Xu via jlowe)
YARN-4252. Log container-executor invocation details when exit code is non-zero.
(Sidharta Seethana via vvasudev)
YARN-4258. Add support for controlling capabilities for docker containers.
(Sidharta Seethana via vvasudev)
YARN-4162. CapacityScheduler: Add resource usage by partition and queue capacity
by partition to REST API. (Naganarasimha G R via wangda)
YARN-4170. AM need to be notified with priority in AllocateResponse.
(Sunil G via jianhe)
YARN-4267. Add additional logging to container launch implementations in
container-executor. (Sidharta Seethana via vvasudev)
YARN-3985. Make ReservationSystem persist state using RMStateStore
reservation APIs. (adhoot via asuresh)
OPTIMIZATIONS
YARN-3339. TestDockerContainerExecutor should pull a single image and not
the entire centos repository. (Ravindra Kumar Naik via raviprak)
YARN-3469. ZKRMStateStore: Avoid setting watches that are not required.
(Jun Gong via kasha)
YARN-3006. Improve the error message when attempting manual failover with
auto-failover enabled. (Akira AJISAKA via wangda)
YARN-3547. FairScheduler: Apps that have no resource demand should not participate
scheduling. (Xianyin Xin via kasha)
YARN-3259. FairScheduler: Trigger fairShare updates on node events.
(Anubhav Dhoot via kasha)
YARN-2768. Avoid cloning Resource in FSAppAttempt#updateDemand.
(Hong Zhiguo via kasha)
YARN-3983. Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend
container allocation logic. (Wangda Tan via jianhe)
YARN-3635. Refactored current queue mapping implementation in CapacityScheduler
to use a generic PlacementManager framework. (Wangda Tan via jianhe)
YARN-4066. Large number of queues choke fair scheduler.
(Johan Gustavsson via kasha)
BUG FIXES
YARN-3197. Confusing log generated by CapacityScheduler. (Varun Saxena
via devaraj)
YARN-3305. Normalize AM resource request on app submission. (Rohith Sharmaks
via jianhe)
YARN-3205 FileSystemRMStateStore should disable FileSystem Cache to avoid
get a Filesystem with an old configuration. (Zhihai Xu via ozawa)
YARN-3269. Yarn.nodemanager.remote-app-log-dir could not be configured to
fully qualified path. (Xuan Gong via junping_du)
YARN-3241. FairScheduler handles "invalid" queue names inconsistently.
(Zhihai Xu via kasha)
YARN-3383. AdminService should use "warn" instead of "info" to log exception
when operation fails. (Li Lu via wangda)
YARN-3400. [JDK 8] Build Failure due to unreported exceptions in
RPCUtil (rkanter)
YARN-3412. RM tests should use MockRM where possible. (kasha)
YARN-3425. NPE from RMNodeLabelsManager.serviceStop when
NodeLabelsManager.serviceInit failed. (Bibin A Chundatt via wangda)
YARN-3415. Non-AM containers can be counted towards amResourceUsage of a
Fair Scheduler queue (Zhihai Xu via Sandy Ryza)
YARN-3435. AM container to be allocated Appattempt AM container shown as null.
(Bibin A Chundatt via xgong)
YARN-2666. TestFairScheduler.testContinuousScheduling fails Intermittently.
(Zhihai Xu via ozawa)
YARN-3429. TestAMRMTokens.testTokenExpiry fails Intermittently with
error message:Invalid AMRMToken (zxu via rkanter)
YARN-3110. Few issues in ApplicationHistory web ui. (Naganarasimha G R via xgong)
YARN-3459. Fix failiure of TestLog4jWarningErrorMetricsAppender.
(Varun Vasudev via wangda)
YARN-3266. RMContext#inactiveNodes should have NodeId as map key.
(Chengbing Liu via jianhe)
YARN-3436. Fix URIs in documantion of YARN web service REST APIs.
(Bibin A Chundatt via ozawa)
YARN-3021. YARN's delegation-token handling disallows certain trust setups
to operate properly over DistCp. (Yongjun Zhang via jianhe)
YARN-3136. Fixed a synchronization problem of
AbstractYarnScheduler#getTransferredContainers. (Sunil G via jianhe)
YARN-3495. Confusing log generated by FairScheduler.
(Brahma Reddy Battula via ozawa)
YARN-3387. Previous AM's container completed status couldn't pass to current
AM if AM and RM restarted during the same time. (sandflee via jianhe)
YARN-3444. Fix typo capabililty. (Gabor Liptak via aajisaka)
YARN-3530. ATS throws exception on trying to filter results without otherinfo.
(zhijie shen via xgong)
YARN-2740. Fix NodeLabelsManager to properly handle node label modifications
when distributed node label configuration enabled. (Naganarasimha G R via wangda)
YARN-3517. RM web ui for dumping scheduler logs should be for admins only
(Varun Vasudev via tgraves)
YARN-3533. Test: Fix launchAM in MockRM to wait for attempt to be scheduled.
(Anubhav Dhoot via jianhe)
YARN-3564. Fix TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable
fails randomly. (Jian He via wangda)
YARN-2893. AMLaucher: sporadic job failures due to EOFException in
readTokenStorageStream. (Zhihai Xu via gera)
YARN-2454. Fix compareTo of variable UNBOUNDED in o.a.h.y.util.resource.Resources.
(Xu Yang via junping_du)
YARN-1993. Cross-site scripting vulnerability in TextView.java. (Kenji Kikushima
via ozawa)
YARN-3097. Logging of resource recovery on NM restart has redundancies
(Eric Payne via jlowe)
YARN-3375. NodeHealthScriptRunner.shouldRun() check is performing 3 times for
starting NodeHealthScriptRunner. (Devaraj K via wangda)
YARN-2725. Added test cases of retrying creating znode in ZKRMStateStore.
(Tsuyoshi Ozawa via jianhe)
YARN-3552. RM Web UI shows -1 running containers for completed apps
(Rohith via jlowe)
YARN-2123. Progress bars in Web UI always at 100% due to non-US locale.
(Akira AJISAKA via xgong)
YARN-3343. Increased TestCapacitySchedulerNodeLabelUpdate#testNodeUpdate
timeout. (Rohith Sharmaks via jianhe)
YARN-3582. NPE in WebAppProxyServlet. (jian he via xgong)
YARN-3580. [JDK8] TestClientRMService.testGetLabelsToNodes fails. (Robert Kanter
via junping_du)
YARN-3577. Misspelling of threshold in log4j.properties for tests.
(Brahma Reddy Battula via aajisaka)
YARN-3584. Fixed attempt diagnostics format shown on the UI. (nijel via
jianhe)
YARN-1832. Fix wrong MockLocalizerStatus#equals implementation.
(Hong Zhiguo via aajisaka)
YARN-3572. Correct typos in WritingYarnApplications.md.
(Gabor Liptak via aajisaka)
YARN-3592. Fix typos in RMNodeLabelsManager. (Sunil G via devaraj)
YARN-3589. RM and AH web UI display DOCTYPE wrongly. (Rohith via ozawa)
YARN-3600. AM container link is broken (Naganarasimha G R via tgraves)
YARN-3604. Fixed ZKRMStateStore#removeApplication to also disable watch.
(zhihai xu via jianhe)
YARN-3473. Fix RM Web UI configuration for some properties (rchiang via rkanter)
YARN-2206. Updated document for applications REST API response examples. (Kenji
Kikushima and Brahma Reddy Battula via zjshen)
YARN-3602. TestResourceLocalizationService.testPublicResourceInitializesLocalDir
fails Intermittently due to IOException from cleanup. (zhihai xu via xgong)
YARN-3629. NodeID is always printed as "null" in node manager initialization log.
(nijel via devaraj)
YARN-2921. Fix MockRM/MockAM#waitForState sleep too long.
(Tsuyoshi Ozawa via wangda)
YARN-1519. Check in container-executor if sysconf is implemented before
using it (Radim Kolar and Eric Payne via raviprak)
YARN-2421. RM still allocates containers to an app in the FINISHING
state (Chang Li via jlowe)
YARN-3302. TestDockerContainerExecutor should run automatically if it can
detect docker in the usual place (Ravindra Kumar Naik via raviprak)
YARN-2821. Fixed a problem that DistributedShell AM may hang if restarted.
(Varun Vasudev via jianhe)
YARN-3654. ContainerLogsPage web UI should not have meta-refresh. (Xuan Gong
via jianhe)
YARN-3707. RM Web UI queue filter doesn't work. (Wangda Tan via jianhe)
YARN-3632. Ordering policy should be allowed to reorder an application when
demand changes. (Craig Welch via jianhe)
YARN-3716. Node-label-expression should be included by
ResourceRequestPBImpl.toString. (Xianyin Xin via wangda)
YARN-3751. Fixed AppInfo to check if used resources are null. (Sunil G via
zjshen)
YARN-3762. FairScheduler: CME on FSParentQueue#getQueueUserAclInfo. (kasha)
YARN-3749. We should make a copy of configuration when init MiniYARNCluster
with multiple RMs. (Chun Chen via xgong)
YARN-3766. Fixed the apps table column error of generic history web UI.
(Xuan Gong via zjshen)
YARN-3655. FairScheduler: potential livelock due to maxAMShare limitation
and container reservation. (Zhihai Xu via kasha)
YARN-3747. TestLocalDirsHandlerService should delete the created test directory logDir2.
(David Moore via devaraj)
YARN-3778. Fix Yarn resourcemanger CLI usage. (Brahma Reddy Battula via xgong)
YARN-3794. TestRMEmbeddedElector fails because of ambiguous LOG reference.
(Chengbing Liu via devaraj)
YARN-3714. AM proxy filter can not get RM webapp address from
yarn.resourcemanager.hostname.rm-id. (Masatake Iwasaki via xgong)
YARN-3617. Fix WindowsResourceCalculatorPlugin.getCpuFrequency()
returning always -1. (J.Andreina via devaraj)
YARN-3824. Fix two minor nits in member variable properties
of YarnConfiguration. (Ray Chiang via devaraj)
YARN-3835. hadoop-yarn-server-resourcemanager test package bundles
core-site.xml, yarn-site.xml (vamsee via rkanter)
YARN-3790. usedResource from rootQueue metrics may get stale data for FS
scheduler after recovering the container (Zhihai Xu via rohithsharmaks)
YARN-3826. Race condition in ResourceTrackerService leads to
wrong diagnostics messages. (Chengbing Liu via devaraj)
YARN-3745. SerializedException should also try to instantiate internal
exception with the default constructor. (Lavkesh Lahngir via devaraj)
YARN-2871. TestRMRestart#testRMRestartGetApplicationList sometime fails in trunk.
(zhihai xu via xgong)
YARN-3859. LeafQueue doesn't print user properly for application add.
(Varun Saxena via devaraj)
YARN-3860. rmadmin -transitionToActive should check the state of non-target node.
(Masatake Iwasaki via junping_du)
YARN-3695. ServerProxy (NMProxy, etc.) shouldn't retry forever for non
network exception. (Raju Bairishetti via jianhe)
YARN-3770. SerializedException should also handle java.lang.Error on
de-serialization. (Lavkesh Lahngir via jianhe)
YARN-3768. ArrayIndexOutOfBoundsException with empty environment variables.
(Zhihai Xu via gera)
YARN-3823. Fix mismatch in default values for
yarn.scheduler.maximum-allocation-vcores property. (Ray Chiang via devaraj)
YARN-3830. AbstractYarnScheduler.createReleaseCache may try to clean a null
attempt. (nijel via devaraj)
YARN-3875. FSSchedulerNode#reserveResource() doesn't print Application Id
properly in log. (Bibin A Chundatt via devaraj)
YARN-3882. AggregatedLogFormat should close aclScanner and ownerScanner
after create them. (zhihai xu via xgong)
YARN-3837. javadocs of TimelineAuthenticationFilterInitializer give wrong
prefix for auth options. (Bibin A Chundatt via devaraj)
YARN-2194. Fix bug causing CGroups functionality to fail on RHEL7.
(Wei Yan via vvasudev)
YARN-3892. Fixed NPE on RMStateStore#serviceStop when
CapacityScheduler#serviceInit fails. (Bibin A Chundatt via jianhe)
YARN-3888. ApplicationMaster link is broken in RM WebUI when appstate is NEW.
(Bibin A Chundatt via xgong)
YARN-3849. Too much of preemption activity causing continuos killing of
containers across queues. (Sunil G via wangda)
YARN-3917. getResourceCalculatorPlugin for the default should intercept all
exceptions. (gera)
YARN-3894. RM startup should fail for wrong CS xml NodeLabel capacity
configuration. (Bibin A Chundatt via wangda)
YARN-3381. Fix typo InvalidStateTransitonException.
(Brahma Reddy Battula via aajisaka)
YARN-3453. Ensure preemption logic in FairScheduler uses DominantResourceCalculator
in DRF queues to prevent unnecessary thrashing. (asuresh)
YARN-3174. Consolidate the NodeManager and NodeManagerRestart documentation
into one. (Masatake Iwasaki via ozawa)
YARN-3805. Update the documentation of Disk Checker based on YARN-90.
(Masatake Iwasaki via ozawa)
YARN-3930. FileSystemNodeLabelsStore should make sure edit log file closed when
exception is thrown. (Dian Fu via wangda)
YARN-3885. ProportionalCapacityPreemptionPolicy doesn't preempt if queue is
more than 2 level. (Ajith S via wangda)
YARN-3932. SchedulerApplicationAttempt#getResourceUsageReport and UserInfo
should based on total-used-resources. (Bibin A Chundatt via wangda)
YARN-3954. Fix TestYarnConfigurationFields#testCompareConfigurationClassAgainstXml.
(varun saxena via rohithsharmaks)
YARN-3956. Fix TestNodeManagerHardwareUtils fails on Mac (Varun Vasudev via wangda)
YARN-3941. Proportional Preemption policy should try to avoid sending duplicate
PREEMPT_CONTAINER event to scheduler. (Sunil G via wangda)
YARN-3900. Protobuf layout of yarn_security_token causes errors in other protos
that include it (adhoot via rkanter)
YARN-3845. Scheduler page does not render RGBA color combinations in IE11.
(Contributed by Mohammad Shahid Khan)
YARN-3957. FairScheduler NPE In FairSchedulerQueueInfo causing scheduler page to
return 500. (Anubhav Dhoot via kasha)
YARN-3973. Recent changes to application priority management break
reservation system from YARN-1051. (Carlo Curino via wangda)
YARN-3958. TestYarnConfigurationFields should be moved to hadoop-yarn-api
module. (Varun Saxena via aajisaka)
YARN-3846. RM Web UI queue filter is not working for sub queue.
(Mohammad Shahid Khan via jianhe)
YARN-3982. container-executor parsing of container-executor.cfg broken in
trunk and branch-2. (Varun Vasudev via xgong)
YARN-3919. NPEs' while stopping service after exception during
CommonNodeLabelsManager#start. (varun saxane via rohithsharmaks)
YARN-3963. AddNodeLabel on duplicate label addition shows success.
(Bibin A Chundatt via wangda)
YARN-3971. Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue
on nodelabel recovery. (Bibin A Chundatt via wangda)
YARN-433. When RM is catching up with node updates then it should not expire
acquired containers. (Xuan Gong via zxu)
YARN-3992. TestApplicationPriority.testApplicationPriorityAllocation fails
intermittently. (Contributed by Sunil G)
YARN-3987. Send AM container completed msg to NM once AM finishes.
(sandflee via jianhe)
YARN-4028. AppBlock page key update and diagnostics value null on recovery
(Bibin A Chundatt via xgong)
YARN-3986. getTransferredContainers in AbstractYarnScheduler should be present
in YarnScheduler interface instead. (Varun Saxena via rohithsharmaks)
YARN-4082. Container shouldn't be killed when node's label updated.
(Wangda Tan via vvasudev)
YARN-4073. Removed unused ApplicationACLsManager in ContainerManagerImpl constructor.
(Naganarasimha G R via rohithsharmaks)
YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat.
(Hong Zhiguo via wangda)
YARN-3591. Resource localization on a bad disk causes subsequent containers failure.
(Lavkesh Lahngir via vvasudev)
YARN-4106. NodeLabels for NM in distributed mode is not updated even after
clusterNodelabel addition in RM. (Bibin A Chundatt via wangda)
YARN-4115. Reduce loglevel of ContainerManagementProtocolProxy to Debug
(adhoot via rkanter)
YARN-4151. Fix findbugs errors in hadoop-yarn-server-common module.
(Meng Ding via wangda)
YARN-4078. Add getPendingResourceRequestForAttempt in YarnScheduler interface.
(Naganarasimha G R via jianhe)
YARN-4135. Improve the assertion message in MockRM while failing after waiting for the state.
(Nijel S F via rohithsharmaks)
YARN-4167. NPE on RMActiveServices#serviceStop when store is null. (Bibin A Chundatt via rohithsharmaks)
YARN-4113. RM should respect retry-interval when uses RetryPolicies.RETRY_FOREVER.
(Sunil G via wangda)
YARN-4188. Make MoveApplicationAcrossQueues abstract, newInstance static.
(Giovanni Matteo Fumarola via cdouglas)
YARN-4171. Fix findbugs warnings in YARN-1197 branch. (Wangda Tan via jianhe)
YARN-4152. NodeManager crash with NPE when LogAggregationService#stopContainer called for
absent container. (Bibin A Chundatt via rohithsharmaks)
YARN-4044. Running applications information changes such as movequeue is not published to
TimeLine server. (Sunil G via rohithsharmaks)
YARN-4204. ConcurrentModificationException in FairSchedulerQueueInfo. (adhoot)
YARN-4141. Runtime Application Priority change should not throw exception
for applications at finishing states (Sunil G via jlowe)
YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (Bibin A Chundatt via rohithsharmaks)
YARN-4235. FairScheduler PrimaryGroup does not handle empty groups returned
for a user. (Anubhav Dhoot via rohithsharmaks)
YARN-4140. RM container allocation delayed incase of app submitted to
Nodelabel partition. (Bibin A Chundatt via wangda)
YARN-4201. AMBlacklist does not work for minicluster. (Jun Gong via zxu)
YARN-4230. RM crashes with NPE when increasing container resource if there is no headroom left.
(Meng Ding via jianhe)
YARN-4017. container-executor overuses PATH_MAX. (Sidharta Seethana via vvasudev)
YARN-4253. Standardize on using PrivilegedOperationExecutor for all
invocations of container-executor in LinuxContainerExecutor. (Sidharta Seethana via vvasudev)
YARN-4255. container-executor does not clean up docker operation command files.
(Sidharta Seethana via vvasudev)
YARN-4250. NPE in AppSchedulingInfo#isRequestLabelChanged. (Brahma Reddy Battula via rohithsharmaks)
YARN-4000. RM crashes with NPE if leaf queue becomes parent queue during restart.
(Varun Saxena via jianhe)
YARN-4155. TestLogAggregationService.testLogAggregationServiceWithInterval failing
(Bibin A Chundatt via stevel)
YARN-4270. Limit application resource reservation on nodes for non-node/rack
specific requests (asuresh)
Release 2.7.2 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
YARN-2019. Retrospect on decision of making RM crashed if any exception throw
in ZKRMStateStore. (Jian He via junping_du)
IMPROVEMENTS
YARN-3170. YARN architecture document needs updating. (Brahma Reddy Battula
via ozawa)
YARN-3967. Fetch the application report from the AHS if the RM does not know about it.
(Mit Desai via xgong)
YARN-4092. Fixed UI redirection to print useful messages when both RMs are
in standby mode. (Xuan Gong via jianhe)
YARN-2801. Add documentation for node labels feature. (Wangda Tan and Naganarasimha
G R via ozawa)
OPTIMIZATIONS
BUG FIXES