-
Notifications
You must be signed in to change notification settings - Fork 8.7k
/
CHANGES.txt
4655 lines (3033 loc) · 172 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Hadoop YARN Change Log
Trunk - Unreleased
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
YARN-2438. yarn-env.sh cleanup (aw)
YARN-2472. yarn-daemons.sh should jsut call yarn directly (Masatake Iwasaki
via aw)
YARN-2437. start-yarn.sh/stop-yarn should give info (Varun Saxena via aw)
OPTIMIZATIONS
BUG FIXES
YARN-524 TestYarnVersionInfo failing if generated properties doesn't
include an SVN URL. (stevel)
YARN-1471. The SLS simulator is not running the preemption policy
for CapacityScheduler (Carlo Curino via cdouglas)
YARN-2436. [post-HADOOP-9902] yarn application help doesn't work (aw)
YARN-2525. yarn logs command gives error on trunk (Akira AJISAKA via aw)
YARN-3002. YARN documentation needs updating post-shell rewrite (aw)
YARN-2428. LCE default banned user list should have yarn (Varun
Saxena via aw)
Release 2.7.0 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
YARN-2179. [YARN-1492] Initial cache manager structure and context.
(Chris Trezzo via kasha)
YARN-2180. [YARN-1492] In-memory backing store for cache manager.
(Chris Trezzo via kasha)
YARN-2183. [YARN-1492] Cleaner service for cache manager.
(Chris Trezzo and Sangjin Lee via kasha)
YARN-2186. [YARN-1492] Node Manager uploader service for cache manager.
(Chris Trezzo and Sangjin Lee via kasha)
YARN-2236. [YARN-1492] Shared Cache uploader service on the Node
Manager. (Chris Trezzo and Sangjin Lee via kasha)
YARN-2188. [YARN-1492] Client service for cache manager.
(Chris Trezzo and Sangjin Lee via kasha)
YARN-2189. [YARN-1492] Admin service for cache manager.
(Chris Trezzo via kasha)
YARN-2765. Added leveldb-based implementation for RMStateStore. (Jason Lowe
via jianhe)
YARN-2203. [YARN-1492] Web UI for cache manager. (Chris Trezzo via kasha)
YARN-2738. [YARN-2574] Add FairReservationSystem for FairScheduler.
(Anubhav Dhoot via kasha)
YARN-2881. [YARN-2574] Implement PlanFollower for FairScheduler.
(Anubhav Dhoot via kasha)
YARN-2427. Added the API of moving apps between queues in RM web services.
(Varun Vasudev via zjshen)
YARN-2217. [YARN-1492] Shared cache client side changes.
(Chris Trezzo via kasha)
YARN-2616 [YARN-913] Add CLI client to the registry to list, view
and manipulate entries. (Akshay Radia via stevel)
IMPROVEMENTS
YARN-3005. [JDK7] Use switch statement for String instead of if-else
statement in RegistrySecurity.java (Kengo Seki via aajisaka)
YARN-2950. Change message to mandate, not suggest JS requirement on UI.
(Dustin Cote via harsh)
YARN-2891. Failed Container Executor does not provide a clear error
message. (Dustin Cote via harsh)
YARN-1979. TestDirectoryCollection fails when the umask is unusual.
(Vinod Kumar Vavilapalli and Tsuyoshi OZAWA via junping_du)
YARN-2641. Decommission nodes on -refreshNodes instead of next
NM-RM heartbeat. (Zhihai Xu via kasha)
YARN-2742. FairSchedulerConfiguration should allow extra spaces
between value and unit. (Wei Yan via kasha)
YARN-2712. TestWorkPreservingRMRestart: Augment FS tests with
queue and headroom checks. (Tsuyoshi Ozawa via kasha)
YARN-2735. diskUtilizationPercentageCutoff and diskUtilizationSpaceCutoff
are initialized twice in DirectoryCollection. (Zhihai Xu via kasha)
YARN-570. Time strings are formated in different timezone.
(Akira Ajisaka and Peng Zhang via kasha)
YARN-2780. Log aggregated resource allocation in rm-appsummary.log (Eric
Payne via jlowe)
YARN-2690. [YARN-2574] Make ReservationSystem and its dependent classes
independent of Scheduler type. (Anubhav Dhoot via kasha)
YARN-2157. Added YARN metrics in the documentaion. (Akira AJISAKA via
jianhe)
YARN-2802. ClusterMetrics to include AM launch and register delays.
(Zhihai Xu via kasha)
YARN-2375. Allow enabling/disabling timeline server per framework.
(Mit Desai via jeagles)
YARN-2604. Scheduler should consider max-allocation-* in conjunction
with the largest node. (Robert Kanter via kasha)
YARN-2679. Add metric for container launch duration. (Zhihai Xu via kasha)
YARN-2669. FairScheduler: queue names shouldn't allow periods
(Wei Yan via Sandy Ryza)
YARN-2404. Removed ApplicationAttemptState and ApplicationState class in
RMStateStore. (Tsuyoshi OZAWA via jianhe)
YARN-2165. Added the sanity check for the numeric configuration values of
the timeline service. (Vasanth kumar RJ via zjshen)
YARN-2907. SchedulerNode#toString should print all resource detail instead
of only memory. (Rohith via junping_du)
YARN-2136. Changed RMStateStore to ignore store opearations when fenced.
(Varun Saxena via jianhe)
YARN-1156. Enhance NodeManager AllocatedGB and AvailableGB metrics
for aggregation of decimal values. (Tsuyoshi OZAWA via junping_du)
YARN-2301. Improved yarn container command. (Naganarasimha G R via jianhe)
YARN-2056. Disable preemption at Queue level (Eric Payne via jlowe)
YARN-2762. Fixed RMAdminCLI to trim and check node-label related arguments
before sending to RM. (Rohith Sharmaks via jianhe)
YARN-2972. DelegationTokenRenewer thread pool never expands. (Jason Lowe
via junping_du)
YARN-2949. Add documentation for CGroups (Varun Vasudev via junping_du)
YARN-2970. NodeLabel operations in RMAdmin CLI get missing in help command.
(Varun Saxena via junping_du)
YARN-2837. Support TimeLine server to recover delegation token when
restarting. (Zhijie Shen via jianhe)
YARN-2993. Several fixes (missing acl check, error log msg ...) and some
refinement in AdminService. (Yi Liu via junping_du)
YARN-2943. Added node-labels page on RM web UI. (Wangda Tan via jianhe)
YARN-2998. Abstract out scheduler independent PlanFollower components.
(Anubhav Dhoot via kasha)
YARN-2360. Fair Scheduler: Display dynamic fair share for queues on the
scheduler page. (Ashwin Shankar and Wei Yan via kasha)
YARN-2880. Added a test to make sure node labels will be recovered
if RM restart is enabled. (Rohith Sharmaks via jianhe)
YARN-2996. Improved synchronization and I/O operations of FS- and Mem-
RMStateStore. (Yi Liu via zjshen)
YARN-2956. Added missing links in YARN documentation. (Masatake Iwasaki via
jianhe)
YARN-2957. Create unit test to automatically compare YarnConfiguration
and yarn-default.xml. (rchiang via rkanter)
YARN-2643. Don't create a new DominantResourceCalculator on every
FairScheduler.allocate call. (kasha via rkanter)
YARN-3019. Make work-preserving-recovery the default mechanism for RM
recovery. (Jian He via junping_du)
YARN-2807. Option "--forceactive" not works as described in usage of
"yarn rmadmin -transitionToActive". (Masatake Iwasaki via xgong)
YARN-2984. Metrics for container's actual memory usage. (kasha)
YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disable
node labels feature. (Wangda Tan via ozawa)
YARN-3024. LocalizerRunner should give DIE action when all resources are
localized. (Chengbing Liu via xgong)
YARN-3092. Created a common ResourceUsage class to track labeled resource
usages in Capacity Scheduler. (Wangda Tan via jianhe)
YARN-3086. Make NodeManager memory configurable in MiniYARNCluster.
(Robert Metzger via ozawa)
YARN-2897. CrossOriginFilter needs more log statements (Mit Desai via
jeagles)
YARN-3028. Better syntax for replaceLabelsOnNode in RMAdmin CLI
(Rohith Sharmaks via wangda)
YARN-2932. Add entry for "preemptable" status (enabled/disabled) to
scheduler web UI and queue initialize/refresh logging.
(Eric Payne via wangda)
YARN-3108. ApplicationHistoryServer doesn't process -D arguments (Chang Li
via jeagles)
YARN-2808. Made YARN CLI list attempt’s finished containers of a running
application. (Naganarasimha G R via zjshen)
YARN-3085. Application summary should include the application type (Rohith
via jlowe)
YARN-3022. Expose Container resource information from NodeManager for
monitoring (adhoot via ranter)
YARN-3098. Created common QueueCapacities class in Capacity Scheduler to
track capacities-by-labels of queues. (Wangda Tan via jianhe)
YARN-3075. NodeLabelsManager implementation to retrieve label to node
mapping (Varun Saxena via wangda)
YARN-1393. SLS: Add how-to-use instructions. (Wei Yan via kasha)
YARN-1723. AMRMClientAsync missing blacklist addition and removal
functionality. (Bartosz Ługowski via sseth)
YARN-3123. Made YARN CLI show a single completed container even if the app
is running. (Naganarasimha G R via zjshen)
YARN-1582. Capacity Scheduler: add a maximum-allocation-mb setting per
queue (Thomas Graves via jlowe)
YARN-1904. Ensure exceptions thrown in ClientRMService &
ApplicationHistoryClientService are uniform when application-attempt is
not found. (zjshen via acmurthy)
YARN-3144. Configuration for making delegation token failures to timeline
server not-fatal (Jonathan Eagles via jlowe)
YARN-3155. Refactor the exception handling code for TimelineClientImpl's
retryOn method (Li Lu via wangda)
YARN-3100. Made YARN authorization pluggable. (Jian He via zjshen)
YARN-2683. [YARN-913] registry config options: document and move to
core-default. (stevel)
OPTIMIZATIONS
YARN-2990. FairScheduler's delay-scheduling always waits for node-local and
rack-local delays, even for off-rack-only requests. (kasha)
BUG FIXES
YARN-3071. Remove invalid char from sample conf in doc of FairScheduler.
(Masatake Iwasaki via aajisaka)
YARN-2254. TestRMWebServicesAppsModification should run against both
CS and FS. (Zhihai Xu via kasha)
YARN-2713. "RM Home" link in NM should point to one of the RMs in an
HA setup. (kasha)
YARN-2766. Made ApplicationHistoryManager return a sorted list of apps,
attempts and containers. (Robert Kanter via zjshen)
YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state
on app recovery. (Rohith Sharmaks via jianhe)
YARN-2857. ConcurrentModificationException in ContainerLogAppender
(Mohammad Kamrul Islam via jlowe)
YARN-2816. NM fail to start with NPE during container recovery (Zhihai Xu
via jlowe)
YARN-2432. RMStateStore should process the pending events before close.
(Varun Saxena via jianhe)
YARN-1703. Fixed ResourceManager web-proxy to close connections correctly.
(Rohith Sharma via vinodkv)
YARN-2414. RM web UI: app page will crash if app is failed before any
attempt has been created (Wangda Tan via jlowe)
YARN-2870. Updated the command to run the timeline server in the document.
(Masatake Iwasaki via zjshen)
YARN-2878. Fix DockerContainerExecutor.apt.vm formatting. (Abin Shahab via
jianhe)
YARN-2865. Fixed RM to always create a new RMContext when transtions from
StandBy to Active. (Rohith Sharmaks via jianhe)
YARN-2315. FairScheduler: Set current capacity in addition to capacity.
(Zhihai Xu via kasha)
YARN-1984. LeveldbTimelineStore does not handle db exceptions properly
(Varun Saxena via jlowe)
YARN-2697. Remove useless RMAuthenticationHandler. (Haosong Huang via zjshen)
YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users.
(Jason Lowe via jianhe)
YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated
log file is corrupted (Varun Saxena via jlowe)
YARN-2894. Fixed a bug regarding application view acl when RM fails over.
(Rohith Sharmaks via jianhe)
YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to execute
any further apps. (Naganarasimha G R via kasha)
YARN-2461. Fix PROCFS_USE_SMAPS_BASED_RSS_ENABLED property in
YarnConfiguration. (rchiang via rkanter)
YARN-2869. CapacityScheduler should trim sub queue names when parse
configuration. (Wangda Tan via jianhe)
YARN-2927. [YARN-1492] InMemorySCMStore properties are inconsistent.
(Ray Chiang via kasha)
YARN-2931. PublicLocalizer may fail until directory is initialized by
LocalizeRunner. (Anubhav Dhoot via kasha)
YARN-2910. FSLeafQueue can throw ConcurrentModificationException.
(Wilfred Spiegelenburg via kasha)
YARN-2930. Fixed TestRMRestart#testRMRestartRecoveringNodeLabelManager
intermittent failure. (Wangda Tan via jianhe)
YARN-2924. Fixed RMAdminCLI to not convert node labels to lower case.
(Wangda Tan via jianhe)
YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher
(Rohith Sharmaks via jianhe)
YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in
SchedulerApplicationAttempt ctor. (devaraj)
YARN-2912 Jersey Tests failing with port in use. (varun saxena via stevel)
YARN-2356. yarn status command for non-existent application/application
attempt/container is too verbose. (Sunil G via devaraj)
YARN-2914. [YARN-1492] Potential race condition in Singleton implementation of
SharedCacheUploaderMetrics, CleanerMetrics, ClientSCMMetrics. (Varun Saxena via kasha)
YARN-2964. FSLeafQueue#assignContainer - document the reason for using both write and
read locks. (Tsuyoshi Ozawa via kasha)
YARN-2944. InMemorySCMStore can not be instantiated with ReflectionUtils#newInstance.
(Chris Trezzo via kasha)
YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie)
(Jian He via jlowe)
YARN-2675. containersKilled metrics is not updated when the container is killed
during localization. (Zhihai Xu via kasha)
YARN-2952. Fixed incorrect version check in StateStore. (Rohith Sharmaks
via jianhe)
YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha)
YARN-2977. Fixed intermittent TestNMClient failure.
(Junping Du via ozawa)
YARN-2939. Fix new findbugs warnings in hadoop-yarn-common. (Li Lu via junping_du)
YARN-2920. Changed CapacityScheduler to kill containers on nodes where
node labels are changed. (Wangda Tan via jianhe)
YARN-2340. Fixed NPE when queue is stopped during RM restart.
(Rohith Sharmaks via jianhe)
YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Li Lu
via junping_du)
YARN-2937. Fixed new findbugs warnings in hadoop-yarn-nodemanager. (Varun Saxena
via zjshen)
YARN-2946. Fixed potential deadlock in RMStateStore. (Rohith Sharmaks via
jianhe)
YARN-2988. Graph#save() may leak file descriptors. (Ted Yu via ozawa)
YARN-2992. ZKRMStateStore crashes due to session expiry. (Karthik Kambatla
via jianhe)
YARN-2938. Fixed new findbugs warnings in hadoop-yarn-resourcemanager and
hadoop-yarn-applicationhistoryservice. (Varun Saxena via zjshen)
YARN-2987. Fixed ClientRMService#getQueueInfo to check against queue and
app ACLs. (Varun Saxena via jianhe)
YARN-2991. Fixed DrainDispatcher to reuse the draining code path in
AsyncDispatcher. (Rohith Sharmaks via zjshen)
YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue.
(Rohith Sharmaks via ozawa)
YARN-2958. Made RMStateStore not update the last sequence number when updating the
delegation token. (Varun Saxena via zjshen)
YARN-2978. Fixed potential NPE while getting queue info. (Varun Saxena via
jianhe)
YARN-2230. Fixed few configs description in yarn-default.xml. (Vijay Bhat
via jianhe)
YARN-3010. Fixed findbugs warning in AbstractYarnScheduler. (Yi Liu via
jianhe)
YARN-2936. Changed YARNDelegationTokenIdentifier to set proto fields on
getProto method. (Varun Saxena via jianhe)
YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed
container statuses on heartbeat. (Chengbing Liu via jianhe)
YARN-3014. Replaces labels on a host should update all NM's labels on that
host. (Wangda Tan via jianhe)
YARN-3027. Scheduler should use totalAvailable resource from node instead of
availableResource for maxAllocation. (adhoot via rkanter)
YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler
when activating applications. (Craig Welch via jianhe)
YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs.
(Zhijie Shen via jianhe)
YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync
failure with allocation timeout. (Jian He via junping_du)
YARN-2815. Excluded transitive dependency of JLine in hadoop-yarn-server-common.
(Ferdinand Xu via zjshen)
YARN-3070. TestRMAdminCLI#testHelp fails for transitionToActive command.
(Contributed by Junping Du)
YARN-3015. yarn classpath command should support same options as hadoop
classpath. (Contributed by Varun Saxena)
YARN-2933. Capacity Scheduler preemption policy should only consider capacity
without labels temporarily. (Mayank Bansal via wangda)
YARN-2731. Fixed RegisterApplicationMasterResponsePBImpl to properly invoke
maybeInitBuilder. (Carlo Curino via wangda)
YARN-3078. LogCLIHelpers lacks of a blank space before string 'does not exist'.
(Sam Liu via ozawa)
YARN-3082. Non thread safe access to systemCredentials in NodeHeartbeatResponse
processing. (Anubhav Dhoot via ozawa)
YARN-3088. LinuxContainerExecutor.deleteAsUser can throw NPE if native
executor returns an error (Eric Payne via jlowe)
YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService
might lead NM to crash. (Varun Saxena via jianhe)
YARN-3103. AMRMClientImpl does not update AMRM token properly. (Jason Lowe
via jianhe)
YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource.
(Zhihai Xu via wangda)
YARN-3029. FSDownload.unpack() uses local locale for FS case conversion, may not
work everywhere. (Varun Saxena via ozawa)
YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage
to track used-resources-by-label.(Wangda Tan via jianhe)
YARN-3077. Fixed RM to create zk root path recursively. (Chun Chen via jianhe)
YARN-3113. Release audit warning for Sorting icons.psd. (stevel via kihwal)
YARN-3056. Add verification for containerLaunchDuration
in TestNodeManagerMetrics. (zhihai xu via xgong)
YARN-2543. Made resource usage be published to the timeline server too.
(Naganarasimha G R via zjshen)
YARN-3058. Fix error message of tokens' activation delay configuration.
(Yi Liu via ozawa)
YARN-3101. In Fair Scheduler, fix canceling of reservations for exceeding
max share (Anubhav Dhoot via Sandy Ryza)
YARN-3149. Fix typo in message for invalid application id.
(Bibin A Chundatt via xgong)
YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler
ParentQueue#getQueueUserAclInfo. (Tsuyoshi OZAWA via jianhe)
YARN-1537. Fix race condition in
TestLocalResourcesTrackerImpl.testLocalResourceCache. (xgong via acmurthy)
YARN-2694. Ensure only single node label specified in ResourceRequest.
(Wangda Tan via jianhe)
YARN-3089. LinuxContainerExecutor does not handle file arguments to
deleteAsUser (Eric Payne via jlowe)
YARN-3143. RM Apps REST API can return NPE or entries missing id and other
fields (jlowe)
YARN-3094. Reset timer for liveness monitors after RM recovery. (Jun Gong
via jianhe)
YARN-2971. RM uses conf instead of token service address to renew timeline
delegation tokens (jeagles)
YARN-3090. DeletionService can silently ignore deletion task failures
(Varun Saxena via jlowe)
YARN-2809. Implement workaround for linux kernel panic when removing
cgroup (Nathan Roberts via jlowe)
YARN-2246. Made the proxy tracking URL always be
http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. (Devaraj
K via zjshen)
YARN-3160. Fix non-atomic operation on nodeUpdateQueue in RMNodeImpl.
(Chengbing Liu via junping_du)
YARN-3074. Nodemanager dies when localizer runner tries to write to a full
disk (Varun Saxena via jlowe)
Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES
NEW FEATURES
YARN-1964. Create Docker analog of the LinuxContainerExecutor in YARN. (Abin
Shahab via raviprak)
YARN-2131. Add a way to format the RMStateStore. (Robert Kanter via kasha)
YARN-1367. Changed NM to not kill containers on NM resync if RM work-preserving
restart is enabled. (Anubhav Dhoot via jianhe)
YARN-1366. Changed AMRMClient to re-register with RM and send outstanding requests
back to RM on work-preserving RM restart. (Rohith via jianhe)
YARN-2181. Added preemption info to logs and RM web UI. (Wangda Tan via
jianhe)
YARN-1354. Recover applications upon nodemanager restart. (Jason Lowe via
junping_du)
YARN-1337. Recover containers upon nodemanager restart. (Jason Lowe via
junping_du)
YARN-2277. Added cross-origin support for the timeline server web services.
(Jonathan Eagles via zjshen)
YARN-2378. Added support for moving applications across queues in
CapacityScheduler. (Subramaniam Venkatraman Krishnan via jianhe)
YARN-2411. Support simple user and group mappings to queues. (Ram Venkatesh
via jianhe)
YARN-2174. Enable HTTPs for the writer REST API of TimelineServer.
(Zhijie Shen via jianhe)
YARN-2393. FairScheduler: Add the notion of steady fair share.
(Wei Yan via kasha)
YARN-2395. FairScheduler: Preemption timeout should be configurable per
queue. (Wei Yan via kasha)
YARN-2394. FairScheduler: Configure fairSharePreemptionThreshold per queue.
(Wei Yan via kasha)
YARN-415. Capture aggregate memory allocation at the app-level for chargeback.
(Eric Payne & Andrey Klochkov via jianhe)
YARN-2440. Enabled Nodemanagers to limit the aggregate cpu usage across all
containers to a preconfigured limit. (Varun Vasudev via vinodkv)
YARN-2033. Merging generic-history into the Timeline Store
(Zhijie Shen via junping_du)
YARN-611. Added an API to let apps specify an interval beyond which AM
failures should be ignored towards counting max-attempts. (Xuan Gong via
vinodkv)
YARN-2531. Added a configuration for admins to be able to override app-configs
and enforce/not-enforce strict control of per-container cpu usage. (Varun
Vasudev via vinodkv)
YARN-1250. Generic history service should support application-acls. (Zhijie Shen
via junping_du)
YARN-2569. Added the log handling APIs for the long running services. (Xuan
Gong via zjshen)
YARN-2102. Added the concept of a Timeline Domain to handle read/write ACLs
on Timeline service event data. (Zhijie Shen via vinodkv)
YARN-2581. Passed LogAggregationContext to NM via ContainerTokenIdentifier.
(Xuan Gong via zjshen)
YARN-1063. Augmented Hadoop common winutils to have the ability to create
containers as domain users. (Remus Rusanu via vinodkv)
YARN-1972. Added a secure container-executor for Windows. (Remus Rusanu via
vinodkv)
YARN-2613. Support retry in NMClient for rolling-upgrades. (Jian He via
junping_du)
YARN-2446. Augmented Timeline service APIs to start taking in domains as a
parameter while posting entities and events. (Zhijie Shen via vinodkv)
YARN-2468. Enhanced NodeManager to support log handling APIs (YARN-2569) for
use by long running services. (Xuan Gong via vinodkv)
YARN-1051. Add a system for creating reservations of cluster capacity.
(see breakdown below)
YARN-913. Add a way to register long-lived services in a YARN cluster.
(stevel)
YARN-2493. Added user-APIs for using node-labels. (Wangda Tan via vinodkv)
YARN-2544. Added admin-API objects for using node-labels. (Wangda Tan via
vinodkv)
YARN-2494. Added NodeLabels Manager internal API and implementation. (Wangda
Tan via vinodkv)
YARN-2501. Enhanced AMRMClient library to support requests against node
labels. (Wangda Tan via vinodkv)
YARN-2656. Made RM web services authentication filter support proxy user.
(Varun Vasudev and Zhijie Shen via zjshen)
YARN-2496. Enhanced Capacity Scheduler to have basic support for allocating
resources based on node-labels. (Wangda Tan via vinodkv)
YARN-2500. Enhaced ResourceManager to support schedulers allocating resources
based on node-labels. (Wangda Tan via vinodkv)
YARN-2504. Enhanced RM Admin CLI to support management of node-labels.
(Wangda Tan via vinodkv)
YARN-2198. Remove the need to run NodeManager as privileged account for
Windows Secure Container Executor. (Remus Rusanu via jianhe)
YARN-2647. Added a queue CLI for getting queue information. (Sunil Govind via
vinodkv)
YARN-2632. Document NM Restart feature. (Junping Du and Vinod Kumar
Vavilapalli via jlowe)
YARN-2505. Supported get/add/remove/change labels in RM REST API. (Craig Welch
via zjshen)
YARN-2811. In Fair Scheduler, reservation fulfillments shouldn't ignore max
share (Siqi Li via Sandy Ryza)
IMPROVEMENTS
YARN-2197. Add a link to YARN CHANGES.txt in the left side of doc
(Akira AJISAKA via aw)
YARN-1918. Typo in description and error message for
'yarn.resourcemanager.cluster-id' (Anandha L Ranganathan via aw)
YARN-2242. Improve exception information on AM launch crashes. (Li Lu
via junping_du)
YARN-2274. FairScheduler: Add debug information about cluster capacity,
availability and reservations. (kasha)
YARN-2228. Augmented TimelineServer to load pseudo authentication filter when
authentication = simple. (Zhijie Shen via vinodkv)
YARN-1341. Recover NMTokens upon nodemanager restart. (Jason Lowe via
junping_du)
YARN-2208. AMRMTokenManager need to have a way to roll over AMRMToken. (xgong)
YARN-2323. FairShareComparator creates too many Resource objects (Hong Zhiguo
via Sandy Ryza)
YARN-2045. Data persisted in NM should be versioned (Junping Du via jlowe)
YARN-2013. The diagnostics is always the ExitCodeException stack when the container
crashes. (Tsuyoshi OZAWA via junping_du)
YARN-2295. Refactored DistributedShell to use public APIs of protocol records.
(Li Lu via jianhe)
YARN-1342. Recover container tokens upon nodemanager restart. (Jason Lowe via
devaraj)
YARN-2214. FairScheduler: preemptContainerPreCheck() in FSParentQueue delays
convergence towards fairness. (Ashwin Shankar via kasha)
YARN-2211. Persist AMRMToken master key in RMStateStore for RM recovery.
(Xuan Gong via jianhe)
YARN-2328. FairScheduler: Verify update and continuous scheduling threads are
stopped when the scheduler is stopped. (kasha)
YARN-2347. Consolidated RMStateVersion and NMDBSchemaVersion into Version in
yarn-server-common. (Junping Du via zjshen)
YARN-1994. Expose YARN/MR endpoints on multiple interfaces. (Craig Welch,
Milan Potocnik, Arpit Agarwal via xgong)
YARN-2343. Improve NMToken expire exception message. (Li Lu via jianhe)
YARN-2370. Fix comment in o.a.h.y.server.resourcemanager.schedulerAppSchedulingInfo
(Wenwu Peng via junping_du)
YARN-2298. Move TimelineClient to yarn-common project (Zhijie Shen via
junping_du)
YARN-2288. Made persisted data in LevelDB timeline store be versioned. (Junping Du
via zjshen)
YARN-2352. FairScheduler: Collect metrics on duration of critical methods that
affect performance. (kasha)
YARN-2212. ApplicationMaster needs to find a way to update the AMRMToken
periodically. (xgong)
YARN-2026. Fair scheduler: Consider only active queues for computing fairshare.
(Ashwin Shankar via kasha)
YARN-1954. Added waitFor to AMRMClient(Async). (Tsuyoshi Ozawa via zjshen)
YARN-2302. Refactor TimelineWebServices. (Zhijie Shen via junping_du)
YARN-2337. ResourceManager sets ClientRMService in RMContext multiple times.
(Zhihai Xu via kasha)
YARN-2138. Cleaned up notifyDone* APIs in RMStateStore. (Varun Saxena via
jianhe)
YARN-2373. Changed WebAppUtils to use Configuration#getPassword for
accessing SSL passwords. (Larry McCay via jianhe)
YARN-2317. Updated the document about how to write YARN applications. (Li Lu via
zjshen)
YARN-2399. FairScheduler: Merge AppSchedulable and FSSchedulerApp into
FSAppAttempt. (kasha)
YARN-1370. Fair scheduler to re-populate container allocation state.
(Anubhav Dhoot via kasha)
YARN-2389. Added functionality for schedulers to kill all applications in a
queue. (Subramaniam Venkatraman Krishnan via jianhe)
YARN-1326. RM should log using RMStore at startup time.
(Tsuyoshi Ozawa via kasha)
YARN-2182. Updated ContainerId#toString() to append RM Epoch number.
(Tsuyoshi OZAWA via jianhe)
YARN-2406. Move RM recovery related proto to
yarn_server_resourcemanager_recovery.proto. (Tsuyoshi Ozawa via jianhe)
YARN-1506. Changed RMNode/SchedulerNode to update resource with event
notification. (Junping Du via jianhe)
YARN-2509. Enable Cross Origin Filter for timeline server only and not all
Yarn servers (Mit Desai via jeagles)
YARN-2511. Allowed all origins by default when CrossOriginFilter is
enabled. (Jonathan Eagles via zjshen)
YARN-2508. Cross Origin configuration parameters prefix are not honored
(Mit Desai via jeagles)
YARN-2512. Allowed pattern matching for origins in CrossOriginFilter.
(Jonathan Eagles via zjshen)
YARN-2507. Documented CrossOriginFilter configurations for the timeline
server. (Jonathan Eagles via zjshen)
YARN-2515. Updated ConverterUtils#toContainerId to parse epoch.
(Tsuyoshi OZAWA via jianhe)
YARN-2448. Changed ApplicationMasterProtocol to expose RM-recognized resource
types to the AMs. (Varun Vasudev via vinodkv)
YARN-2538. Added logs when RM sends roll-overed AMRMToken to AM. (Xuan Gong
via zjshen)
YARN-2229. Changed the integer field of ContainerId to be long type.
(Tsuyoshi OZAWA via jianhe)
YARN-2547. Cross Origin Filter throws UnsupportedOperationException upon
destroy (Mit Desai via jeagles)
YARN-2557. Add a parameter "attempt_Failures_Validity_Interval" into
DistributedShell (xgong)
YARN-2001. Added a time threshold for RM to wait before starting container
allocations after restart/failover. (Jian He via vinodkv)
YARN-1372. Ensure all completed containers are reported to the AMs across
RM restart. (Anubhav Dhoot via jianhe)
YARN-2539. FairScheduler: Set the default value for maxAMShare to 0.5.
(Wei Yan via kasha)
YARN-1959. Fix headroom calculation in FairScheduler.
(Anubhav Dhoot via kasha)
YARN-2577. Clarify ACL delimiter and how to configure ACL groups only
(Miklos Christine via aw)
YARN-2372. There are Chinese Characters in the FairScheduler's document
(Fengdong Yu via aw)
YARN-668. Changed NMTokenIdentifier/AMRMTokenIdentifier/ContainerTokenIdentifier
to use protobuf object as the payload. (Junping Du via jianhe)
YARN-1769. CapacityScheduler: Improve reservations (Thomas Graves via
jlowe)
YARN-2627. Added the info logs of attemptFailuresValidityInterval and number
of previous failed attempts. (Xuan Gong via zjshen)
YARN-2562. Changed ContainerId#toString() to be more readable. (Tsuyoshi
OZAWA via jianhe)
YARN-2615. Changed ClientToAMTokenIdentifier/RM(Timeline)DelegationTokenIdentifier
to use protobuf as payload. (Junping Du via jianhe)
YARN-2629. Made the distributed shell use the domain-based timeline ACLs.
(zjshen)
YARN-2583. Modified AggregatedLogDeletionService to be able to delete rolling
aggregated logs. (Xuan Gong via zjshen)
YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to
use ContainerId#getContainerId instead. (Tsuyoshi OZAWA via jianhe)
YARN-2621. Simplify the output when the user doesn't have the access for
getDomain(s). (Zhijie Shen via jianhe)
YARN-1879. Marked Idempotent/AtMostOnce annotations to ApplicationMasterProtocol
for RM fail over. (Tsuyoshi OZAWA via jianhe)
YARN-2676. Enhanced Timeline auth-filter to support proxy users. (Zhijie Shen
via vinodkv)
YARN-2673. Made timeline client put APIs retry if ConnectException happens.
(Li Lu via zjshen)
YARN-2582. Fixed Log CLI and Web UI for showing aggregated logs of LRS. (Xuan
Gong via zjshen)
YARN-90. NodeManager should identify failed disks becoming good again
(Varun Vasudev via jlowe)
YARN-2709. Made timeline client getDelegationToken API retry if ConnectException
happens. (Li Lu via zjshen)
YARN-2682. Updated WindowsSecureContainerExecutor to not use
DefaultContainerExecutor#getFirstApplicationDir and use getWorkingDir()
instead. (Zhihai Xu via jianhe)
YARN-2209. Replaced AM resync/shutdown command with corresponding exceptions and
made related MR changes. (Jian He via zjshen)
YARN-2703. Added logUploadedTime into LogValue for better display. (Xuan Gong
via zjshen)
YARN-2704. Changed ResourceManager to optionally obtain tokens itself for the
sake of localization and log-aggregation for long-running services. (Jian He
via vinodkv)
YARN-2502. Changed DistributedShell to support node labels. (Wangda Tan via
jianhe)
YARN-2760. Remove 'experimental' from FairScheduler docs. (Harsh J via kasha)
YARN-2503. Added node lablels in web UI. (Wangda Tan via jianhe)
YARN-2779. Fixed ResourceManager to not require delegation tokens for
communicating with Timeline Service. (Zhijie Shen via vinodkv)
YARN-2778. Moved node-lables' reports to the yarn nodes CLI from the admin
CLI. (Wangda Tan via vinodkv)
YARN-2770. Added functionality to renew/cancel TimeLineDelegationToken.
(Zhijie Shen via jianhe)
YARN-2818. Removed the now unnecessary user entity injection from Timeline
service given we now have domains. (Zhijie Shen via vinodkv)
YARN-2635. TestRM, TestRMRestart, TestClientToAMTokens should run
with both CS and FS. (Wei Yan and kasha via kasha)
OPTIMIZATIONS
BUG FIXES
YARN-2251. Avoid negative elapsed time in JHS/MRAM web UI and services.
(Zhijie Shen via junping_du)
YARN-2088. Fixed a bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder.
(Binglin Chang via jianhe)
YARN-2260. Fixed ResourceManager's RMNode to correctly remember containers
when nodes resync during work-preserving RM restart. (Jian He via vinodkv)
YARN-2264. Fixed a race condition in DrainDispatcher which may cause random
test failures. (Li Lu via jianhe)
YARN-2219. Changed ResourceManager to avoid AMs and NMs getting exceptions
after RM recovery but before scheduler learns about apps and app-attempts.
(Jian He via vinodkv)
YARN-2244. FairScheduler missing handling of containers for unknown
application attempts. (Anubhav Dhoot via kasha)
YARN-2321. NodeManager web UI can incorrectly report Pmem enforcement
(Leitao Guo via jlowe)
YARN-2273. NPE in ContinuousScheduling thread when we lose a node.
(Wei Yan via kasha)
YARN-2313. Livelock can occur in FairScheduler when there are lots of
running apps (Tsuyoshi Ozawa via Sandy Ryza)
YARN-2147. client lacks delegation token exception details when
application submit fails (Chen He via jlowe)
YARN-1796. container-executor shouldn't require o-r permissions (atm)
YARN-2354. DistributedShell may allocate more containers than client
specified after AM restarts. (Li Lu via jianhe)
YARN-2051. Fix bug in PBimpls and add more unit tests with reflection.
(Binglin Chang via junping_du)
YARN-2374. Fixed TestDistributedShell#testDSShell failure due to hostname
dismatch. (Varun Vasudev via jianhe)
YARN-2359. Application hangs when it fails to launch AM container.
(Zhihai Xu via kasha)
YARN-2388. Fixed TestTimelineWebServices failure due to HADOOP-10791. (zjshen)
YARN-2008. Fixed CapacityScheduler to calculate headroom based on max available
capacity instead of configured max capacity. (Craig Welch via jianhe)
YARN-2400. Fixed TestAMRestart fails intermittently. (Jian He via xgong)
YARN-2361. RMAppAttempt state machine entries for KILLED state has duplicate
event entries. (Zhihai Xu via kasha)
YARN-2070. Made DistributedShell publish the short user name to the timeline
server. (Robert Kanter via zjshen)
YARN-2397. Avoided loading two authentication filters for RM and TS web
interfaces. (Varun Vasudev via zjshen)
YARN-2409. RM ActiveToStandBy transition missing stoping previous rmDispatcher.
(Rohith via jianhe)
YARN-2249. Avoided AM release requests being lost on work preserving RM
restart. (Jian He via zjshen)
YARN-2034. Description for yarn.nodemanager.localizer.cache.target-size-mb
is incorrect (Chen He via jlowe)