/
CHANGES.txt
7762 lines (5080 loc) · 287 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Hadoop MapReduce Change Log
Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES
NEW FEATURES
MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving
RM-restart. Contributed by Rohith
MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server.
(Robert Kanter via zjshen)
IMPROVEMENTS
MAPREDUCE-5971. Move the default options for distcp -p to
DistCpOptionSwitch. (clamb via wang)
MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with
compatible/incompatible changes (Junping Du via jlowe)
MAPREDUCE-6019. MapReduce changes for exposing YARN/MR endpoints on multiple
interfaces. (Craig Welch, Milan Potocnik, Arpit Agarwal via xgong)
MAPREDUCE-883. harchive: Document how to unarchive (Akira AJISAKA and
Koji Noguchi via aw)
MAPREDUCE-4791. Javadoc for KeyValueTextInputFormat should include default
separator and how to change it (Akira AJISAKA via aw)
MAPREDUCE-5906. Inconsistent configuration in property
"mapreduce.reduce.shuffle.input.buffer.percent" (Akira AJISAKA via aw)
MAPREDUCE-5974. Allow specifying multiple MapOutputCollectors with
fallback. (Todd Lipcon via kasha)
MAPREDUCE-5130. Add missing job config options to mapred-default.xml
(Ray Chiang via Sandy Ryza)
MAPREDUCE-5891. Improved shuffle error handling across NM restarts
(Junping Du via jlowe)
MAPREDUCE-5279. Made MR headroom calculation honor cpu dimension when YARN
scheduler resource type is memory plus cpu. (Peng Zhang and Varun Vasudev
via zjshen)
MAPREDUCE-6072. Remove INSTALL document (Akira AJISAKA via aw)
MAPREDUCE-5970. Provide a boolean switch to enable MR-AM profiling (Gera
Shegalov via jlowe)
MAPREDUCE-6018. Added an MR specific config to enable emitting job history
data to the timeline server. (Robert Kanter via zjshen)
MAPREDUCE-6052. Supported overriding the default container-log4j.properties
file per job. (Junping Du via zjshen)
OPTIMIZATIONS
BUG FIXES
MAPREDUCE-5866. TestFixedLengthInputFormat fails in windows.
(Varun Vasudev via cnauroth)
MAPREDUCE-5956. Made MR AM not use maxAttempts to determine if the current
attempt is the last retry. (Wangda Tan via zjshen)
MAPREDUCE-5957. AM throws ClassNotFoundException with job classloader
enabled if custom output format/committer is used (Sangjin Lee via jlowe)
MAPREDUCE-5756. CombineFileInputFormat.getSplits() including directories
in its results (Jason Dere via jlowe)
MAPREDUCE-6014. New task status field in task attempts table can lead to
an empty web page (Mit Desai via jlowe)
MAPREDUCE-6021. MR AM should have working directory in LD_LIBRARY_PATH
(jlowe)
MAPREDUCE-5944. Remove MRv1 commands from CommandsManual.apt.vm
(Akira AJISAKA via aw)
MAPREDUCE-5943. Separate mapred commands from CommandManual.apt.vm
(Akira AJISAKA via aw)
MAPREDUCE-5363. Fix doc and spelling for TaskCompletionEvent#getTaskStatus
and getStatus (Akira AJISAKA via aw)
MAPREDUCE-6010. HistoryServerFileSystemStateStore fails to update tokens
(jlowe)
MAPREDUCE-5595. Typo in MergeManagerImpl.java (Akira AJISAKA via aw)
MAPREDUCE-5597. Missing alternatives in javadocs for deprecated constructors
in mapreduce.Job (Akira AJISAKA via aw)
MAPREDUCE-5950. incorrect description in distcp2 document (Akira AJISAKA
via aw)
MAPREDUCE-5998. CompositeInputFormat javadoc is broken (Akira AJISAKA via
aw)
MAPREDUCE-5999. Fix dead link in InputFormat javadoc (Akira AJISAKA via aw)
MAPREDUCE-5878. some standard JDK APIs are not part of system classes
defaults (Sangjin Lee via jlowe)
MAPREDUCE-6032. Made MR jobs write job history files on the default FS when
the current context's FS is different. (Benjamin Zhitomirsky via zjshen)
MAPREDUCE-6024. Shortened the time when Fetcher is stuck in retrying before
concluding the failure by configuration. (Yunjiong Zhao via zjshen)
MAPREDUCE-6036. TestJobEndNotifier fails intermittently in branch-2 (chang
li via jlowe)
MAPREDUCE-6012. DBInputSplit creates invalid ranges on Oracle.
(Wei Yan via kasha)
MAPREDUCE-6044. Fully qualified intermediate done dir path breaks per-user dir
creation on Windows. (zjshen)
MAPREDUCE-5885. build/test/test.mapred.spill causes release audit warnings
(Chen He via jlowe)
BREAKDOWN OF HDFS-6134 AND HADOOP-10150 SUBTASKS AND RELATED JIRAS
MAPREDUCE-5890. Support for encrypting Intermediate
data and spills in local filesystem. (asuresh via tucu)
MAPREDUCE-6007. Add support to distcp to preserve raw.* namespace
extended attributes. (clamb)
MAPREDUCE-6041. Fix TestOptionsParser. (clamb)
--
MAPREDUCE-6051. Fix typos in log messages. (Ray Chiang via cdouglas)
MAPREDUCE-5931. Validate SleepJob command line parameters (Gera Shegalov
via jlowe)
MAPREDUCE-6063. Correct spill size calculation for spills wrapping the
circular buffer. (zhihai xu via cdouglas)
MAPREDUCE-6071. JobImpl#makeUberDecision doesn't log that Uber mode is
disabled because of too much CPUs (Tsuyoshi OZAWA via jlowe)
MAPREDUCE-6075. HistoryServerFileSystemStateStore can create zero-length
files (jlowe)
MAPREDUCE-6070. yarn.app.am.resource.mb/cpu-vcores affects uber mode but
is not documented (Tsuyoshi OZAWA via jlowe)
MAPREDUCE-6090. mapred hsadmin getGroups fails to connect in some cases
(Robert Kanter via jlowe)
MAPREDUCE-6086. mapreduce.job.credentials.binary should allow all URIs.
(Zhihai Xu via kasha)
MAPREDUCE-6091. YARNRunner.getJobStatus() fails with
ApplicationNotFoundException if the job rolled off the RM view (Sangjin
Lee via jlowe)
MAPREDUCE-6095. Enable DistributedCache for uber-mode Jobs (Gera Shegalov
via jlowe)
MAPREDUCE-6104. TestJobHistoryParsing.testPartialJob fails in branch-2
(Mit Desai via jlowe)
MAPREDUCE-6109. Fix minor typo in distcp -p usage text (Charles Lamb
via aw)
MAPREDUCE-6093. minor distcp doc edits (Charles Lamb via aw)
MAPREDUCE-5831. Make MR client ignore unknown counters received from AM.
(Junping Du via zjshen)
MAPREDUCE-6073. Description of mapreduce.job.speculative.slowtaskthreshold
in mapred-default should be moved into description tags (Tsuyoshi OZAWA
via aw)
MAPREDUCE-5796. Use current version of the archive name in
DistributedCacheDeploy document (Akira AJISAKA via aw)
MAPREDUCE-5945. Update the description of GenericOptionsParser -jt
option (Akira AJISAKA via aw)
MAPREDUCE-6087. Fixed wrong config name of
MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS. Contributed by
Akira AJISAKA. (Akira AJISAKA via jianhe)
MAPREDUCE-6094. TestMRCJCFileInputFormat.testAddInputPath() fails on trunk
(Akira AJISAKA via jlowe)
MAPREDUCE-6029. TestCommitterEventHandler fails in trunk (Mit Desai via
jlowe)
MAPREDUCE-6122. TestLineRecordReader may fail due to test data files checked
out of git with incorrect line endings. (cnauroth)
MAPREDUCE-6123. TestCombineFileInputFormat incorrectly starts 2
MiniDFSCluster instances. (cnauroth)
MAPREDUCE-6125. TestContainerLauncherImpl sometimes fails (Mit Desai via
jlowe)
MAPREDUCE-6115. TestPipeApplication#testSubmitter fails in trunk (Binglin
Chang via jlowe)
MAPREDUCE-5873. Shuffle bandwidth computation includes time spent waiting
for maps (Siqi Li via jlowe)
MAPREDUCE-5542. Killing a job just as it finishes can generate an NPE in
client (Rohith via jlowe)
MAPREDUCE-6126. Fixed Rumen JobBuilder to ignore NormalizedResourceEvent.
(Junping Du via jianhe)
MAPREDUCE-6142. Fixed test failures in TestJobHistoryEventHandler and
TestMRTimelineEventHandling. (Zhijie Shen via vinodkv)
MAPREDUCE-6022. map_input_file is missing from streaming job environment.
(jlowe via kihwal)
MAPREDUCE-6048. Fixed TestJavaSerialization failure. (Varun Vasudev via
jianhe)
MAPREDUCE-5960. JobSubmitter's check whether job.jar is local is incorrect
with no authority in job jar path. (Gera Shegalov via jlowe)
MAPREDUCE-5958. Wrong reduce task progress if map output is compressed
(Emilio Coppa and jlowe via kihwal)
MAPREDUCE-6156. Fetcher - connect() doesn't handle connection refused
correctly (Junping Du via jlowe)
Release 2.5.2 - 2014-11-10
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
Release 2.5.1 - 2014-09-05
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
MAPREDUCE-6033. Updated access check for displaying job information
(Yu Gao via Eric Yang)
Release 2.5.0 - 2014-08-11
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
MAPREDUCE-5671. NaN can be created by client and assign to Progress (Chen
He via jeagles)
MAPREDUCE-5665. Add audience annotations to MiniMRYarnCluster and
MiniMRCluster. (Anubhav Dhoot via kasha)
MAPREDUCE-5765. Update hadoop-pipes examples README (Mit Desai via jeagles)
MAPREDUCE-5713. InputFormat and JobConf JavaDoc Fixes (Chen He via jeagles)
MAPREDUCE-5456. TestFetcher.testCopyFromHostExtraBytes is missing (Jason
Lowe via jeagles)
MAPREDUCE-5804. TestMRJobsWithProfiler#testProfiler timesout (Mit Desai
via kihwal)
MAPREDUCE-5825. Provide diagnostics for reducers killed during ramp down
(Gera Shegalov via jlowe)
MAPREDUCE-5836. Fix typo in RandomTextWriter (Akira AJISAKA via jeagles)
MAPREDUCE-5852. Prepare MapReduce codebase for JUnit 4.11. (cnauroth)
MAPREDUCE-5639. Port DistCp2 document to trunk (Akira AJISAKA via jeagles)
MAPREDUCE-5812. Make job context available to
OutputCommitter.isRecoverySupported() (Mohammad Kamrul Islam via jlowe)
MAPREDUCE-5638. Port Hadoop Archives document to trunk (Akira AJISAKA via
jeagles)
MAPREDUCE-5402. In DynamicInputFormat, change MAX_CHUNKS_TOLERABLE,
MAX_CHUNKS_IDEAL, MIN_RECORDS_PER_CHUNK and SPLIT_RATIO to be configurable.
(Tsuyoshi OZAWA via szetszwo)
MAPREDUCE-5637. Convert Hadoop Streaming document to APT (Akira AJISAKA via
jeagles)
MAPREDUCE-5636. Convert MapReduce Tutorial document to APT (Akira AJISAKA
via jeagles)
MAPREDUCE-5774. Job overview in History UI should list reducer phases in
chronological order. (Gera Shegalov via kasha)
MAPREDUCE-5652. NM Recovery. ShuffleHandler should handle NM restarts.
(Jason Lowe via kasha)
MAPREDUCE-5861. finishedSubMaps field in LocalContainerLauncher does not
need to be volatile. (Tsuyoshi OZAWA via junping_du)
MAPREDUCE-5809. Enhance distcp to support preserving HDFS ACLs. (cnauroth)
MAPREDUCE-5899. Support incremental data copy in DistCp. (jing9)
MAPREDUCE-5886. Allow wordcount example job to accept multiple input paths.
(cnauroth)
MAPREDUCE-5834. Increased test-timeouts in TestGridMixClasses to avoid
occassional failures. (Mit Desai via vinodkv)
MAPREDUCE-5896. InputSplits should indicate which locations have the block
cached in memory. (Sandy Ryza via kasha)
MAPREDUCE-5844. Add a configurable delay to reducer-preemption.
(Maysam Yabandeh via kasha)
MAPREDUCE-5790. Made it easier to enable hprof profile options by default.
(Gera Shegalov via vinodkv)
OPTIMIZATIONS
BUG FIXES
MAPREDUCE-5759. Remove unnecessary conf load in Limits (Sandy Ryza)
MAPREDUCE-5014. Extend Distcp to accept a custom CopyListing.
(Srikanth Sundarrajan via amareshwari)
MAPREDUCE-5775. Remove unnecessary job.setNumReduceTasks in SleepJob.createJob
(jhanver chand sharma via devaraj)
MAPREDUCE-4937. MR AM handles an oversized split metainfo file poorly
(Eric Payne via jlowe)
MAPREDUCE-5642. TestMiniMRChildTask fails on Windows.
(Chuan Liu via cnauroth)
MAPREDUCE-5846. Rumen doesn't understand JobQueueChangedEvent (Nathan Roberts via raviprak)
MAPREDUCE-5837. MRAppMaster fails when checking on uber mode. (wheat9)
MAPREDUCE-5749. TestRMContainerAllocator#testReportedAppProgress Failed
(jlowe)
MAPREDUCE-5884. History server uses short user name when canceling tokens
(Mohammad Kamrul Islam via jlowe)
MAPREDUCE-5888. Failed job leaves hung AM after it unregisters (Jason Lowe
via jeagles)
MAPREDUCE-5814. fat jar with *-default.xml may fail when
mapreduce.job.classloader=true. (Gera Shegalov via jlowe)
MAPREDUCE-5309. 2.0.4 JobHistoryParser can't parse certain failed job
history files generated by 2.0.3 history server (Rushabh S Shah via jlowe)
MAPREDUCE-5862. Line records longer than 2x split size aren't handled
correctly (bc Wong via jlowe)
MAPREDUCE-5895. Close streams properly to avoid leakage in TaskLog.
(Kousuke Saruta via devaraj)
MAPREDUCE-5777. Support utf-8 text with Byte Order Marker.
(Zhihai Xu via kasha)
MAPREDUCE-5898. distcp to support preserving HDFS extended attributes(XAttrs)
(Yi Liu via umamahesh)
MAPREDUCE-5920. Add Xattr option in DistCp docs. (Yi Liu via cnauroth)
MAPREDUCE-5924. Changed TaskAttemptImpl to ignore TA_COMMIT_PENDING event
at COMMIT_PENDING state. (Zhijie Shen via jianhe)
MAPREDUCE-5939. StartTime showing up as the epoch time in JHS UI after
upgrade (Chen He via jlowe)
MAPREDUCE-5900. Changed to the interpret container preemption exit code as a
task attempt killing event. (Mayank Bansal via zjshen)
MAPREDUCE-5868. Fixed an issue with TestPipeApplication that was causing the
nightly builds to fail. (Akira Ajisaka via vinodkv)
MAPREDUCE-5517. Fixed MapReduce ApplicationMaster to not validate reduce side
resource configuration for deciding uber-mode on map-only jobs. (Siqi Li via
vinodkv)
MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly
assumes a single dir for mapOutIndex. (Gera Shegalov via kasha)
MAPREDUCE-6002. Made MR task avoid reporting error to AM when the task process
is shutting down. (Wangda Tan via zjshen)
Release 2.4.1 - 2014-06-23
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
MAPREDUCE-5830. Added back the private API HostUtil.getTaskLogUrl(..) for
binary compatibility with older clients like Hive 0.13. (Akira Ajisaka via
vinodkv)
OPTIMIZATIONS
BUG FIXES
MAPREDUCE-5714. Removed forceful JVM exit in shutDownJob.
(Jinghui Wang via Eric Yang)
MAPREDUCE-5818. Added "hsadmin" command into mapred.cmd. (Jian He via zjshen)
MAPREDUCE-5824. Fixed test-failure of TestPipesNonJavaInputFormat in
Windows. (Xuan Gong via vinodkv)
MAPREDUCE-5815. Fixed test-failure of TestMRAppMaster by making MRAppMaster
gracefully handle empty-queue names. (Akira Ajisaka via vinodkv)
MAPREDUCE-5826. Fixed HistoryServerFileSystemStore to use right permissions
on Windows for temporary files and thus also fix the test-issue with
TestHistoryServerFileSystemStateStoreService. (Varun Vasudev via vinodkv)
MAPREDUCE-5828. Fixed a test issue with TestMapReduceJobControl that was
causing it to fail on Windows. (vinodkv)
MAPREDUCE-5827. TestSpeculativeExecutionWithMRApp fails.
(Zhijie Shen via cnauroth)
MAPREDUCE-5833. TestRMContainerAllocator fails ocassionally.
(Zhijie Shen via cnauroth)
MAPREDUCE-5832. Fixed TestJobClient to not fail on JDK7 or on Windows. (Jian
He and Vinod Kumar Vavilapalli via vinodkv)
MAPREDUCE-5841. uber job doesn't terminate on getting mapred job kill
(Sangjin Lee via jlowe)
MAPREDUCE-5843. Fixed TestMRKeyValueTextInputFormat to not leak files and
thus avoid failing on Windows. (Varun Vasudev via vinodkv)
MAPREDUCE-5835. Killing Task might cause the job to go to ERROR state
(Ming Ma via jlowe)
MAPREDUCE-5821. Avoid unintentional reallocation of byte arrays in segments
during merge. (Todd Lipcon via cdouglas)
Release 2.4.0 - 2014-04-07
INCOMPATIBLE CHANGES
NEW FEATURES
MAPREDUCE-5787. Added the ability to keep alive shuffle connections in the
MapReduce shuffle-handler. (Rajesh Balamohan via vinodkv)
IMPROVEMENTS
MAPREDUCE-5464. Add analogs of the SLOTS_MILLIS counters that jive with the
YARN resource model (Sandy Ryza)
MAPREDUCE-5732. Report proper queue when job has been automatically placed
(Sandy Ryza)
MAPREDUCE-5699. Allow setting tags on MR jobs (kasha)
MAPREDUCE-5761. Added a simple log message to denote when encrypted shuffle
is on in the shuffle-handler. (Jian He via vinodkv)
MAPREDUCE-5754. Preserve Job diagnostics in history (Gera Shegalov via
jlowe)
MAPREDUCE-5773. Provide dedicated MRAppMaster syslog length limit (Gera
Shegalov via jlowe)
MAPREDUCE-5553. Allow users to easily access
completed/pending/successful/failed tasks on MR AM web-ui. (Paul Han via
acmurthy)
MAPREDUCE-4052. Improved MapReduce clients to use NodeManagers' ability to
handle cross platform application submissions. (Jian He via vinodkv)
MAPREDUCE-2349. Modified FileInputFormat to be able to issue file and block
location calls in parallel. (Siddharth Seth via vinodkv)
OPTIMIZATIONS
BUG FIXES
MAPREDUCE-5746. Job diagnostics can implicate wrong task for a failed job.
(Jason Lowe via kasha)
MAPREDUCE-5670. CombineFileRecordReader should report progress when moving
to the next file (Chen He via jlowe)
MAPREDUCE-5757. ConcurrentModificationException in JobControl.toList
(jlowe)
MAPREDUCE-5770. Fixed MapReduce ApplicationMaster to correctly redirect
to the YARN's web-app proxy with the correct scheme prefix. (Jian He via
vinodkv)
MAPREDUCE-5768. TestMRJobs.testContainerRollingLog fails on trunk (Gera
Shegalov via jlowe)
MAPREDUCE-5780. SliveTest should use the specified path to get the
particular FileSystem instead of using the default FileSystem. (szetszwo)
MAPREDUCE-5028. Fixed a bug in MapTask that was causing mappers to fail
when a large value of io.sort.mb is set. (Karthik Kambatla via vinodkv)
MAPREDUCE-5778. JobSummary does not escape newlines in the job name (Akira
AJISAKA via jlowe)
MAPREDUCE-5789. Average Reduce time is incorrect on Job Overview page
(Rushabh S Shah via jlowe)
MAPREDUCE-5794. SliveMapper always uses default FileSystem. (szetszwo via
Arpit Agarwal)
MAPREDUCE-5751. MR app master fails to start in some cases if
mapreduce.job.classloader is true (Sangjin Lee via jlowe)
MAPREDUCE-5688. TestStagingCleanup fails intermittently with JDK7 (Mit
Desai via jeagles)
MAPREDUCE-5769. Unregistration to RM should not be called if AM is crashed
before registering with RM (Rohith via jlowe)
MAPREDUCE-5570. Map task attempt with fetch failure has incorrect attempt
finish time (Rushabh S Shah via jlowe)
MAPREDUCE-5806. Fixed a bug in MRAppMaster so as to enable users to properly
override HADOOP_ROOT_LOGGER or HADOOP_CLIENT_OPTS. (Varun Vasudev via
vinodkv)
MAPREDUCE-5791. Shuffle phase is slow in Windows -
FadviseFileRegion::transferTo does not read disks efficiently.
(Nikola Vujic via cnauroth)
MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it
recovers from a commit during a previous attempt. (Xuan Gong via vinodkv)
MAPREDUCE-5805. Fixed MapReduce JobHistory encoding of queue-name to escape
hyphens and thus avoid parsing errors. (Akira AJISAKA via vinodkv)
MAPREDUCE-5810. Removed the faulty and failing streaming test
TestStreamingTaskLog. (Akira Ajisaka via vinodkv)
MAPREDUCE-5813. Fix YarnChild to explicitly load job.xml from the
local-filesystem, rather than rely on the classpath. (Gera Shegalov via
acmurthy)
Release 2.3.1 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
Release 2.3.0 - 2014-02-18
INCOMPATIBLE CHANGES
NEW FEATURES
MAPREDUCE-5265. History server admin service to refresh user and superuser
group mappings (Ashwin Shankar via jlowe)
MAPREDUCE-5356. Ability to refresh aggregated log retention period and
check interval (Ashwin Shankar via jlowe)
MAPREDUCE-5386. Ability to refresh history server job retention and job
cleaner settings (Ashwin Shankar via jlowe)
MAPREDUCE-5411. Refresh size of loaded job cache on history server (Ashwin
Shankar via jlowe)
MAPREDUCE-5332. Support token-preserving restart of history server (jlowe)
IMPROVEMENTS
MAPREDUCE-5329. Allow MR applications to use additional AuxServices,
which are compatible with the default MapReduce shuffle.
(Avner BenHanoch via sseth)
MAPREDUCE-5463. Deprecate SLOTS_MILLIS counters (Tzuyoshi Ozawa via Sandy
Ryza)
MAPREDUCE-5457. Add a KeyOnlyTextOutputReader to enable streaming to write
out text files without separators (Sandy Ryza)
MAPREDUCE-5596. Allow configuring the number of threads used to serve
shuffle connections (Sandy Ryza via jlowe)
MAPREDUCE-434. LocalJobRunner limited to single reducer (Sandy Ryza and
Aaron Kimball via Sandy Ryza)
MAPREDUCE-4421. Run MapReduce framework via the distributed cache (jlowe)
MAPREDUCE-1176. FixedLengthInputFormat and FixedLengthRecordReader
(Mariappan Asokan and BitsOfInfo via Sandy Ryza)
MAPREDUCE-5613. DefaultSpeculator holds and checks hashmap that is always
empty (Gera Shegalov via Sandy Ryza)
MAPREDUCE-5431. Missing pom dependency in MR-client (Timothy St. Clair
via stevel)
MAPREDUCE-5624 Move grizzly-test and junit dependencies to test scope
(Ted Yu via stevel)
MAPREDUCE-5481. Enable uber jobs to have multiple reducers (Sandy Ryza)
MAPREDUCE-5052. Job History UI and web services confusing job start time and
job submit time (Chen He via jeagles)
MAPREDUCE-5692. Add explicit diagnostics when a task attempt is killed due
to speculative execution (Gera Shegalov via Sandy Ryza)
MAPREDUCE-5550. Task Status message (reporter.setStatus) not shown in UI
with Hadoop 2.0 (Gera Shegalov via Sandy Ryza)
MAPREDUCE-3310. Custom grouping comparator cannot be set for Combiners (tucu)
MAPREDUCE-5672. Provide optional RollingFileAppender for container log4j
(syslog) (Gera Shegalov via jlowe)
MAPREDUCE-5725. Make explicit that TestNetworkedJob relies on the Capacity
Scheduler (Sandy Ryza)
MAPREDUCE-5744. Job hangs because
RMContainerAllocator$AssignedRequests.preemptReduce() violates the
comparator contract (Gera Shegalov via kasha)
OPTIMIZATIONS
MAPREDUCE-4680. Job history cleaner should only check timestamps of files in
old enough directories (Robert Kanter via Sandy Ryza)
MAPREDUCE-5484. YarnChild unnecessarily loads job conf twice (Sandy Ryza)
MAPREDUCE-5487. In task processes, JobConf is unnecessarily loaded again
in Limits (Sandy Ryza)
MAPREDUCE-5601. ShuffleHandler fadvises file regions as DONTNEED even when
fetch fails (Sandy Ryza)
BUG FIXES
MAPREDUCE-5569. FloatSplitter is not generating correct splits (Nathan
Roberts via jlowe)
MAPREDUCE-5546. mapred.cmd on Windows set HADOOP_OPTS incorrectly (Chuan Liu
via cnauroth)
MAPREDUCE-5518. Fixed typo "can't read paritions file". (Albert Chu
via devaraj)
MAPREDUCE-5561. org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl
testcase failing on trunk (Karthik Kambatla via jlowe)
MAPREDUCE-5598. TestUserDefinedCounters.testMapReduceJob is flakey
(Robert Kanter via jlowe)
MAPREDUCE-5604. TestMRAMWithNonNormalizedCapabilities fails on Windows due to
exceeding max path length. (cnauroth)
MAPREDUCE-5451. MR uses LD_LIBRARY_PATH which doesn't mean anything in
Windows. (Yingda Chen via cnauroth)
MAPREDUCE-5409. MRAppMaster throws InvalidStateTransitonException: Invalid
event: TA_TOO_MANY_FETCH_FAILURE at KILLED for TaskAttemptImpl (Gera
Shegalov via jlowe)
MAPREDUCE-5674. Missing start and finish time in mapred.JobStatus.
(Chuan Liu via cnauroth)
MAPREDUCE-5650. Job fails when hprof mapreduce.task.profile.map/reduce.params
is specified (Gera Shegalov via Sandy Ryza)
MAPREDUCE-5316. job -list-attempt-ids command does not handle illegal
task-state (Ashwin Shankar via jlowe)
MAPREDUCE-5380. Invalid mapred command should return non-zero exit code
(Stephen Chu via jlowe)
MAPREDUCE-5404. HSAdminServer does not use ephemeral ports in minicluster
mode (Ted Yu via jlowe)
MAPREDUCE-5522. Incorrect oreder expected from JobQueueInfo (Jinghui Wang
via bobby)
MAPREDUCE-5514. Fix TestRMContainerAllocator. (Zhijie Shen via acmurthy)
MAPREDUCE-5102. fix coverage org.apache.hadoop.mapreduce.lib.db and
org.apache.hadoop.mapred.lib.db (Aleksey Gorshkov, Andrey Klochkov, and
Nathan Roberts via jlowe)
MAPREDUCE-5585. TestCopyCommitter#testNoCommitAction Fails on JDK7
(jeagles)
MAPREDUCE-5186. mapreduce.job.max.split.locations causes some splits
created by CombineFileInputFormat to fail (Robert Parker and Jason Lowe
via jlowe)
MAPREDUCE-5610. TestSleepJob fails in jdk7 (Jonathan Eagles via jlowe)
MAPREDUCE-5616. MR Client-AppMaster RPC max retries on socket timeout is too
high. (cnauroth)
MAPREDUCE-5625. TestFixedLengthInputFormat fails in jdk7 environment
(Mariappan Asokan via jeagles)
MAPREDUCE-5631. TestJobEndNotifier.testNotifyRetries fails with Should
have taken more than 5 seconds in jdk7 (Jonathan Eagles via jlowe)
MAPREDUCE-5645. TestFixedLengthInputFormat fails with native libs (Mit
Desai via jeagles)
MAPREDUCE-5632. TestRMContainerAllocator#testUpdatedNodes fails (jeagles)
MAPREDUCE-5656. bzip2 codec can drop records when reading data in splits
(jlowe)
MAPREDUCE-5623. TestJobCleanup fails because of RejectedExecutionException
and NPE. (jlowe)
MAPREDUCE-5679. TestJobHistoryParsing has race condition (Liyin Liang via
jlowe)
MAPREDUCE-5687. Fixed failure in TestYARNRunner caused by YARN-1446. (Jian He
via vinodkv)
MAPREDUCE-5694. Fixed MR AppMaster to shutdown the LogManager so as to avoid
losing syslog in some conditions. (Mohammad Kamrul Islam via vinodkv)
MAPREDUCE-5685. Fixed a bug with JobContext getCacheFiles API inside the
WrappedReducer class. (Yi Song via vinodkv)
MAPREDUCE-5689. MRAppMaster does not preempt reducers when scheduled maps
cannot be fulfilled. (lohit via kasha)
MAPREDUCE-5724. JobHistoryServer does not start if HDFS is not running.
(tucu)
MAPREDUCE-5729. mapred job -list throws NPE (kasha)
MAPREDUCE-5693. Restore MRv1 behavior for log flush (Gera Shegalov via
jlowe)
MAPREDUCE-5723. MR AM container log can be truncated or empty.
(Mohammad Kamrul Islam via kasha)
MAPREDUCE-5743. Fixed the test failure in TestRMContainerAllocator.
(Ted Yu and Vinod Kumar Vavilapalli via zjshen)
Release 2.2.0 - 2013-10-13
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
MAPREDUCE-5504. mapred queue -info inconsistent with types (Kousuke Saruta
via tgraves)
MAPREDUCE-5488. Changed MR client to keep trying to reach the application
when it sees that on attempt's AM is down. (Jian He via vinodkv)
MAPREDUCE-5515. Fixed MR AM's webapp to depend on a new config
mapreduce.ssl.enabled to enable https and disabling it by default as MR AM
needs to set up its own certificates etc and not depend on clusters'.
(Omkar Vinit Joshi via vinodkv)
MAPREDUCE-5505. Clients should be notified job finished only after job
successfully unregistered (Zhijie Shen via bikas)
MAPREDUCE-5503. Fixed a test issue in TestMRJobClient. (Jian He via vinodkv)
MAPREDUCE-5170. Fixed a wrong log message in CombineFileInputFormat class.
(Sangjin Lee via vinodkv)
MAPREDUCE-5525. Increase timeout of TestDFSIO.testAppend and
TestMRJobsWithHistoryService.testJobHistoryData. (Chuan Liu via cnauroth)
MAPREDUCE-5513. ConcurrentModificationException in JobControl (Robert
Parker via jlowe)
MAPREDUCE-5531. Fix compat with hadoop-1 in mapreduce.(TaskID,
TaskAttemptID) by re-introducing missing constructors. (Robert Kanter via
acmurthy)
MAPREDUCE-5545. org.apache.hadoop.mapred.TestTaskAttemptListenerImpl.testCommitWindow
times out (Robert Kanter via jlowe)
MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner
by re-introducing (get,set)PartitionFile which takes in JobConf. (Robert
Kanter via acmurthy)
MAPREDUCE-5538. Fixed MR AppMaster to send job-notification URL only after
the job is really done - a bug caused by MAPREDUCE-5505. (Zhijie Shen via
vinodkv)
MAPREDUCE-5551. Fix compat with hadoop-1 in
SequenceFileAsBinaryOutputFormat.WritableValueBytes by re-introducing
missing constructors. (Zhijie Shen via acmurthy)
MAPREDUCE-5544. JobClient#getJob loads job conf twice. (Sandy Ryza)
MAPREDUCE-5536. Fixed MR AM and JHS to respect
mapreduce.jobhistory.webapp.https.address. (Omkar Vinit Joshi via vinodkv)
MAPREDUCE-5530. Fix compat with hadoop-1 in
mapred.lib.CombinFileInputFormat by re-introducing
isSplittable(FileSystem, Path) api and ensuring semantic compatibility.
(Robert Kanter via acmurthy)
MAPREDUCE-5459. Update documentation on how to run MRv1 examples on YARN.
(Zhijie Shen via acmurthy)
MAPREDUCE-5554. hdfs-site.xml included in hadoop-mapreduce-client-jobclient
tests jar is breaking tests for downstream components (Robert Kanter via
Sandy Ryza)
MAPREDUCE-5489. MR jobs hangs as it does not use the node-blacklisting
feature in RM requests (Zhijie Shen via bikas)
MAPREDUCE-5442. $HADOOP_MAPRED_HOME/$HADOOP_CONF_DIR setting not working on
Windows. (Yingda Chen via cnauroth)
MAPREDUCE-5533. Fixed MR speculation code to track any TaskAttempts that
aren't heart-beating for a while, so that we can aggressively speculate
instead of waiting for task-timeout (Xuan Gong via vinodkv)
MAPREDUCE-5562. Fixed MR App Master to perform pending tasks like staging-dir
cleanup, sending job-end notification correctly when unregister with RM
fails. (Zhijie Shen via vinodkv)
Release 2.1.1-beta - 2013-09-23
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
MAPREDUCE-5478. TeraInputFormat unnecessarily defines its own FileSplit
subclass (Sandy Ryza)
MAPREDUCE-5497. Changed MRAppMaster to sleep only after doing everything else
but just before ClientService to avoid race conditions during RM restart.
(Jian He via vinodkv)
MAPREDUCE-5379. Include token tracking ids in jobconf. (kkambatl via tucu)
MAPREDUCE-5523. Added separate configuration properties for https for JHS
without which even when https is enabled, it starts on http port itself.
(Omkar Vinit Joshi via vinodkv)
OPTIMIZATIONS
MAPREDUCE-5446. TestJobHistoryEvents and TestJobHistoryParsing have race
conditions (jlowe via kihwal)
MAPREDUCE-5462. In map-side sort, swap entire meta entries instead of
indexes for better cache performance. (Sandy Ryza)
MAPREDUCE-1981. Improve getSplits performance by using listLocatedStatus
(Hairong Kuang and Jason Lowe via jlowe)
BUG FIXES
MAPREDUCE-5385. Fixed a bug with JobContext getCacheFiles API. (Omkar Vinit
Joshi via vinodkv)
MAPREDUCE-5428. HistoryFileManager doesn't stop threads when service is
stopped (Karthik Kambatla via jlowe)
MAPREDUCE-5251. Reducer should not implicate map attempt if it has
insufficient space to fetch map output (Ashwin Shankar via jlowe)
MAPREDUCE-5317. Stale files left behind for failed jobs (Ravi Prakash via
jlowe)
MAPREDUCE-5358. MRAppMaster throws invalid transitions for JobImpl
(Devaraj K via jlowe)
MAPREDUCE-3193. FileInputFormat doesn't read files recursively in the
input path dir (Devaraj K via jlowe)
MAPREDUCE-5440. TestCopyCommitter Fails on JDK7 (Robert Parker via jlowe)
MAPREDUCE-5367. Local jobs all use same local working directory
(Sandy Ryza)
MAPREDUCE-5425. Junit in TestJobHistoryServer failing in jdk 7 (Robert
Parker via jlowe)
MAPREDUCE-5454. TestDFSIO fails intermittently on JDK7 (Karthik Kambatla
via Sandy Ryza)
MAPREDUCE-5001. LocalJobRunner has race condition resulting in job
failures (Sandy Ryza via jlowe)
MAPREDUCE-5466. Changed MR AM to not promote history files of intermediate
AMs in case they are exiting because of errors and thus help history-server
pick up the right history file for the last successful AM. (Jian He via
vinodkv)
MAPREDUCE-5468. Fix MR AM recovery for map-only jobs. (vinodkv via
acmurthy)
MAPREDUCE-5470. LocalJobRunner does not work on Windows. (Sandy Ryza via
cnauroth)
MAPREDUCE-5476. Changed MR AM recovery code to cleanup staging-directory
only after unregistering from the RM. (Jian He via vinodkv)
MAPREDUCE-5483. revert MAPREDUCE-5357. (rkanter via tucu)
MAPREDUCE-5441. Changed MR AM to return RUNNING state if exiting when RM
commands to reboot, so that client can continue to track the overall job.
(Jian He via vinodkv)
MAPREDUCE-5475. MRClientService does not verify ACLs properly (jlowe)
MAPREDUCE-5414. TestTaskAttempt fails in JDK7 with NPE (Nemon Lou via
devaraj)
MAPREDUCE-5020. Compile failure with JDK8 (Trevor Robinson via tgraves)
MAPREDUCE-5164. mapred job and queue commands omit HADOOP_CLIENT_OPTS
(Nemon Lou via devaraj)
MAPREDUCE-5493. Cleanup in-memory & on-disk segments to prevent leak on
shuffle completion. (jlowe via acmurthy)
Release 2.1.0-beta - 2013-08-22
INCOMPATIBLE CHANGES
MAPREDUCE-4067. Changed MRClientProtocol api to throw IOException only (Xuan
Gong via vinodkv)
MAPREDUCE-5234. Change mapred.TaskReport and mapreduce.TaskReport for binary
compatibility with mapred in 1.x but incompatible with 0.23.x. (Mayank Bansal
via vinodkv)
MAPREDUCE-5156. Change hadoop examples ProgramDriver to be able to run
1.x examples jar on top of YARN. This change breaks 0.23.x direct usages of
ProgramDriver. (Zhijie Shen via vinodkv)
MAPREDUCE-5233. Add methods that are changed or removed from JobControl.Job
when compared to 1.x. This breaks 0.23.x users of one API in Job. (Mayank
Bansal via vinodkv)
MAPREDUCE-5237. Add methods that were removed from ClusterStatus back into
2.x so as to be compatible with 1.x. Incompatible as
ClusterStatus.UNINITIALIZED_MEMORY_VALUE is a long now and so breaks 0.23.x
but it shouldn't be a big deal in reality. (Zhijie Shen via vinodkv)
NEW FEATURES
HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows
Azure environments. (See breakdown of tasks below for subtasks and
contributors)