This repository has been archived by the owner on Feb 9, 2021. It is now read-only.
/
CHANGES.txt
5423 lines (3418 loc) · 192 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Hadoop HDFS Change Log
Trunk (Unreleased)
INCOMPATIBLE CHANGES
HDFS-3034. Remove the deprecated DFSOutputStream.sync() method. (szetszwo)
NEW FEATURES
HDFS-3125. Add JournalService to enable Journal Daemon. (suresh)
HDFS-3601. Add BlockPlacementPolicyWithNodeGroup to support block placement
with 4-layer network topology. (Junping Du via szetszwo)
HDFS-3077. Implement QuorumJournalManager, a distributed mechanism for
reliably storing HDFS edit logs. See dedicated section below for breakdown
of subtasks.
IMPROVEMENTS
HDFS-1620. Rename HdfsConstants -> HdfsServerConstants, FSConstants ->
HdfsConstants. (Harsh J Chouraria via atm)
HDFS-2197. Refactor RPC call implementations out of NameNode class (todd)
HDFS-2572. Remove unnecessary double-check in DN#getHostName. (harsh)
HDFS-2857. Cleanup BlockInfo class. (suresh)
HDFS-2786. Fix host-based token incompatibilities in DFSUtil. (Kihwal Lee
via jitendra)
HDFS-2486. Remove unnecessary priority level checks in
UnderReplicatedBlocks. (Uma Maheswara Rao G via szetszwo)
HDFS-2878. Fix TestBlockRecovery and move it back into main test directory.
(todd)
HDFS-2655. BlockReaderLocal#skip performs unnecessary IO.
(Brandon Li via jitendra)
HDFS-3009. Remove duplicate code in DFSClient#isLocalAddress by using
NetUtils. (Hari Mankude via suresh)
HDFS-3002. TestNameNodeMetrics need not wait for metrics update.
(suresh)
HDFS-3016. Security in unit tests. (Jaimin Jetly via jitendra)
HDFS-3030. Remove getProtocolVersion and getProtocolSignature from translators.
(jitendra)
HDFS-2976. Remove unnecessary method (tokenRefetchNeeded) in DFSClient.
HDFS-3111. Missing license headers in trunk. (umamahesh)
HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in
the config description for the smaller clusters. (szetszwo via umamahesh)
HDFS-309. FSEditLog should log progress during replay. (Sho Shimauchi
via todd)
HDFS-3131. Improve TestStorageRestore. (Brandon Li via atm)
HDFS-3178. Add states and state handler for journal synchronization in
JournalService. (szetszwo)
HDFS-3273. Refactor BackupImage and FSEditLog, and rename
JournalListener.rollLogs(..) to startLogSegment(..). (szetszwo)
HDFS-3292. Remove the deprecated DiskStatus, getDiskStatus(), getRawUsed()
and getRawCapacity() from DistributedFileSystem. (Arpit Gupta via szetszwo)
HADOOP-8285. HDFS changes for Use ProtoBuf for RpcPayLoadHeader. (sanjay
radia)
HDFS-2743. Streamline usage of bookkeeper journal manager.
(Ivan Kelly via umamahesh)
HDFS-3293. Add toString(), equals(..) and hashCode() to JournalInfo.
(Hari Mankude via szetszwo)
HDFS-3197. Incorrect class comments in a few tests. (Andy Isaacson via eli)
HDFS-3476. Correct the default used in TestDFSClientRetries.busyTest()
after HDFS-3462 (harsh)
HDFS-3040. TestMulitipleNNDataBlockScanner is misspelled. (Madhukara Phatak
via atm)
HDFS-3049. During the normal NN startup process, fall back on a different
edit log if we see one that is corrupt (Colin Patrick McCabe via todd)
HDFS-3478. Test quotas with Long.Max_Value. (Sujay Rau via eli)
HDFS-3498. Support replica removal in BlockPlacementPolicy and make
BlockPlacementPolicyDefault extensible for reusing code in subclasses.
(Junping Du via szetszwo)
HDFS-3571. Allow EditLogFileInputStream to read from a remote URL (todd)
HDFS-3510. Editlog pre-allocation is performed prior to writing edits
to avoid partial edits case disk out of space.(Colin McCabe via suresh)
HDFS-3630 Modify TestPersistBlocks to use both flush and hflush (sanjay)
HDFS-3768. Exception in TestJettyHelper is incorrect.
(Eli Reisman via jghoman)
HDFS-3723. Add support -h, -help to all the commands. (Jing Zhao via
suresh)
HDFS-3803. Change BlockPoolSliceScanner chatty INFO log to DEBUG.
(Andrew Purtell via suresh)
HDFS-3817. Avoid printing SafeModeException stack trace.
(Brandon Li via suresh)
HDFS-3819. Should check whether invalidate work percentage default value is
not greater than 1.0f. (Jing Zhao via jitendra)
HDFS-3844. Add @Override and remove {@inheritdoc} and unnecessary
imports. (Jing Zhao via suresh)
HDFS-3851. DFSOutputStream class code cleanup. (Jing Zhao via suresh)
HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. (harsh)
HDFS-3880. Use Builder to build RPC server in HDFS.
(Brandon Li vias suresh)
HDFS-2127. Add a test that ensure AccessControlExceptions contain
a full path. (Stephen Chu via eli)
HDFS-3995. Use DFSTestUtil.createFile() for file creation and
writing in test cases. (Jing Zhao via suresh)
HDFS-3735. NameNode WebUI should allow sorting live datanode list by fields
Block Pool Used, Block Pool Used(%) and Failed Volumes.
(Brahma Reddy Battula via suresh)
HDFS-4052. BlockManager#invalidateWork should print log outside the lock.
(Jing Zhao via suresh)
HDFS-4110. Refine a log printed in JNStorage. (Liang Xie via suresh)
HDFS-4124. Refactor INodeDirectory#getExistingPathINodes() to enable
returning more than INode array. (Jing Zhao via suresh)
HDFS-4129. Add utility methods to dump NameNode in memory tree for
testing. (szetszwo via suresh)
HDFS-4151. Change the methods in FSDirectory to pass INodesInPath instead
of INode[] as a parameter. (szetszwo)
HDFS-4152. Add a new class BlocksMapUpdateInfo for the parameter in
INode.collectSubtreeBlocksAndClear(..). (Jing Zhao via szetszwo)
HDFS-4153. Add START_MSG/SHUTDOWN_MSG for JournalNode. (liang xie via atm)
HDFS-3935. Add JournalNode to the start/stop scripts (Andy Isaacson via todd)
HDFS-4206. Change the fields in INode and its subclasses to private.
(szetszwo)
OPTIMIZATIONS
BUG FIXES
HDFS-2299. TestOfflineEditsViewer is failing on trunk. (Uma Maheswara Rao G
via atm)
HDFS-2310. TestBackupNode fails since HADOOP-7524 went in.
(Ivan Kelly via todd)
HDFS-2313. Rat excludes has a typo for excluding editsStored files. (atm)
HDFS-2314. MRV1 test compilation broken after HDFS-2197 (todd)
HDFS-46. Change default namespace quota of root directory from
Integer.MAX_VALUE to Long.MAX_VALUE. (Uma Maheswara Rao G via szetszwo)
HDFS-2373. Commands using WebHDFS and hftp print unnecessary debug
info on the console with security enabled. (Arpit Gupta via suresh)
HDFS-2776. Missing interface annotation on JournalSet.
(Brandon Li via jitendra)
HDFS-2908. Add apache license header for StorageReport.java. (Brandon Li
via jitendra)
HDFS-3037. TestMulitipleNNDataBlockScanner#testBlockScannerAfterRestart is
racy. (atm)
HDFS-2966. TestNameNodeMetrics tests can fail under load. (stevel)
HDFS-3116. Typo in fetchdt error message. (AOE Takashi via atm)
HDFS-3126. Journal stream from Namenode to BackupNode needs to have
timeout. (Hari Mankude via suresh)
HDFS-3121. Add HDFS tests for HADOOP-8014 change. (John George via
suresh)
HDFS-3119. Overreplicated block is not deleted even after the replication
factor is reduced after sync follwed by closing that file. (Ashish Singhi
via umamahesh)
HDFS-3265. PowerPc Build error. (Kumar Ravi via mattf)
HDFS-2312. FSNamesystem javadoc incorrectly says its for DNs. (harsh)
HDFS-3163. TestHDFSCLI.testAll fails if the user name is not all lowercase.
(Brandon Li via atm)
HDFS-3368. Missing blocks due to bad DataNodes coming up and down. (shv)
HDFS-3462. TestDFSClientRetries.busyTest() should restore default
xceiver count in the config. (Madhukara Phatak via harsh)
HDFS-3550. Fix raid javadoc warnings. (Jason Lowe via daryn)
HDFS-3549. Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn)
HDFS-3541. Deadlock between recovery, xceiver and packet responder (Vinay via umamahesh)
HDFS-3482. hdfs balancer throws ArrayIndexOutOfBoundsException
if option is specified without values. ( Madhukara Phatak via umamahesh)
HDFS-3614. Revert unused MiniDFSCluster constructor from HDFS-3049.
(acmurthy via eli)
HDFS-3792. Fix two findbugs introduced by HDFS-3695 (todd)
HDFS-3827. TestHASafeMode#assertSafemode method should be made static.
(Jing Zhao via suresh)
HDFS-3834. Remove unused static fields NAME, DESCRIPTION and Usage from
Command. (Jing Zhao via suresh)
HADOOP-8158. Interrupting hadoop fs -put from the command line
causes a LeaseExpiredException. (daryn via harsh)
HDFS-2434. TestNameNodeMetrics.testCorruptBlock fails intermittently.
(Jing Zhao via suresh)
HDFS-4067. TestUnderReplicatedBlocks intermittently fails due to
ReplicaAlreadyExistsException. (Jing Zhao via suresh)
HDFS-4115. TestHDFSCLI.testAll fails one test due to number format.
(Trevor Robinson via suresh)
HDFS-4165. Faulty sanity check in FsDirectory.unprotectedSetQuota.
(Binglin Chang via suresh)
HDFS-4105. The SPNEGO user for secondary namenode should use the web
keytab. (Arpit Gupta via jitendra)
BREAKDOWN OF HDFS-3077 SUBTASKS
HDFS-3077. Quorum-based protocol for reading and writing edit logs.
(todd, Brandon Li, and Hari Mankude via todd)
HDFS-3694. Fix getEditLogManifest to fetch httpPort if necessary (todd)
HDFS-3692. Support purgeEditLogs() call to remotely purge logs on JNs
(todd)
HDFS-3693. JNStorage should read its storage info even before a writer
becomes active (todd)
HDFS-3725. Fix QJM startup when individual JNs have gaps (todd)
HDFS-3741. Exhaustive failure injection test for skipped RPCs (todd)
HDFS-3773. TestNNWithQJM fails after HDFS-3741. (atm)
HDFS-3793. Implement genericized format() in QJM (todd)
HDFS-3795. QJM: validate journal dir at startup (todd)
HDFS-3798. Avoid throwing NPE when finalizeSegment() is called on invalid
segment (todd)
HDFS-3799. QJM: handle empty log segments during recovery (todd)
HDFS-3797. QJM: add segment txid as a parameter to journal() RPC (todd)
HDFS-3800. improvements to QJM fault testing (todd)
HDFS-3823. QJM: TestQJMWithFaults fails occasionally because of missed
setting of HTTP port. (todd and atm)
HDFS-3826. QJM: Some trivial logging / exception text improvements. (todd
and atm)
HDFS-3839. QJM: hadoop-daemon.sh should be updated to accept "journalnode"
(eli)
HDFS-3845. Fixes for edge cases in QJM recovery protocol (todd)
HDFS-3877. QJM: Provide defaults for dfs.journalnode.*address (eli)
HDFS-3863. Track last "committed" txid in QJM (todd)
HDFS-3869. Expose non-file journal manager details in web UI (todd)
HDFS-3884. Journal format() should reset cached values (todd)
HDFS-3870. Add metrics to JournalNode (todd)
HDFS-3891. Make selectInputStreams throw IOE instead of RTE (todd)
HDFS-3726. If a logger misses an RPC, don't retry that logger until next
segment (todd)
HDFS-3893. QJM: Make QJM work with security enabled. (atm)
HDFS-3897. QJM: TestBlockToken fails after HDFS-3893. (atm)
HDFS-3898. QJM: enable TCP_NODELAY for IPC (todd)
HDFS-3885. QJM: optimize log sync when JN is lagging behind (todd)
HDFS-3900. QJM: avoid validating log segments on log rolls (todd)
HDFS-3901. QJM: send 'heartbeat' messages to JNs even when they are
out-of-sync (todd)
HDFS-3899. QJM: Add client-side metrics (todd)
HDFS-3914. QJM: acceptRecovery should abort current segment (todd)
HDFS-3915. QJM: Failover fails with auth error in secure cluster (todd)
HDFS-3906. QJM: quorum timeout on failover with large log segment (todd)
HDFS-3840. JournalNodes log JournalNotFormattedException backtrace error
before being formatted (todd)
HDFS-3894. QJM: testRecoverAfterDoubleFailures can be flaky due to IPC
client caching (todd)
HDFS-3926. QJM: Add user documentation for QJM. (atm)
HDFS-3943. QJM: remove currently-unused md5sum field (todd)
HDFS-3950. QJM: misc TODO cleanup, improved log messages, etc. (todd)
HDFS-3955. QJM: Make acceptRecovery() atomic. (todd)
HDFS-3956. QJM: purge temporary files when no longer within retention
period (todd)
HDFS-4004. TestJournalNode#testJournal fails because of test case execution
order (Chao Shi via todd)
HDFS-4017. Unclosed FileInputStream in GetJournalEditServlet
(Chao Shi via todd)
Release 2.0.3-alpha - Unreleased
INCOMPATIBLE CHANGES
HDFS-4122. Cleanup HDFS logs and reduce the size of logged messages.
(suresh)
NEW FEATURES
HDFS-2656. Add libwebhdfs, a pure C client based on WebHDFS.
(Jaimin D Jetly and Jing Zhao via szetszwo)
HDFS-3912. Detect and avoid stale datanodes for writes.
(Jing Zhao via suresh)
HDFS-4059. Add number of stale DataNodes to metrics. (Jing Zhao via suresh)
HDFS-4155. libhdfs implementation of hsync API (Liang Xie via todd)
IMPROVEMENTS
HDFS-3925. Prettify PipelineAck#toString() for printing to a log
(Andrew Wang via todd)
HDFS-3939. NN RPC address cleanup. (eli)
HDFS-3373. Change DFSClient input stream socket cache to global static and
add a thread to cleanup expired cache entries. (John George via szetszwo)
HDFS-3896. Add descriptions for dfs.namenode.rpc-address and
dfs.namenode.servicerpc-address to hdfs-default.xml. (Jeff Lord via atm)
HDFS-3996. Add debug log removed in HDFS-3873 back. (eli)
HDFS-3916. libwebhdfs (C client) code cleanups.
(Colin Patrick McCabe via eli)
HDFS-3813. Log error message if security and WebHDFS are enabled but
principal/keytab are not configured. (Stephen Chu via atm)
HDFS-3483. Better error message when hdfs fsck is run against a ViewFS
config. (Stephen Fritz via atm)
HDFS-3682. MiniDFSCluster#init should provide more info when it fails.
(todd via eli)
HDFS-4008. TestBalancerWithEncryptedTransfer needs a timeout. (eli)
HDFS-4007. Rehabilitate bit-rotted unit tests under
hadoop-hdfs-project/hadoop-hdfs/src/test/unit/
(Colin Patrick McCabe via todd)
HDFS-4041. Hadoop HDFS Maven protoc calls must not depend on external
sh script. (Chris Nauroth via suresh)
HADOOP-8911. CRLF characters in source and text files.
(Raja Aluri via suresh)
HDFS-4037. Rename the getReplication() method in BlockCollection to
getBlockReplication(). (szetszwo)
HDFS-4036. Remove "throws UnresolvedLinkException" from
FSDirectory.unprotectedAddFile(..). (Jing Zhao via szetszwo)
HDFS-2946. HA: Put a cap on the number of completed edits files retained
by the NN. (atm)
HDFS-4029. GenerationStamp should use an AtomicLong. (eli)
HDFS-4068. DatanodeID and DatanodeInfo member should be private. (eli)
HDFS-4073. Two minor improvements to FSDirectory. (Jing Zhao via szetszwo)
HDFS-4074. Remove the unused default constructor from INode. (Brandon Li
via szetszwo)
HDFS-4053. Increase the default block size. (eli)
HDFS-4088. Remove "throws QuotaExceededException" from an
INodeDirectoryWithQuota constructor. (szetszwo)
HDFS-4099. Clean up replication code and add more javadoc. (szetszwo)
HDFS-4107. Add utility methods for casting INode to INodeFile and
INodeFileUnderConstruction. (szetszwo)
HDFS-4112. A few improvements on INodeDirectory include adding a utility
method for casting; avoiding creation of new empty lists; cleaning up
some code and rewriting some javadoc. (szetszwo)
HDFS-4121. Add namespace declarations in hdfs .proto files for languages
other than java. (Binglin Chang via suresh)
HDFS-3573. Supply NamespaceInfo when instantiating JournalManagers (todd)
HDFS-3695. Genericize format() to non-file JournalManagers. (todd)
HDFS-3789. JournalManager#format() should be able to throw IOException
(Ivan Kelly via todd)
HDFS-3916. libwebhdfs testing code cleanup. (Jing Zhao via suresh)
HDFS-4143. Change blocks to private in INodeFile and renames isLink() to
isSymlink() in INode. (szetszwo)
HDFS-4046. Rename ChecksumTypeProto enum NULL since it is illegal in
C/C++. (Binglin Chang via suresh)
HDFS-4048. Use ERROR instead of INFO for volume failure logs.
(Stephen Chu via eli)
HDFS-1322. Document umask in DistributedFileSystem#mkdirs javadocs.
(Colin Patrick McCabe via eli)
HDFS-4038. Override toString() for BookKeeperEditLogInputStream.
(Vinay via umamahesh)
OPTIMIZATIONS
BUG FIXES
HDFS-3919. MiniDFSCluster:waitClusterUp can hang forever.
(Andy Isaacson via eli)
HDFS-3924. Multi-byte id in HdfsVolumeId. (Andrew Wang via atm)
HDFS-3936. MiniDFSCluster shutdown races with BlocksMap usage. (eli)
HDFS-3951. datanode web ui does not work over HTTPS when datanode is started in secure mode. (tucu)
HDFS-3949. NameNodeRpcServer#join should join on both client and
server RPC servers. (eli)
HDFS-3932. NameNode Web UI broken if the rpc-address is set to the wildcard.
(Colin Patrick McCabe via eli)
HDFS-3931. TestDatanodeBlockScanner#testBlockCorruptionPolicy2 is broken.
(Andy Isaacson via eli)
HDFS-3964. Make NN log of fs.defaultFS debug rather than info. (eli)
HDFS-3992. Method org.apache.hadoop.hdfs.TestHftpFileSystem.tearDown()
sometimes throws NPEs. (Ivan A. Veselovsky via atm)
HDFS-3753. Tests don't run with native libraries.
(Colin Patrick McCabe via eli)
HDFS-4000. TestParallelLocalRead fails with "input ByteBuffers
must be direct buffers". (Colin Patrick McCabe via eli)
HDFS-3999. HttpFS OPEN operation expects len parameter, it should be length. (tucu)
HDFS-4006. TestCheckpoint#testSecondaryHasVeryOutOfDateImage
occasionally fails due to unexpected exit. (todd via eli)
HDFS-4003. test-patch should build the common native libs before
running hdfs tests. (Colin Patrick McCabe via eli)
HDFS-4018. testMiniDFSClusterWithMultipleNN is missing some
cluster cleanup. (eli)
HDFS-4020. TestRBWBlockInvalidation may time out. (eli)
HDFS-4021. Misleading error message when resources are low on the NameNode.
(Christopher Conner via atm)
HDFS-4044. Duplicate ChecksumType definition in HDFS .proto files.
(Binglin Chang via suresh)
HDFS-4049. Fix hflush performance regression due to nagling delays
(todd)
HDFS-3678. Edit log files are never being purged from 2NN. (atm)
HDFS-4058. DirectoryScanner may fail with IOOB if the directory
scanning threads return out of volume order. (eli)
HDFS-3985. Add timeouts to TestMulitipleNNDataBlockScanner. (todd via eli)
HDFS-4061. TestBalancer and TestUnderReplicatedBlocks need timeouts. (eli)
HDFS-3997. OfflineImageViewer incorrectly passes value of imageVersion when
visiting IS_COMPRESSED element. (Mithun Radhakrishnan via atm)
HDFS-4055. TestAuditLogs is flaky. (Binglin Chang via eli)
HDFS-4072. On file deletion remove corresponding blocks pending
replications. (Jing Zhao via suresh)
HDFS-4022. Replication not happening for appended block.
(Vinay via umamahesh)
HDFS-3948. Do not use hflush in TestWebHDFS.testNamenodeRestart() since the
out stream returned by WebHdfsFileSystem does not support it. (Jing Zhao
via szetszwo)
HDFS-3616. Fix a ConcurrentModificationException bug that BP actor threads
may not be shutdown properly in DataNode. (Jing Zhao via szetszwo)
HDFS-4127. Log message is not correct in case of short of replica.
(Junping Du via suresh)
HADOOP-8994. TestDFSShell creates file named "noFileHere", making further
tests hard to understand (Andy Isaacson via daryn)
HDFS-3809. Make BKJM use protobufs for all serialization with ZK.
(Ivan Kelly via umamahesh)
HDFS-3804. TestHftpFileSystem fails intermittently with JDK7
(Trevor Robinson via daryn)
HDFS-4132. When libwebhdfs is not enabled, nativeMiniDfsClient frees
uninitialized memory (Colin Patrick McCabe via todd)
HDFS-1331. dfs -test should work like /bin/test (Andy Isaacson via daryn)
HDFS-3979. For hsync, datanode should wait for the local sync to complete
before sending ack. (Lars Hofhansl via szetszwo)
HDFS-3625. Fix TestBackupNode by properly initializing edit log during
startup. (Junping Du via todd)
HDFS-4138. BackupNode startup fails due to uninitialized edit log.
(Kihwal Lee via shv)
HDFS-3810. Implement format() for BKJM (Ivan Kelly via umamahesh)
HDFS-4162. Some malformed and unquoted HTML strings are returned from
datanode web ui. (Darek Dagit via suresh)
HDFS-4164. fuse_dfs: add -lrt to the compiler command line on Linux.
(Colin Patrick McCabe via eli)
HDFS-3921. NN will prematurely consider blocks missing when entering active
state while still in safe mode. (atm)
HDFS-4106. BPServiceActor#lastHeartbeat, lastBlockReport and
lastDeletedReport should be volatile. (Jing Zhao via suresh)
HDFS-4139. fuse-dfs RO mode still allows file truncation.
(Colin Patrick McCabe via eli)
HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory
(Andy Isaacson via daryn)
HDFS-3623. BKJM: zkLatchWaitTimeout hard coded to 6000. Make use of ZKSessionTimeout instead.
(umamahesh)
HDFS-4100. Fix all findbug security warings. (Liang Xie via eli)
HDFS-3507. DFS#isInSafeMode needs to execute only on Active NameNode.
(Vinay via atm)
HDFS-4156. Seeking to a negative position should throw an IOE.
(Eli Reisman via eli)
HDFS-4171. WebHDFS and HttpFs should accept only valid Unix user
names. (tucu)
HDFS-4178. Shell scripts should not close stderr (Andy Isaacson via daryn)
Release 2.0.2-alpha - 2012-09-07
INCOMPATIBLE CHANGES
HDFS-3446. HostsFileReader silently ignores bad includes/excludes
(Matthew Jacobs via todd)
HDFS-3755. Creating an already-open-for-write file with overwrite=true fails
(todd)
NEW FEATURES
HDFS-744. Support hsync in HDFS. (Lars Hofhansl via szetszwo)
HDFS-3042. Automatic failover support for NameNode HA (todd)
(see dedicated section below for breakdown of subtasks)
HDFS-3518. Add a utility method HdfsUtils.isHealthy(uri) for checking if
the given HDFS is healthy. (szetszwo)
HDFS-3113. httpfs does not support delegation tokens. (tucu)
HDFS-3513. HttpFS should cache filesystems. (tucu)
HDFS-3637. Add support for encrypting the DataTransferProtocol. (atm)
HDFS-3150. Add option for clients to contact DNs via hostname. (eli)
HDFS-2793. Add an admin command to trigger an edit log roll. (todd)
HDFS-3703. Datanodes are marked stale if heartbeat is not received in
configured timeout and are selected as the last location to read from.
(Jing Zhao via suresh)
IMPROVEMENTS
HDFS-3390. DFSAdmin should print full stack traces of errors when DEBUG
logging is enabled. (atm)
HDFS-3341. Change minimum RPC versions to respective SNAPSHOTs instead of
final releases. (todd)
HDFS-3369. Rename {get|set|add}INode(..) methods in BlockManager and
BlocksMap to {get|set|add}BlockCollection(..). (John George via szetszwo)
HDFS-3134. harden edit log loader against malformed or malicious input.
(Colin Patrick McCabe via eli)
HDFS-3230. Cleanup DatanodeID creation in the tests. (eli)
HDFS-3401. Cleanup DatanodeDescriptor creation in the tests. (eli)
HDFS-3400. DNs should be able start with jsvc even if security is disabled.
(atm via eli)
HDFS-3404. Make putImage in GetImageServlet infer remote address to fetch
from request. (atm)
HDFS-3335. check for edit log corruption at the end of the log
(Colin Patrick McCabe via todd)
HDFS-3417. Rename BalancerDatanode#getName to getDisplayName to be
consistent with Datanode. (eli)
HDFS-3416. Cleanup DatanodeID and DatanodeRegistration
constructors used by testing. (eli)
HDFS-3419. Cleanup LocatedBlock. (eli)
HDFS-3440. More effectively limit stream memory consumption when reading
corrupt edit logs (Colin Patrick McCabe via todd)
HDFS-3438. BootstrapStandby should not require a rollEdits on active node
(todd)
HDFS-2885. Remove "federation" from the nameservice config options.
(Tsz Wo (Nicholas) Sze via eli)
HDFS-3394. Do not use generic in INodeFile.getLastBlock(): the run-time
ClassCastException check is useless since generic type information is only
available in compile-time. (szetszwo)
HDFS-3454. Balancer unconditionally logs InterruptedException at
INFO level on shutdown if security is enabled. (eli)
HDFS-1013. Miscellaneous improvements to HTML markup for web UIs
(Eugene Koontz via todd)
HDFS-3052. Change INodeFile and INodeFileUnderConstruction to package
private. (szetszwo)
HDFS-3520. Add transfer rate logging to TransferFsImage. (eli)
HDFS-3504. Support configurable retry policy in DFSClient for RPC
connections and RPC calls, and add MultipleLinearRandomRetry, a new retry
policy. (szetszwo)
HDFS-3372. offlineEditsViewer should be able to read a binary
edits file with recovery mode. (Colin Patrick McCabe via eli)
HDFS-3516. Check content-type in WebHdfsFileSystem. (szetszwo)
HDFS-3535. Audit logging should log denied accesses. (Andy Isaacson via eli)
HDFS-3481. Refactor HttpFS handling of JAX-RS query string parameters (tucu)
HDFS-3572. Cleanup code which inits SPNEGO in HttpServer (todd)
HDFS-3475. Make the replication monitor multipliers configurable.
(harsh via eli)
HDFS-3343. Improve metrics for DN read latency (Andrew Wang via todd)
HDFS-3170. Add more useful metrics for write latency (Matthew Jacobs via
todd)
HDFS-3604. Add dfs.webhdfs.enabled to hdfs-default.xml. (eli)
HDFS-2988. Improve error message when storage directory lock fails
(Miomir Boljanovic via harsh)
HDFS-2391. Newly set BalancerBandwidth value is not displayed anywhere.
(harsh)
HDFS-3067. NPE in DFSInputStream.readBuffer if read is repeated on
corrupted block. (Henry Robinson via atm)
HDFS-3555. idle client socket triggers DN ERROR log
(should be INFO or DEBUG). (Andy Isaacson via harsh)
HDFS-3568. fuse_dfs: add support for security. (Colin McCabe via atm)
HDFS-3629. Fix the typo in the error message about inconsistent
storage layout version. (Brandon Li via harsh)
HDFS-3613. GSet prints some INFO level values, which aren't
really very useful to all (Andrew Wang via harsh)
HDFS-3611. NameNode prints unnecessary WARNs about edit log normally skipping
a few bytes. (Colin Patrick McCabe via harsh)
HDFS-3582. Hook daemon process exit for testing. (eli)
HDFS-3641. Move server Util time methods to common and use now
instead of System#currentTimeMillis. (eli)
HDFS-3633. libhdfs: hdfsDelete should pass JNI_FALSE or JNI_TRUE.
(Colin Patrick McCabe via eli)
HDFS-799. libhdfs must call DetachCurrentThread when a thread is destroyed.
(Colin Patrick McCabe via eli)
HDFS-3306. fuse_dfs: don't lock release operations.
(Colin Patrick McCabe via eli)
HDFS-3612. Single namenode image directory config warning can
be improved. (Andy Isaacson via harsh)
HDFS-3606. libhdfs: create self-contained unit test.
(Colin Patrick McCabe via eli)
HDFS-3539. libhdfs code cleanups. (Colin Patrick McCabe via eli)
HDFS-3610. fuse_dfs: Provide a way to use the default (configured) NN URI.
(Colin Patrick McCabe via eli)
HDFS-3663. MiniDFSCluster should capture the code path that led to
the first ExitException. (eli)
HDFS-3659. Add missing @Override to methods across the hadoop-hdfs
project. (Brandon Li via harsh)
HDFS-3537. Move libhdfs and fuse-dfs source to native subdirectories.
(Colin Patrick McCabe via eli)
HDFS-3665. Add a test for renaming across file systems via a symlink. (eli)
HDFS-3666. Plumb more exception messages to terminate. (eli)
HDFS-3673. libhdfs: fix some compiler warnings. (Colin Patrick McCabe via eli)
HDFS-3675. libhdfs: follow documented return codes. (Colin Patrick McCabe via eli)
HDFS-1249. With fuse-dfs, chown which only has owner (or only group)
argument fails with Input/output error. (Colin Patrick McCabe via eli)
HDFS-3583. Convert remaining tests to Junit4. (Andrew Wang via atm)
HDFS-3711. Manually convert remaining tests to JUnit4. (Andrew Wang via atm)
HDFS-3650. Use MutableQuantiles to provide latency histograms for various
operations. (Andrew Wang via atm)
HDFS-3667. Add retry support to WebHdfsFileSystem. (szetszwo)
HDFS-3291. add test that covers HttpFS working w/ a non-HDFS Hadoop
filesystem (tucu)
HDFS-3634. Add self-contained, mavenized fuse_dfs test. (Colin Patrick
McCabe via atm)
HDFS-3190. Simple refactors in existing NN code to assist
QuorumJournalManager extension. (todd)
HDFS-3276. initializeSharedEdits should have a -nonInteractive flag (todd)
HDFS-3765. namenode -initializeSharedEdits should be able to initialize
all shared storages. (Vinay and todd via todd)
HDFS-3802. StartupOption.name in HdfsServerConstants should be final.
(Jing Zhao via szetszwo)
HDFS-3796. Speed up edit log tests by avoiding fsync() (todd)
HDFS-2963. Console Output is confusing while executing metasave
(dfsadmin command). (Andrew Wang via eli)
HDFS-3672. Expose disk-location information for blocks to enable better
scheduling. (Andrew Wang via atm)
HDFS-2727. libhdfs should get the default block size from the server.
(Colin Patrick McCabe via eli)
HDFS-2686. Remove DistributedUpgrade related code. (suresh)
HDFS-3832. Remove protocol methods related to DistributedUpgrade. (suresh)
HDFS-3177. Update DFSClient and DataXceiver to handle different checkum
types in file checksum computation. (Kihwal Lee via szetszwo)
HDFS-3871. Change NameNodeProxies to use RetryUtils. (Arun C Murthy
via szetszwo)
HDFS-3887. Remove redundant chooseTarget methods in BlockPlacementPolicy.
(Jing Zhao via szetszwo)
HDFS-3888. Clean up BlockPlacementPolicyDefault. (Jing Zhao via szetszwo)
HDFS-3907. Allow multiple users for local block readers. (eli)
HDFS-3910. DFSTestUtil#waitReplication should timeout. (eli)
HDFS-3920. libwebdhfs string processing and using strerror consistently
to handle all errors. (Jing Zhao via suresh)
OPTIMIZATIONS
HDFS-2982. Startup performance suffers when there are many edit log
segments. (Colin Patrick McCabe via todd)
HDFS-2834. Add a ByteBuffer-based read API to DFSInputStream.
(Henry Robinson via todd)
HDFS-3110. Use directRead API to reduce the number of buffer copies in
libhdfs (Henry Robinson via todd)
HDFS-3697. Enable fadvise readahead by default. (todd)
HDFS-2421. Improve the concurrency of SerialNumberMap in NameNode.
(Jing Zhao and Weiyan Wang via szetszwo)
HDFS-3866. HttpFS POM should have property where to download tomcat from (zero45 via tucu)
BUG FIXES
HDFS-3385. The last block of INodeFileUnderConstruction is not
necessarily a BlockInfoUnderConstruction, so do not cast it in
FSNamesystem.recoverLeaseInternal(..). (szetszwo)
HDFS-3414. Balancer does not find NameNode if rpc-address or
servicerpc-address are not set in client configs. (atm)
HDFS-3031. Fix complete() and getAdditionalBlock() RPCs to be idempotent
(todd)
HDFS-2759. Pre-allocate HDFS edit log files after writing version number.
(atm)
HDFS-3413. TestFailureToReadEdits timing out. (atm)
HDFS-3422. TestStandbyIsHot timeouts too aggressive (todd)
HDFS-3433. GetImageServlet should allow administrative requestors when
security is enabled. (atm)
HDFS-1153. dfsnodelist.jsp should handle invalid input parameters.
(Ravi Phulari via eli)
HDFS-3434. InvalidProtocolBufferException when visiting DN
browseDirectory.jsp (eli)
HDFS-2800. Fix cancellation of checkpoints in the standby node to be more
reliable. (todd)
HDFS-3391. Fix InvalidateBlocks to compare blocks including their
generation stamps. (todd)
HDFS-3444. hdfs groups command doesn't work with security enabled. (atm)
HDFS-3415. Make sure all layout versions are the same for all storage
directories in the Namenode. (Brandon Li via szetszwo)
HDFS-3436. In DataNode.transferReplicaForPipelineRecovery(..), it should
use the stored generation stamp to check if the block is valid. (Vinay
via szetszwo)
HDFS-3460. HttpFS proxyuser validation with Kerberos ON uses full
principal name. (tucu)
HDFS-3484. hdfs fsck doesn't work if NN HTTP address is set to
0.0.0.0 even if NN RPC address is configured. (atm via eli)
HDFS-3486. offlineimageviewer can't read fsimage files that contain
persistent delegation tokens. (Colin Patrick McCabe via eli)
HDFS-3487. offlineimageviewer should give byte offset information
when it encounters an exception. (Colin Patrick McCabe via eli)
HDFS-3442. Incorrect count for Missing Replicas in FSCK report. (Andrew
Wang via atm)
HDFS-3501. Checkpointing with security enabled will stop working
after ticket lifetime expires. (atm via eli)
HDFS-3266. DFSTestUtil#waitCorruptReplicas doesn't sleep between checks.
(Madhukara Phatak via atm)
HDFS-3505. DirectoryScanner does not join all threads in shutdown.
(Colin Patrick McCabe via eli)
HDFS-3485. DataTransferThrottler will over-throttle when currentTimeMillis
jumps (Andy Isaacson via todd)
HDFS-2914. HA: Standby should not enter safemode when resources are low.
(Vinay via atm)
HDFS-3235. MiniDFSClusterManager doesn't correctly support -format option.
(Henry Robinson via atm)
HDFS-3514. Add missing TestParallelLocalRead. (Henry Robinson via atm)
HDFS-3243. TestParallelRead timing out on jenkins. (Henry Robinson via todd)
HDFS-3490. DatanodeWebHdfsMethods throws NullPointerException if
NamenodeRpcAddressParam is not set. (szetszwo)
HDFS-2797. Fix misuses of InputStream#skip in the edit log code.
(Colin Patrick McCabe via eli)
HDFS-3517. TestStartup should bind ephemeral ports. (eli)
HDFS-3522. If a namenode is in safemode, it should throw SafeModeException
when getBlockLocations has zero locations. (Brandon Li via szetszwo)
HDFS-3408. BKJM : Namenode format fails, if there is no BK root. (Rakesh R via umamahesh)
HDFS-3389. Document the BKJM usage in Namenode HA. (umamahesh and Ivan Kelly via umamahesh)
HDFS-3531. EditLogFileOutputStream#preallocate should check for
incomplete writes. (Colin Patrick McCabe via eli)
HDFS-766. Error message not clear for set space quota out of boundary
values. (Jon Zuanich via atm)
HDFS-3480. Multiple SLF4J binding warning. (Vinay via eli)
HDFS-3524. Update TestFileLengthOnClusterRestart for HDFS-3522. (Brandon
Li via szetszwo)
HDFS-3559. DFSTestUtil: use Builder class to construct DFSTestUtil
instances. (Colin Patrick McCabe via atm)
HDFS-3551. WebHDFS CREATE should use client location for HTTP redirection.
(szetszwo)
HDFS-3157. Fix a bug in the case that the generation stamps of the stored
block in a namenode and the reported block from a datanode do not match.
(Ashish Singhi via szetszwo)