-
Notifications
You must be signed in to change notification settings - Fork 8.7k
/
CHANGES.txt
11662 lines (7371 loc) · 412 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Hadoop HDFS Change Log
Trunk (Unreleased)
INCOMPATIBLE CHANGES
HDFS-3034. Remove the deprecated DFSOutputStream.sync() method. (szetszwo)
HDFS-5079. Cleaning up NNHAStatusHeartbeat.State from
DatanodeProtocolProtos. (Tao Luo via shv)
HDFS-5570. Deprecate hftp / hsftp and replace them with webhdfs / swebhdfs.
(wheat9)
HDFS-2538. option to disable fsck dots (Mohammad Kamrul Islam via aw)
HDFS-7302. Remove "downgrade" from "namenode -rollingUpgrade" startup
option since it may incorrectly finalize an ongoing rolling upgrade.
(Kai Sasaki via szetszwo)
HDFS-7985. WebHDFS should be always enabled. (Li Lu via wheat9)
NEW FEATURES
HDFS-3125. Add JournalService to enable Journal Daemon. (suresh)
IMPROVEMENTS
HDFS-4665. Move TestNetworkTopologyWithNodeGroup to common.
(Junping Du via llu)
HDFS-1620. Rename HdfsConstants -> HdfsServerConstants, FSConstants ->
HdfsConstants. (Harsh J Chouraria via atm)
HDFS-2197. Refactor RPC call implementations out of NameNode class (todd)
HDFS-2572. Remove unnecessary double-check in DN#getHostName. (harsh)
HDFS-2857. Cleanup BlockInfo class. (suresh)
HDFS-2786. Fix host-based token incompatibilities in DFSUtil. (Kihwal Lee
via jitendra)
HDFS-2878. Fix TestBlockRecovery and move it back into main test directory.
(todd)
HDFS-2655. BlockReaderLocal#skip performs unnecessary IO.
(Brandon Li via jitendra)
HDFS-3009. Remove duplicate code in DFSClient#isLocalAddress by using
NetUtils. (Hari Mankude via suresh)
HDFS-3002. TestNameNodeMetrics need not wait for metrics update.
(suresh)
HDFS-3016. Security in unit tests. (Jaimin Jetly via jitendra)
HDFS-3030. Remove getProtocolVersion and getProtocolSignature from translators.
(jitendra)
HDFS-3111. Missing license headers in trunk. (umamahesh)
HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in
the config description for the smaller clusters. (szetszwo via umamahesh)
HDFS-309. FSEditLog should log progress during replay. (Sho Shimauchi
via todd)
HDFS-3131. Improve TestStorageRestore. (Brandon Li via atm)
HDFS-3178. Add states and state handler for journal synchronization in
JournalService. (szetszwo)
HDFS-3273. Refactor BackupImage and FSEditLog, and rename
JournalListener.rollLogs(..) to startLogSegment(..). (szetszwo)
HDFS-3292. Remove the deprecated DiskStatus, getDiskStatus(), getRawUsed()
and getRawCapacity() from DistributedFileSystem. (Arpit Gupta via szetszwo)
HADOOP-8285. HDFS changes for Use ProtoBuf for RpcPayLoadHeader. (sanjay
radia)
HDFS-2743. Streamline usage of bookkeeper journal manager.
(Ivan Kelly via umamahesh)
HDFS-3293. Add toString(), equals(..) and hashCode() to JournalInfo.
(Hari Mankude via szetszwo)
HDFS-3197. Incorrect class comments in a few tests. (Andy Isaacson via eli)
HDFS-3476. Correct the default used in TestDFSClientRetries.busyTest()
after HDFS-3462 (harsh)
HDFS-3478. Test quotas with Long.Max_Value. (Sujay Rau via eli)
HDFS-3630 Modify TestPersistBlocks to use both flush and hflush (sanjay)
HDFS-3768. Exception in TestJettyHelper is incorrect.
(Eli Reisman via jghoman)
HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. (harsh)
HDFS-2127. Add a test that ensure AccessControlExceptions contain
a full path. (Stephen Chu via eli)
HDFS-3995. Use DFSTestUtil.createFile() for file creation and
writing in test cases. (Jing Zhao via suresh)
HDFS-3735. NameNode WebUI should allow sorting live datanode list by fields
Block Pool Used, Block Pool Used(%) and Failed Volumes.
(Brahma Reddy Battula via suresh)
HDFS-4052. BlockManager#invalidateWork should print log outside the lock.
(Jing Zhao via suresh)
HDFS-3358. Specify explicitly that the NN UI status total is talking
of persistent objects on heap. (harsh)
HDFS-4904. Remove JournalService. (Arpit Agarwal via cnauroth)
HDFS-5041. Add the time of last heartbeat to dead server Web UI (Shinichi
Yamashita via brandonli)
HDFS-5721. sharedEditsImage in Namenode#initializeSharedEdits() should be
closed before method returns. (Ted Yu via junping_du)
HDFS-6228. comments typo fix for FsDatasetImpl.java (zhaoyunjiong via umamahesh)
HDFS-6246. Remove 'dfs.support.append' flag from trunk code. (umamahesh)
HDFS-6609. Use DirectorySnapshottableFeature to represent a snapshottable
directory. (Jing Zhao via wheat9)
HDFS-7591. hdfs classpath command should support same options as hadoop
classpath (Varun Saxena via Arpit Agarwal)
HDFS-7546. Document, and set an accepting default for
dfs.namenode.kerberos.principal.pattern (Harsh J via aw)
HDFS-316. Balancer should run for a configurable # of iterations (Xiaoyu
Yao via aw)
HDFS-7322. deprecate sbin/hadoop-daemon.sh (aw)
HDFS-7668. Convert site documentation from apt to markdown (Masatake
Iwasaki via aw)
HDFS-7460. Rewrite httpfs to use new shell framework (John Smith via aw)
HDFS-6353. Check and make checkpoint before stopping the NameNode. (jing9)
OPTIMIZATIONS
BUG FIXES
HADOOP-9635 Fix potential Stack Overflow in DomainSocket.c (V. Karthik Kumar
via cmccabe)
HDFS-2299. TestOfflineEditsViewer is failing on trunk. (Uma Maheswara Rao G
via atm)
HDFS-2310. TestBackupNode fails since HADOOP-7524 went in.
(Ivan Kelly via todd)
HDFS-2313. Rat excludes has a typo for excluding editsStored files. (atm)
HDFS-2314. MRV1 test compilation broken after HDFS-2197 (todd)
HDFS-46. Change default namespace quota of root directory from
Integer.MAX_VALUE to Long.MAX_VALUE. (Uma Maheswara Rao G via szetszwo)
HDFS-2373. Commands using WebHDFS and hftp print unnecessary debug
info on the console with security enabled. (Arpit Gupta via suresh)
HDFS-2776. Missing interface annotation on JournalSet.
(Brandon Li via jitendra)
HDFS-3037. TestMulitipleNNDataBlockScanner#testBlockScannerAfterRestart is
racy. (atm)
HDFS-3116. Typo in fetchdt error message. (AOE Takashi via atm)
HDFS-3126. Journal stream from Namenode to BackupNode needs to have
timeout. (Hari Mankude via suresh)
HDFS-3121. Add HDFS tests for HADOOP-8014 change. (John George via
suresh)
HDFS-3119. Overreplicated block is not deleted even after the replication
factor is reduced after sync follwed by closing that file. (Ashish Singhi
via umamahesh)
HDFS-3265. PowerPc Build error. (Kumar Ravi via mattf)
HDFS-2312. FSNamesystem javadoc incorrectly says its for DNs. (harsh)
HDFS-3163. TestHDFSCLI.testAll fails if the user name is not all lowercase.
(Brandon Li via atm)
HDFS-3462. TestDFSClientRetries.busyTest() should restore default
xceiver count in the config. (Madhukara Phatak via harsh)
HDFS-3550. Fix raid javadoc warnings. (Jason Lowe via daryn)
HDFS-3549. Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn)
HDFS-3614. Revert unused MiniDFSCluster constructor from HDFS-3049.
(acmurthy via eli)
HDFS-3792. Fix two findbugs introduced by HDFS-3695 (todd)
HDFS-3827. TestHASafeMode#assertSafemode method should be made static.
(Jing Zhao via suresh)
HDFS-3834. Remove unused static fields NAME, DESCRIPTION and Usage from
Command. (Jing Zhao via suresh)
HDFS-2434. TestNameNodeMetrics.testCorruptBlock fails intermittently.
(Jing Zhao via suresh)
HDFS-4067. TestUnderReplicatedBlocks intermittently fails due to
ReplicaAlreadyExistsException. (Jing Zhao via suresh)
HDFS-4115. TestHDFSCLI.testAll fails one test due to number format.
(Trevor Robinson via suresh)
HDFS-4105. The SPNEGO user for secondary namenode should use the web
keytab. (Arpit Gupta via jitendra)
HDFS-4003. test-patch should build the common native libs before
running hdfs tests. (Colin Patrick McCabe via eli)
HDFS-4260 Fix HDFS tests to set test dir to a valid HDFS path as opposed
to the local build path (Chri Nauroth via Sanjay)
HDFS-4310. fix test org.apache.hadoop.hdfs.server.datanode.
TestStartSecureDataNode (Ivan A. Veselovsky via atm)
HDFS-4275. MiniDFSCluster-based tests fail on Windows due to failure
to delete test namenode directory. (Chris Nauroth via suresh)
HDFS-4338. TestNameNodeMetrics#testCorruptBlock is flaky. (Andrew Wang via
atm)
HDFS-4391. TestDataTransferKeepalive fails when tests are executed in a
certain order. (Andrew Wang via atm)
HDFS-4757. Update FSDirectory#inodeMap when replacing an INodeDirectory
while setting quota. (Jing Zhao via szetszwo)
HDFS-4761. When resetting FSDirectory, the inodeMap should also be reset.
(Jing Zhao via szetszwo)
HDFS-4687. TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with
JDK7. (Andrew Wang via atm)
HDFS-3934. duplicative dfs_hosts entries handled wrong. (Colin Patrick
McCabe)
HDFS-4366. Block Replication Policy Implementation May Skip Higher-Priority
Blocks for Lower-Priority Blocks (Derek Dagit via kihwal)
HDFS-5719. FSImage#doRollback() should close prevState before return
(Ted Yu via brandonli)
HDFS-5794. Fix the inconsistency of layout version number of
ADD_DATANODE_AND_STORAGE_UUIDS between trunk and branch-2. (jing9)
HDFS-6694. TestPipelinesFailover.testPipelineRecoveryStress tests fail
intermittently with various symptoms - debugging patch. (Yongjun Zhang via
Arpit Agarwal)
HDFS-6893. crypto subcommand is not sorted properly in hdfs's hadoop_usage
(David Luo via aw)
HDFS-6981. Fix DN upgrade with layout version change. (Arpit Agarwal)
HDFS-7204. balancer doesn't run as a daemon (aw)
HDFS-7319. Remove dead link to HFTP documentation from index.xml.
(Masatake Iwasaki via wheat9)
HDFS-7407. Minor typo in privileged pid/out/log names (aw)
HDFS-7581. HDFS documentation needs updating post-shell rewrite (aw)
HADOOP-11484. hadoop-mapreduce-client-nativetask fails to build on ARM
AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe)
HDFS-7667. Various typos and improvements to HDFS Federation doc
(Charles Lamb via aw)
HDFS-3750. API docs don't include HDFS (Jolly Chen via aw)
HDFS-7320. The appearance of hadoop-hdfs-httpfs site docs is inconsistent
(Masatake Iwasaki via aw)
HDFS-7670. HDFS Quota guide has typos, incomplete command lines
(Brahma Reddy Battula via aw)
HDFS-7669. HDFS Design Doc references commands that no longer exist.
(Brahma Reddy Battula via aw)
HDFS-7755. httpfs shell code has hardcoded path to bash (Dmitry
Sivachenko via aw)
HDFS-7711. Fix various typos in ClusterSetup.md.
(Brahma Reddy Battula via wheat9)
HDFS-7791. dfs count -v should be added to quota documentation (Akira
AJISAKA via aw)
HDFS-7803. Wrong command mentioned in HDFSHighAvailabilityWithQJM
documentation (Arshad Mohammad via aw)
HDFS-4681. TestBlocksWithNotEnoughRacks#testCorruptBlockRereplicatedAcrossRacks
fails using IBM java (Ayappan via aw)
Release 2.8.0 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
HDFS-3918. EditLogTailer shouldn't log WARN when other node
is in standby mode (todd via harsh)
HDFS-4396. Add START_MSG/SHUTDOWN_MSG for ZKFC
(Liang Xie via harsh)
HDFS-7875. Improve log message when wrong value configured for
dfs.datanode.failed.volumes.tolerated.
(nijel via harsh)
HDFS-2360. Ugly stacktrace when quota exceeds. (harsh)
HDFS-7835. make initial sleeptime in locateFollowingBlock configurable for
DFSClient. (Zhihai Xu via Yongjun Zhang)
HDFS-7829. Code clean up for LocatedBlock. (Takanobu Asanuma via jing9)
HDFS-7854. Separate class DataStreamer out of DFSOutputStream. (Li Bo via
jing9)
HDFS-7713. Implement mkdirs in the HDFS Web UI. (Ravi Prakash via wheat9)
HDFS-7928. Scanning blocks from disk during rolling upgrade startup takes
a lot of time if disks are busy (Rushabh S Shah via kihwal)
HDFS-7990. IBR delete ack should not be delayed. (daryn via kihwal)
HDFS-8004. Use KeyProviderCryptoExtension#warmUpEncryptedKeys when creating
an encryption zone. (awang via asuresh)
HDFS-6263. Remove DRFA.MaxBackupIndex config from log4j.properties.
(Abhiraj Butala via aajisaka)
HDFS-6408. Remove redundant definitions in log4j.properties.
(Abhiraj Butala via aajisaka)
HDFS-7890. Improve information on Top users for metrics in
RollingWindowsManager and lower log level (J.Andreina via vinayakumarb)
HDFS-7645. Rolling upgrade is restoring blocks from trash multiple times.
(Vinayakumar B and Keisuke Ogiwara via Arpit Agarwal)
HDFS-7944. Minor cleanup of BlockPoolManager#getAllNamenodeThreads.
(Arpit Agarwal)
HDFS-7671. hdfs user guide should point to the common rack awareness doc.
(Kai Sasaki via aajisaka)
HDFS-8009. Signal congestion on the DataNode. (wheat9)
HDFS-7978. Add LOG.isDebugEnabled() guard for some LOG.debug(..).
(Walter Su via wang)
OPTIMIZATIONS
HDFS-8026. Trace FSOutputSummer#writeChecksumChunks rather than
DFSOutputStream#writeChunk (cmccabe)
BUG FIXES
HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs.
(Gautam Gopalakrishnan via harsh)
HDFS-5356. MiniDFSCluster should close all open FileSystems when shutdown()
(Rakesh R via vinayakumarb)
HDFS-7867. Update action param from "start" to "prepare" in rolling upgrade
javadoc (J.Andreina via vinayakumarb)
HDFS-3325. When configuring "dfs.namenode.safemode.threshold-pct" to a value
greater or equal to 1 there is mismatch in the UI report
(J.Andreina via vinayakumarb)
HDFS-8002. Website refers to /trash directory. (Brahma Reddy Battula via
aajisaka)
HDFS-7261. storageMap is accessed without synchronization in
DatanodeDescriptor#updateHeartbeatState() (Brahma Reddy Battula via Colin
P. McCabe)
HDFS-7997. The first non-existing xattr should also throw IOException.
(zhouyingchao via yliu)
HDFS-6945. BlockManager should remove a block from excessReplicateMap and
decrement ExcessBlocks metric when the block is removed. (aajisaka)
Release 2.7.0 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
HDFS-7278. Add a command that allows sysadmins to manually trigger full
block reports from a DN (cmccabe)
HDFS-6663. Admin command to track file and locations from block id.
(Chen He via kihwal)
HDFS-6982. nntop: top-like tool for name node users.
(Maysam Yabandeh via wang)
HDFS-7424. Add web UI for NFS gateway (brandonli)
HDFS-7449. Add metrics to NFS gateway (brandonli)
HDFS-3107. Introduce truncate. (Plamen Jeliazkov via shv)
HDFS-7056. Snapshot support for truncate. (Plamen Jeliazkov and shv)
HDFS-6673. Add delimited format support to PB OIV tool. (Eddy Xu via wang)
HDFS-7655. Expose truncate API for Web HDFS. (yliu)
HDFS-6133. Add a feature for replica pinning so that a pinned replica
will not be moved by Balancer/Mover. (zhaoyunjiong via szetszwo)
HDFS-3689. Add support for variable length block. (jing9)
HDFS-7584. Enable Quota Support for Storage Types (See breakdown of
tasks below)
HDFS-7656. Expose truncate API for HDFS httpfs. (yliu)
HDFS-6488. Support HDFS superuser in NFS gateway. (brandonli)
HDFS-7838. Expose truncate API for libhdfs. (yliu)
HDFS-6826. Plugin interface to enable delegation of HDFS authorization
assertions. (Arun Suresh via jitendra)
IMPROVEMENTS
HDFS-7752. Improve description for
"dfs.namenode.num.extra.edits.retained"
and "dfs.namenode.num.checkpoints.retained" properties on
hdfs-default.xml (Wellington Chevreuil via harsh)
HDFS-7055. Add tracing to DFSInputStream (cmccabe)
HDFS-7186. Document the "hadoop trace" command. (Masatake Iwasaki via Colin
P. McCabe)
HDFS-7202. Should be able to omit package name of SpanReceiver on "hadoop
trace -add" (iwasakims via cmccabe)
HDFS-7026. Introduce a string constant for "Failed to obtain user group
info...". (Yongjun Zhang via atm)
HDFS-7209. Populate EDEK cache when creating encryption zone. (Yi Liu via wang)
HDFS-6252. Phase out the old web UI in HDFS. (wheat9)
HDFS-7266. HDFS Peercache enabled check should not lock on object (awang
via cmccabe)
HDFS-7254. Add documentation for hot swaping DataNode drives (Lei Xu via
Colin P. McCabe)
HDFS-6877. Avoid calling checkDisk when an HDFS volume is removed during a
write. (Lei Xu via Colin P. McCabe)
HDFS-2486. Remove unnecessary priority level checks in
UnderReplicatedBlocks. (Uma Maheswara Rao G via szetszwo)
HDFS-6824. Additional user documentation for HDFS encryption. (wang)
HDFS-7165. Separate block metrics for files with replication count 1.
(Zhe Zhang via wang)
HDFS-7222. Expose DataNode network errors as a metric. (Charles Lamb via wang)
HDFS-7257. Add the time of last HA state transition to NN's /jmx page.
(Charles Lamb via wheat9)
HDFS-7223. Tracing span description of IPC client is too long (iwasakims
via cmccabe)
HDFS-7283. Bump DataNode OOM log from WARN to ERROR.
(Stephen Chu via wheat9)
HDFS-5928. Show namespace and namenode ID on NN dfshealth page.
(Siqi Li via wheat9)
HDFS-7213. processIncrementalBlockReport performance degradation.
(Eric Payne via kihwal)
HDFS-7280. Use netty 4 in WebImageViewer. (wheat9)
HDFS-3342. SocketTimeoutException in BlockSender.sendChunks could
have a better error message. (Yongjun Zhang via wang)
HDFS-7035. Make adding a new data directory to the DataNode an atomic
operation and improve error handling (Lei Xu via Colin P. McCabe)
HDFS-6917. Add an hdfs debug command to validate blocks, call recoverlease,
etc. (cmccabe)
HDFS-7356. Use DirectoryListing.hasMore() directly in nfs. (Li Lu via jing9)
HDFS-7357. FSNamesystem.checkFileProgress should log file path.
(Tsz Wo Nicholas Sze via wheat9)
HDFS-7335. Redundant checkOperation() in FSN.analyzeFileState().
(Milan Desai via shv)
HDFS-7333. Improve logging in Storage.tryLock(). (shv)
HDFS-7361. TestCheckpoint fails after change of log message related to
locking violation. (shv)
HDFS-7329. Improve logging when MiniDFSCluster fails to start.
(Byron Wong via shv)
HDFS-7336. Unused member DFSInputStream.buffersize. (Milan Desai via shv)
HDFS-7365. Remove hdfs.server.blockmanagement.MutableBlockCollection.
(Li Lu via wheat9)
HDFS-7381. Decouple the management of block id and gen stamps from
FSNamesystem. (wheat9)
HDFS-7375. Move FSClusterStats to o.a.h.h.hdfs.server.blockmanagement.
(wheat9)
HDFS-7386. Replace check "port number < 1024" with shared isPrivilegedPort
method. (Yongjun Zhang via cnauroth)
HDFS-7394. Log at INFO level, not WARN level, when InvalidToken is seen in
ShortCircuitCache (Keith Pak via Colin P. McCabe)
HDFS-7279. Use netty to implement DatanodeWebHdfsMethods. (wheat9)
HDFS-7404. Remove o.a.h.hdfs.server.datanode.web.resources.
(Li Lu via wheat9)
HDFS-7398. Reset cached thread-local FSEditLogOp's on every
FSEditLog#logEdit. (Gera Shegalov via cnauroth)
HDFS-7409. Allow dead nodes to finish decommissioning if all files are
fully replicated. (wang)
HDFS-7413. Some unit tests should use NameNodeProtocols instead of
FSNameSystem. (wheat9)
HDFS-7415. Move FSNameSystem.resolvePath() to FSDirectory. (wheat9)
HDFS-7420. Delegate permission checks to FSDirectory. (wheat9)
HDFS-7331. Add Datanode network counts to datanode jmx page. (Charles Lamb
via atm)
HDFS-7412. Move RetryCache to NameNodeRpcServer. (wheat9)
HDFS-7419. Improve error messages for DataNode hot swap drive feature (Lei
Xu via Colin P. Mccabe)
HDFS-7436. Consolidate implementation of concat(). (wheat9)
HDFS-7440. Consolidate snapshot related operations in a single class.
(wheat9)
HDFS-6803 Document DFSClient#DFSInputStream expectations reading and preading
in concurrent context. (stack via stevel)
HDFS-7310. Mover can give first priority to local DN if it has target storage type
available in local DN. (Vinayakumar B via umamahesh)
HDFS-7210. Avoid two separate RPC's namenode.append() and namenode.getFileInfo()
for an append call from DFSClient. (Vinayakumar B via umamahesh)
HDFS-7450. Consolidate the implementation of GetFileInfo(), GetListings() and
GetContentSummary() into a single class. (wheat9)
HDFS-7438. Consolidate the implementation of rename() into a single class.
(wheat9)
HDFS-7462. Consolidate implementation of mkdirs() into a single class.
(wheat9)
HDFS-7446. HDFS inotify should have the ability to determine what txid it
has read up to (cmccabe)
HDFS-6735. A minor optimization to avoid pread() be blocked by read()
inside the same DFSInputStream (Lars Hofhansl via stack)
HDFS-7458. Add description to the nfs ports in core-site.xml used by nfs
test to avoid confusion (Yongjun Zhang via brandonli)
HDFS-7468. Moving verify* functions to corresponding classes.
(Li Lu via wheat9)
HDFS-7478. Move org.apache.hadoop.hdfs.server.namenode.NNConf to
FSNamesystem. (Li Lu via wheat9)
HDFS-7474. Avoid resolving path in FSPermissionChecker. (jing9)
HDFS-7459. Consolidate cache-related implementation in FSNamesystem into
a single class. (wheat9)
HDFS-7476. Consolidate ACL-related operations to a single class.
(wheat9 via cnauroth)
HDFS-7384. 'getfacl' command and 'getAclStatus' output should be in sync.
(Vinayakumar B via cnauroth)
HDFS-7486. Consolidate XAttr-related implementation into a single class.
(wheat9)
HDFS-7498. Simplify the logic in INodesInPath. (jing9)
HDFS-7463. Simplify FSNamesystem#getBlockLocationsUpdateTimes. (wheat9)
HDFS-7509. Avoid resolving path multiple times. (jing9)
HDFS-7426. Change nntop JMX format to be a JSON blob. (wang)
HDFS-7513. HDFS inotify: add defaultBlockSize to CreateEvent (cmccabe)
HDFS-7536. Remove unused CryptoCodec in org.apache.hadoop.fs.Hdfs.
(Yi Liu via wheat9)
HDFS-7528. Consolidate symlink-related implementation into a single class.
(wheat9)
HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin
P. McCabe)
HDFS-7373. Clean up temporary files after fsimage transfer failures.
(kihwal)
HDFS-7543. Avoid path resolution when getting FileStatus for audit logs.
(wheat9)
HDFS-7530. Allow renaming of encryption zone roots. (Charles Lamb via wang)
HDFS-7484. Make FSDirectory#addINode take existing INodes as its parameter.
(jing9)
HADOOP-11032. Replace use of Guava's Stopwatch with Hadoop's StopWatch
(ozawa)
HADOOP-11470. Remove some uses of obsolete guava APIs from the hadoop
codebase. (Sangjin Lee via Colin P. McCabe)
HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9)
HDFS-7323. Move the get/setStoragePolicy commands out from dfsadmin.
(jing9 via yliu)
HDFS-7326: Add documentation for hdfs debug commands (Vijay Bhat via Colin
P. McCabe)
HDFS-7598. Remove dependency on old version of guava in
TestDFSClientCache#testEviction. (Sangjin Lee via Colin P. McCabe)
HDFS-7600. Refine hdfs admin classes to reuse common code. (jing9)
HDFS-2219. Change fsck to support fully qualified paths so that a
particular namenode in a federated cluster with multiple namenodes
can be specified in the path parameter. (szetszwo)
HDFS-7457. DatanodeID generates excessive garbage. (daryn via kihwal)
HDFS-7189. Add trace spans for DFSClient metadata operations. (Colin P.
McCabe via yliu)
HDFS-7573. Consolidate the implementation of delete() into a single class.
(wheat9)
HDFS-7640. Print NFS Client in the NFS log. (Brandon Li via wheat9)
HDFS-7623. Add htrace configuration properties to core-default.xml and
update user doc about how to enable htrace. (yliu)
HDFS-7224. Allow reuse of NN connections via webhdfs (Eric Payne via
kihwal)
HDFS-7683. Combine usages and percent stats in NameNode UI.
(Vinayakumar B via wheat9)
HDFS-7675. Remove unused member DFSClient#spanReceiverHost (cmccabe)
HDFS-7603. The background replication queue initialization may not let
others run (kihwal)
HDFS-7706. Switch BlockManager logging to use slf4j. (wang)
HDFS-5631. Change BlockMetadataHeader.readHeader(..), ChunkChecksum
class and constructor to public; and fix FsDatasetSpi to use generic type
instead of FsVolumeImpl. (David Powell and Joe Pallas via szetszwo)
HDFS-5782. Change BlockListAsLongs constructor to take Replica as parameter
type instead of concrete classes Block and ReplicaInfo. (David Powell
and Joe Pallas via szetszwo)
HDFS-7681. Change ReplicaInputStreams constructor to take InputStream(s)
instead of FileDescriptor(s). (Joe Pallas via szetszwo)
HDFS-7712. Switch blockStateChangeLog to use slf4j. (wang)
HDFS-7270. Add congestion signaling capability to DataNode write protocol.
(wheat9)
HDFS-7732. Fix the order of the parameters in DFSConfigKeys.
(Brahma Reddy Battula via aajisaka)
HDFS-7710. Remove dead code in BackupImage.java. (Xiaoyu Yao via aajisaka)
HDFS-7738. Revise the exception message for recover lease; add more truncate
tests such as truncate with HA setup, negative tests, truncate with other
operations and multiple truncates. (szetszwo)
HDFS-7743. Code cleanup of BlockInfo and rename BlockInfo to
BlockInfoContiguous. (jing9)
HDFS-7058. Tests for truncate CLI. (Dasha Boudnik via shv)
HDFS-7760. Document truncate for WebHDFS. (shv)
HDFS-7761. cleanup unnecssary code logic in LocatedBlock. (yliu)
HDFS-7703. Support favouredNodes for the append for new blocks
(vinayakumarb)
HDFS-7694. FSDataInputStream should support "unbuffer" (cmccabe)
HDFS-7684. The host:port settings of the daemons should be trimmed before
use. (Anu Engineer via aajisaka)
HDFS-7790. Do not create optional fields in DFSInputStream unless they are
needed (cmccabe)
HDFS-7430. Refactor the BlockScanner to use O(1) memory and use multiple
threads (cmccabe)
HDFS-7604. Track and display failed DataNode storage locations in NameNode.
(cnauroth)
HDFS-7797. Add audit log for setQuota operation (Rakesh R via umamahesh)
HDFS-4266. BKJM: Separate write and ack quorum (Rakesh R via umamahesh)
HDFS-7795. Show warning if not all favored nodes were chosen by namenode
(kihwal)
HDFS-7780. Update use of Iterator to Iterable in DataXceiverServer and
SnapshotDiffInfo. (Ray Chiang via aajisaka)
HDFS-7804. correct the haadmin command usage in #HDFSHighAvailabilityWithQJM.html
(Brahma Reddy Battula via umamahesh)
HDFS-7772. Document hdfs balancer -exclude/-include option in
HDFSCommands.html (Xiaoyu Yao via cnauroth)
HDFS-7773. Additional metrics in HDFS to be accessed via jmx.
(Anu Engineer via cnauroth)
HDFS-7740. Test truncate with DataNodes restarting. (yliu)
HDFS-7495. Remove updatePosition argument from DFSInputStream#getBlockAt()
(cmccabe)
HDFS-7537. Add "UNDER MIN REPL'D BLOCKS" count to fsck. (GAO Rui via
szetszwo)
HDFS-7832. Show 'Last Modified' in Namenode's 'Browse Filesystem'
(vinayakumarb)
HDFS-7819. Log WARN message for the blocks which are not in Block ID based
layout (Rakesh R via Colin P. McCabe)
HDFS-7308. Change the packet chunk size computation in DFSOutputStream in
order to enforce packet size <= 64kB. (Takuya Fukudome via szetszwo)
HDFS-7685. Document dfs.namenode.heartbeat.recheck-interval in
hdfs-default.xml. (Kai Sasaki via aajisaka)
HDFS-5853. Add "hadoop.user.group.metrics.percentiles.intervals" to
hdfs-default.xml. (aajisaka)
HDFS-7439. Add BlockOpResponseProto's message to the exception messages.
(Takanobu Asanuma via szetszwo)
HDFS-7789. DFSck should resolve the path to support cross-FS symlinks.
(gera)
HDFS-7535. Utilize Snapshot diff report for distcp. (jing9)
HDFS-1522. Combine two BLOCK_FILE_PREFIX constants into one.
(Dongming Liang via shv)
HDFS-7746. Add a test randomly mixing append, truncate and snapshot
operations. (szetszwo)
HADOOP-11648. Set DomainSocketWatcher thread name explicitly.
(Liang Xie via ozawa)
HDFS-7855. Separate class Packet from DFSOutputStream. (Li Bo bia jing9)
HDFS-7411. Change decommission logic to throttle by blocks rather than
nodes in each interval. (Andrew Wang via cdouglas)
HDFS-7898. Change TestAppendSnapshotTruncate to fail-fast.
(Tsz Wo Nicholas Sze via jing9)
HDFS-6806. HDFS Rolling upgrade document should mention the versions
available. (J.Andreina via aajisaka)
HDFS-7491. Add incremental blockreport latency to DN metrics.
(Ming Ma via cnauroth)
HDFS-7435. PB encoding of block reports is very inefficient.
(Daryn Sharp via kihwal)
HDFS-2605. Remove redundant "Release 0.21.1" section from CHANGES.txt.
(Allen Wittenauer via shv)
HDFS-7940. Add tracing to DFSClient#setQuotaByStorageType (Rakesh R via
Colin P. McCabe)
HDFS-7054. Make DFSOutputStream tracing more fine-grained (cmccabe)
HDFS.7849. Update documentation for enabling a new feature in rolling
upgrade ( J.Andreina via vinayakumarb )
HDFS-7962. Remove duplicated logs in BlockManager. (yliu)
HDFS-7917. Use file to replace data dirs in test to simulate a disk failure.
(Lei (Eddy) Xu via cnauroth)
HDFS-7956. Improve logging for DatanodeRegistration.
(Plamen Jeliazkov via shv)
HDFS-7976. Update NFS user guide for mount option "sync" to minimize or
avoid reordered writes. (brandonli)
HDFS-7410. Support CreateFlags with append() to support hsync() for
appending streams (Vinayakumar B via Colin P. McCabe)
HDFS-7742. Favoring decommissioning node for replication can cause a block
to stay underreplicated for long periods (Nathan Roberts via kihwal)
OPTIMIZATIONS
HDFS-7454. Reduce memory footprint for AclEntries in NameNode.
(Vinayakumar B via wheat9)
HDFS-7615. Remove longReadLock (kihwal)
BUG FIXES
HDFS-6741. Improve permission denied message when
FSPermissionChecker#checkOwner fails (Stephen Chu and harsh).
HDFS-6538. Comment format error in ShortCircuitRegistry javadoc.
(David Luo via harsh).
HDFS-7194. Fix findbugs "inefficient new String constructor" warning in
DFSClient#PATH (yzhang via cmccabe)
HDFS-7198. Fix or suppress findbugs "unchecked conversion" warning in
DFSClient#getPathTraceScope (cmccabe)
HDFS-6657. Remove link to 'Legacy UI' in trunk's Namenode UI.
(Vinayakumar B via wheat)
HDFS-7201. Fix typos in hdfs-default.xml. (Dawson Choong via wheat9)
HDFS-7190. Bad use of Preconditions in startFileInternal().
(Dawson Choong via wheat9)
HDFS-7242. Code improvement for FSN#checkUnreadableBySuperuser.
(Yi Liu via vinayakumarb)
HDFS-7252. small refinement to the use of isInAnEZ in FSNamesystem.
(Yi Liu via vinayakumarb)
HDFS-7277. Remove explicit dependency on netty 3.2 in BKJournal. (wheat9)
HDFS-7232. Populate hostname in httpfs audit log (Zoran Dimitrijevic
via aw)
HDFS-7258. CacheReplicationMonitor rescan schedule log should use DEBUG
level instead of INFO level. (Xiaoyu Yao via wheat9)
HDFS-7282. Fix intermittent TestShortCircuitCache and
TestBlockReaderFactory failures resulting from TemporarySocketDirectory GC.
(Jinghui Wang via Colin Patrick McCabe)
HDFS-7301. TestMissingBlocksAlert should use MXBeans instead of old web UI.
(Zhe Zhang via wheat9)
HDFS-7235. DataNode#transferBlock should report blocks that don't exist
using reportBadBlock (yzhang via cmccabe)
HDFS-7263. Snapshot read can reveal future bytes for appended files.
(Tao Luo via shv)
HDFS-7315. DFSTestUtil.readFileBuffer opens extra FSDataInputStream.
(Plamen Jeliazkov via wheat9)
HDFS-7324. haadmin command usage prints incorrect command name.
(Brahma Reddy Battula via suresh)
HDFS-7366. BlockInfo should take replication as an short in the constructor.
(Li Lu via wheat9)
HDFS-7389. Named user ACL cannot stop the user from accessing the FS entity.
(Vinayakumar B via cnauroth)
HDFS-6938. Cleanup javac warnings in FSNamesystem (Charles Lamb via wheat9)
HDFS-7358. Clients may get stuck waiting when using ByteArrayManager.
(szetszwo)
HDFS-7395. BlockIdManager#clear() bails out when resetting the
GenerationStampV1Limit. (wheat9)
HDFS-7399. Lack of synchronization in
DFSOutputStream#Packet#getLastByteOffsetBlock() (vinayakumarb)
HDFS-7146. NFS ID/Group lookup requires SSSD enumeration on the server
(Yongjun Zhang via brandonli)
HDFS-7406. SimpleHttpProxyHandler puts incorrect "Connection: Close"
header. (wheat9)
HDFS-7225. Remove stale block invalidation work when DN re-registers with
different UUID. (Zhe Zhang and Andrew Wang)
HDFS-7374. Allow decommissioning of dead DataNodes. (Zhe Zhang)
HDFS-7403. Inaccurate javadoc of BlockUCState#COMPLETE state. (
Yongjun Zhang via yliu)
HDFS-7303. NN UI fails to distinguish datanodes on the same host.
(Benoy Antony via wheat9)
HDFS-7097. Allow block reports to be processed during checkpointing on
standby name node. (kihwal via wang)
HDFS-7444. convertToBlockUnderConstruction should preserve BlockCollection.
(wheat9)
HDFS-7448 TestBookKeeperHACheckpoints fails in trunk build
(Akira Ajisaka via stevel)
HDFS-7472. Fix typo in message of ReplicaNotFoundException.
(Masatake Iwasaki via wheat9)
HDFS-7473. Document setting dfs.namenode.fs-limits.max-directory-items to 0
is invalid. (Akira AJISAKA via cnauroth)
HDFS-7481. Add ACL indicator to the "Permission Denied" exception.
(vinayakumarb)
HDFS-7502. Fix findbugs warning in hdfs-nfs project.
(Brandon Li via wheat9)
HDFS-5578. [JDK8] Fix Javadoc errors caused by incorrect or illegal tags
in doc comments. (Andrew Purtell via wheat9)
HDFS-7475. Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved
deterministic. (Xiaoyu Yao via Arpit Agarwal)
HDFS-7515. Fix new findbugs warnings in hadoop-hdfs. (wheat9)
HDFS-7497. Inconsistent report of decommissioning DataNodes between
dfsadmin and NameNode webui. (Yongjun Zhang via wang)
HDFS-7517. Remove redundant non-null checks in FSNamesystem#
getBlockLocations. (wheat9)
HDFS-7514. TestTextCommand fails on Windows. (Arpit Agarwal)