/
CHANGES-HDFS-EC-7285.txt
executable file
·237 lines (153 loc) · 9.1 KB
/
CHANGES-HDFS-EC-7285.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
BREAKDOWN OF HDFS-7285 SUBTASKS AND RELATED JIRAS
HDFS-7347. Configurable erasure coding policy for individual files and
directories ( Zhe Zhang via vinayakumarb )
HDFS-7339. Representing striped block groups in NameNode with hierarchical
naming protocol ( Zhe Zhang )
HDFS-7652. Process block reports for erasure coded blocks (Zhe Zhang)
HDFS-7716. Erasure Coding: extend BlockInfo to handle EC info (Jing Zhao)
HDFS-7749. Erasure Coding: Add striped block support in INodeFile (Jing Zhao)
HDFS-7837. Erasure Coding: allocate and persist striped blocks in NameNode
(Jing Zhao via Zhe Zhang)
HDFS-7872. Erasure Coding: INodeFile.dumpTreeRecursively() supports to print
striped blocks (Takuya Fukudome via jing9)
HDFS-7853. Erasure coding: extend LocatedBlocks to support reading from
striped files (Jing Zhao)
HDFS-7826. Erasure Coding: Update INodeFile quota computation for striped
blocks ( Kai Sasaki via jing9 )
HDFS-7912. Erasure Coding: track BlockInfo instead of Block in
UnderReplicatedBlocks and PendingReplicationBlocks (Jing Zhao)
HDFS-7369. Erasure coding: distribute recovery work for striped blocks to
DataNode (Zhe Zhang)
HDFS-7864. Erasure Coding: Update safemode calculation for striped blocks
(GAO Rui via jing9)
HDFS-7827. Erasure Coding: support striped blocks in non-protobuf fsimage
( Hui Zheng via jing9 )
HDFS-7616. Add a test for BlockGroup support in FSImage.
(Takuya Fukudome via szetszwo)
HDFS-7907. Erasure Coding: track invalid, corrupt, and under-recovery striped
blocks in NameNode (Jing Zhao)
HDFS-8005. Erasure Coding: simplify striped block recovery work computation
and add tests (Jing Zhao)
HDFS-7617. Add unit tests for editlog transactions for EC
(Hui Zheng via Zhe Zhang)
HDFS-7839. Erasure coding: implement facilities in NameNode to create and
manage EC zones (Zhe Zhang)
HDFS-7969. Erasure coding: NameNode support for lease recovery of striped
block groups. (Zhe Zhang)
HDFS-7782. Erasure coding: pread from files in striped layout.
(Zhe Zhang and Jing Zhao via Zhe Zhang)
HDFS-8023. Erasure Coding: retrieve eraure coding schema for a file from
NameNode (vinayakumarb)
HDFS-8074. Define a system-wide default EC schema. (Kai Zheng)
HDFS-8077. Erasure coding: fix bugs in EC zone and symlinks.
(Jing Zhao and Zhe Zhang via Jing Zhao)
HDFS-8104. Make hard-coded values consistent with the system default schema first before remove them. (Kai Zheng)
HDFS-7889. Subclass DFSOutputStream to support writing striping layout files. (Li Bo via Kai Zheng)
HDFS-8090. Erasure Coding: Add RPC to client-namenode to list all
ECSchemas loaded in Namenode. (vinayakumarb)
HDFS-8122. Erasure Coding: Support specifying ECSchema during creation of ECZone.
(Vinayakumar B via Zhe Zhang)
HDFS-8114. Erasure coding: Add auditlog FSNamesystem#createErasureCodingZone if this
operation fails. (Rakesh R via Zhe Zhang)
HDFS-8123. Erasure Coding: Better to move EC related proto messages to a
separate erasurecoding proto file (Rakesh R via vinayakumarb)
HDFS-7349. Support DFS command for the EC encoding (vinayakumarb)
HDFS-8120. Erasure coding: created util class to analyze striped block groups.
(Contributed by Zhe Zhang and Li Bo via Jing Zhao)
HDFS-7994. Detect if resevered EC Block ID is already used during namenode
startup. (Hui Zheng via szetszwo)
HDFS-8167. BlockManager.addBlockCollectionWithCheck should check if the block is a striped block. (Hui Zheng via zhz).
HDFS-8166. DFSStripedOutputStream should not create empty blocks. (Jing Zhao)
HDFS-7937. Erasure Coding: INodeFile quota computation unit tests.
(Kai Sasaki via Jing Zhao)
HDFS-8145. Fix the editlog corruption exposed by failed TestAddStripedBlocks.
(Jing Zhao)
HDFS-8146. Protobuf changes for BlockECRecoveryCommand and its fields for
making it ready for transfer to DN (Uma Maheswara Rao G via vinayakumarb)
HDFS-8181. createErasureCodingZone sets retryCache state as false always
(Uma Maheswara Rao G via vinayakumarb)
HDFS-8190. StripedBlockUtil.getInternalBlockLength may have overflow error.
(szetszwo)
HDFS-8216. TestDFSStripedOutputStream should use BlockReaderTestUtil to
create BlockReader. (szetszwo via Zhe Zhang)
HDFS-8212. DistributedFileSystem.createErasureCodingZone should pass schema
in FileSystemLinkResolver. (szetszwo via Zhe Zhang)
HDFS-8024. Erasure Coding: ECworker frame, basics, bootstraping and configuration.
(umamahesh)
HDFS-8156. Add/implement necessary APIs even we just have the system default
schema. (Kai Zheng via Zhe Zhang)
HDFS-8136. Client gets and uses EC schema when reads and writes a stripping
file. (Kai Sasaki via Kai Zheng)
HDFS-8233. Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last
stripe is at the block group boundary. (jing9)
HDFS-8223. Should calculate checksum for parity blocks in DFSStripedOutputStream.
(Yi Liu via jing9)
HDFS-8228. Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause
block id conflicts (Jing Zhao via Zhe Zhang)
HDFS-8033. Erasure coding: stateful (non-positional) read from files in
striped layout (Zhe Zhang)
HDFS-8230. Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY
commands from standbynode if any (vinayakumarb)
HDFS-8189. ClientProtocol#createErasureCodingZone API was wrongly annotated
as Idempotent (vinayakumarb)
HDFS-8235. Erasure Coding: Create DFSStripedInputStream in DFSClient#open.
(Kai Sasaki via jing9)
HDFS-8272. Erasure Coding: simplify the retry logic in DFSStripedInputStream
(stateful read). (Jing Zhao via Zhe Zhang)
HDFS-8282. Erasure coding: move striped reading logic to StripedBlockUtil.
(Zhe Zhang)
HDFS-8183. Erasure Coding: Improve DFSStripedOutputStream closing of
datastreamer threads. (Rakesh R via Zhe Zhang)
HDFS-8308. Erasure Coding: NameNode may get blocked in waitForLoadingFSImage()
when loading editlog. (jing9)
HDFS-7949. WebImageViewer need support file size calculation with striped
blocks. (Rakesh R via Zhe Zhang)
HDFS-8316. Erasure coding: refactor EC constants to be consistent with HDFS-8249.
(Zhe Zhang via jing9)
HDFS-8281. Erasure Coding: implement parallel stateful reading for striped layout.
(jing9)
HDFS-8137. Send the EC schema to DataNode via EC encoding/recovering command(umamahesh)
HDFS-8242. Erasure Coding: XML based end-to-end test for ECCli commands
(Rakesh R via vinayakumarb)
HDFS-8324. Add trace info to DFSClient#getErasureCodingZoneInfo(..) (vinayakumarb via
umamahesh)
HDFS-7672. Handle write failure for stripping blocks and refactor the
existing code in DFSStripedOutputStream and StripedDataStreamer. (szetszwo)
HDFS-7348. Erasure Coding: DataNode reconstruct striped blocks.
(Yi Liu via Zhe Zhang)
HADOOP-11921. Enhance tests for erasure coders. (Kai Zheng)
HDFS-8334. Erasure coding: rename DFSStripedInputStream related test
classes. (Zhe Zhang)
HDFS-8129. Erasure Coding: Maintain consistent naming for Erasure Coding related classes - EC/ErasureCoding
(umamahesh)
HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
(Yi Liu via jing9)
HDFS-8289. Erasure Coding: add ECSchema to HdfsFileStatus. (Yong Zhang via
jing9)
HDFS-8355. Erasure Coding: Refactor BlockInfo and BlockInfoUnderConstruction.
(Tsz Wo Nicholas Sze via jing9)
HDFS-7678. Erasure coding: DFSInputStream with decode functionality (pread).
(Zhe Zhang)
HDFS-8372. Erasure coding: compute storage type quotas for striped files,
to be consistent with HDFS-8327. (Zhe Zhang via jing9)
HDFS-8368. Erasure Coding: DFS opening a non-existent file need to be
handled properly (Rakesh R via zhz)
HDFS-8363. Erasure Coding: DFSStripedInputStream#seekToNewSource. (yliu)
HDFS-8195. Erasure coding: Fix file quota change when we complete/commit
the striped blocks. (Takuya Fukudome via zhz)
HDFS-8364. Erasure coding: fix some minor bugs in EC CLI
(Walter Su via vinayakumarb)
HDFS-8391. NN should consider current EC tasks handling count from DN while
assigning new tasks. (umamahesh)
HDFS-8367. BlockInfoStriped uses EC schema. (Kai Sasaki via Kai Zheng)
HDFS-8352. Erasure Coding: test webhdfs read write stripe file. (waltersu4549)
HDFS-8417. Erasure Coding: Pread failed to read data starting from not-first stripe.
(Walter Su via jing9)
HDFS-8418. Fix the isNeededReplication calculation for Striped block in NN.
(Yi Liu via jing9)
HDFS-8320. Erasure coding: consolidate striping-related terminologies. (zhz)
HDFS-8366. Erasure Coding: Make the timeout parameter of polling blocking queue
configurable in DFSStripedOutputStream. (Li Bo)
HDFS-8378. Erasure Coding: Few improvements for the erasure coding worker.
(Rakesh R via waltersu4549)
HDFS-8375. Add cellSize as an XAttr to ECZone. ( Vinayakumar B via zhz).