Closes #4. Use MiniDFSCluster / SimulatedFSDataset to be more efficient #5
Conversation
xkrogen
force-pushed
the
simulated-datanodes
branch
3 times, most recently
from
071b036
to
67fe568
Compare
|
|
||
| if (oStream.getLength() != finalSize) { | ||
| DataNode.LOG.warn("Size passed to finalize (" + finalSize + | ||
| ")does not match what was written:" + oStream.getLength()); |
| private long bytesAcked; | ||
| private long bytesRcvd; | ||
| private boolean pinned = false; | ||
| BInfo(String bpid, Block b, boolean forWriting) throws IOException { |
There was a problem hiding this comment.
nit: add a newline before the method?
| System.out.printf("Error injecting blocks into DataNode %d for block pool %s: %s%n", dnIndex, bpid, | ||
| ExceptionUtils.getFullStackTrace(ioe)); | ||
| } | ||
| System.out.printf("Injected %d blocks into DataNode %d for block pool %s%n", cnt, dnIndex, bpid); |
There was a problem hiding this comment.
Looks like even if mc.injectBlocks throws exception, this line still gets printed out, after the error message. how about moving this line to right after mc.injectBlocks
There was a problem hiding this comment.
Definitely, good catch.
|
|
||
| private final List<SimulatedStorage> storages; | ||
| private final String datanodeUuid; | ||
| private final DataNode datanode; |
There was a problem hiding this comment.
nit: maybe cleaner move these to the beginning of class declaration
|
|
||
| @Override // FSDatasetMBean | ||
| public String[] getFailedStorageLocations() { | ||
| return null; |
There was a problem hiding this comment.
to avoid any potential NPE, maybe better to throw unsupported operation exception, or return an empty array?
…ch uses MiniDFSCluster to launch many DataNodes in the same JVM. Additionally, SimulatedFSDataset is used to avoid the requirement of writing sparse files to the disk - all block metadata is now stored in-memory.
xkrogen
force-pushed
the
simulated-datanodes
branch
from
67fe568
to
5bddd65
Compare
|
Thanks for the review @chen-liang ! Regarding the suggestions about changes to SimulatedMultiStorageFSDataset, this class is primarily just copied directly from Hadoop, I would rather make as few changes as possible to (a) avoid rewriting code that isn't ours, and (b) stay closer to the Hadoop version to increase the ease of porting in changes from upstream if necessary. Let me know what you think. |
|
Thanks for the update, @xkrogen ! +1 |
|
Thanks Chen! |
DataNodes are launched via SimulatedDataNodes, which uses MiniDFSCluster to launch many DataNodes in the same JVM. Additionally, SimulatedFSDataset is used to avoid the requirement of writing sparse files to the disk - all block metadata is now stored in-memory.