APEXMALHAR-2314 Improper functioning in partitioning for sequentialFileRead for FSRecord

[deepak-narkhede]

Fix the StreamCodec for FSRecordReader, initially it was hashcode of blockId's mostly always unique.
Hence unable to satisfy the sequentialFileRead property. Now the StreamCodec is modified to work
with hashcode of filePath. So all blocks related to a file would be partitioned on same operator.

Tested with recordReader and verified for sequentialFileRead that all blocks related to a file are partitioned to single operator.

[deepak-narkhede]

@yogidevendra: Could you please review ?

[yogidevendra]

Let us reuse SequentialFileBlockMetadataCodec from FSInputModule instead of defining separate one.

[deepak-narkhede]

@yogidevendra : Thanks for suggestion. Makes sense. Only thing that worried me about creating unnecessary dependency of FSInputModule's StreamCodec on FSRecordReaderModule. So if someone changes FSInput StreamCodec he must consider same in FSRecoredReaderModule.

[yogidevendra]

This dependency should be OK. FSRecordReader is closely aligned with FSInputModule.

[deepak-narkhede]

Incorporated Yogi's suggestion.

[yogidevendra]

Could you please shorten the commit message?
Keep details to 1-2 lines per change.

Additional details can be captured in the JIRA description. Git log showing long messages would be hard to read.

[yogidevendra]

Also please fix the spelling mistake in the JIRA title and commit message.
sequencialFileRead =>sequentialFileRead.
Check for spellings in the code, javadoc as well.

[deepak-narkhede]

Done. Thanks !

On Wed, Oct 26, 2016 at 11:15 AM, yogidevendra notifications@github.com
wrote:

Also please fix the spelling mistake in the JIRA title and commit message.
sequencialFileRead =>sequentialFileRead.
Check for spellings in the code, javadoc as well.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#468 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASvTtiq6elsdBijUCNmb4Zeci-sfmslBks5q3uj8gaJpZM4Keh-S
.

Thanks & Regards

Deepak Narkhede

[tushargosavi]

[ERROR] src/main/java/org/apache/apex/malhar/lib/fs/FSRecordReaderModule.java:32,8 UnusedImports: Unused import - com.datatorrent.common.partitioner.StatelessPartitioner.
[ERROR] src/main/java/org/apache/apex/malhar/lib/fs/FSRecordReaderModule.java:33,8 UnusedImports: Unused import - com.datatorrent.lib.io.block.FSSliceReader.

please fix above checkstyle error.

[deepak-narkhede]

Sorry my bad. After rebase it occurred. Will resolve the conflicts.

On Nov 7, 2016 3:33 PM, "Tushar R. Gosavi" notifications@github.com wrote:

[ERROR] src/main/java/org/apache/apex/malhar/lib/fs/
FSRecordReaderModule.java:32,8 http://imports UnusedImports: Unused
import - com.datatorrent.common.partitioner.StatelessPartitioner.
[ERROR] src/main/java/org/apache/apex/malhar/lib/fs/
FSRecordReaderModule.java:33,8 http://imports UnusedImports: Unused
import - com.datatorrent.lib.io.block.FSSliceReader.

please fix above checkstyle error.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#468 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASvTtqvJp4rc28gSu76eARinF0EAytHpks5q7vdpgaJpZM4Keh-S
.