These classes add support for java 8 streams and path streams.
I've read the framework is compatible with java 7, so i set java.version to 1.8 only for the added module. Build and tests OK for 1.8.
I hope that it will be useful.
This is fantastic! Thank you very much for this addition
I was planning to make Java 8 as minimum java version required for the upcoming v5 ( see #197 ) but many users asked me to keep Easy Batch Java 7 compatible ( I took into account their request: 6b0eaf5).
Hence, the project is compiled with Java 7 for now (which makes the build to be in error with your PR, but nothing to do with your code which is fine)
I know we can compile with Java 8 and specify the target to be Java 7, but this does not seem to work and I had some issues on another project for the same problem.
Overriding java version in the added module as you did is the way to go, but we should be able to compile all modules with Java 7 except the new one with Java 8. Probably you have managed to do it? any help is welcome!
Thank you for the quick update! Nice trick
Let me merge this addition for now.
A couple of notes though:
I'm really glad to hear this
You are welcome
A java 8 stream is not necessarily "in-memory", The Files.walk(Path) (java.nio) is an example. But its a good compromise.
I agree to keep one directory reader, in theory, but ! :)
I recommend (at least) to set as @deprecated for the v5 : FileRecordReader, FileRecord (which belong to java 7), and keep PathRecordReader.
I think the new module should only contain the StreamRecordReader.
Sure, sorry if I wasn't clear enough. What I mean is that the data source (the stream itself) is in memory. Like an iterable of files: files are on disk, but the data source (ie the iterable object is in memory).
That's why I'm suggesting to refactor it using a lazy API (be it Files.walkFileTree for Java 7 or Files.walk for Java 8). But since v5 should still Java 7 compatible and this change can be done in the core module, we can use Files.walkFileTree
I agree with you, we need to deprecate usage of java.io.File. But, in the end, we are reading "files" from a "directory". So IMHO, FileRecordReader is a good name (regardless of the technical java API representing "files": java.io.File or java.nio.Path). Do you see my point?
What do you think of:
This way, we take the best of both readers and merge them (this can be done in the core module using Java 7 compatible API, namely Files.walkFileTree).
I don't want to bother you with more updates of the PR, If you agree, I can merge it and do the changes.
I've merged your changes and added you to the contributors list. I really appreciate your help! Thanks.
You know, the
Probably in combination with a new Easy Batch RegularFileRecordFilter having this logic
I'm not sure it will load the content of all files in memory, but simply pointers to file descriptors. May be even with only pointers in the list, 1M+ would become a performance issue. Did you have performance issues with FileRecordReader ?