-
Notifications
You must be signed in to change notification settings - Fork 199
java 8 stream support #250
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This is fantastic! Thank you very much for this addition 👍 I was planning to make Java 8 as minimum java version required for the upcoming v5 ( see #197 ) but many users asked me to keep Easy Batch Java 7 compatible ( I took into account their request: 6b0eaf5). Hence, the project is compiled with Java 7 for now (which makes the build to be in error with your PR, but nothing to do with your code which is fine) I know we can compile with Java 8 and specify the target to be Java 7, but this does not seem to work and I had some issues on another project for the same problem. Overriding java version in the added module as you did is the way to go, but we should be able to compile all modules with Java 7 except the new one with Java 8. Probably you have managed to do it? any help is welcome! |
Done. I use this framework in production (and love it). I probably make other small features in the future :) Good luck for all your projects ! |
Thank you for the quick update! Nice trick 😉 I'm only thinking about how to release this new module since maven release plugin will use java 7.. But no worries, I'll find a way to release it. Probably I'll move all extensions to a separate repo and release them separately (each with its required java version). Let me merge this addition for now. A couple of notes though:
I'm really glad to hear this 😄
You are welcome |
A java 8 stream is not necessarily "in-memory", The Files.walk(Path) (java.nio) is an example. But its a good compromise.
I agree to keep one directory reader, in theory, but ! :)
I recommend (at least) to set as @deprecated for the v5 : FileRecordReader, FileRecord (which belong to java 7), and keep PathRecordReader. |
I think the new module should only contain the StreamRecordReader.
Sure, sorry if I wasn't clear enough. What I mean is that the data source (the stream itself) is in memory. Like an iterable of files: files are on disk, but the data source (ie the iterable object is in memory).
That's why I'm suggesting to refactor it using a lazy API (be it Files.walkFileTree for Java 7 or Files.walk for Java 8). But since v5 should still Java 7 compatible and this change can be done in the core module, we can use Files.walkFileTree
I agree with you, we need to deprecate usage of java.io.File. But, in the end, we are reading "files" from a "directory". So IMHO, FileRecordReader is a good name (regardless of the technical java API representing "files": java.io.File or java.nio.Path). Do you see my point? What do you think of:
This way, we take the best of both readers and merge them (this can be done in the core module using Java 7 compatible API, namely Files.walkFileTree). I don't want to bother you with more updates of the PR, If you agree, I can merge it and do the changes. Kind regards |
Ok, i didn't understand that the first time. I agree, you have a better compromise 😄 Good night ! |
Great! I'll merge your PR asap. Thank you for this addition 👍 |
…ill be set to default value)
…ng support of Java 8 Streams)
I've merged your changes and added you to the contributors list. I really appreciate your help! Thanks. You know, the
Probably in combination with a new Easy Batch RegularFileRecordFilter having this logic
I'm not sure it will load the content of all files in memory, but simply pointers to file descriptors. May be even with only pointers in the list, 1M+ would become a performance issue. Did you have performance issues with FileRecordReader ? |
The real problem with the current getfile method is to visit each file physically on the hdd (descriptors or not) to create the list. its not a memory problem, its a delay-before-start one. walkfiletree send immediatly the first file and continue. |
Ok I see, I'll refactor this ugly method asap! |
Thank you for the contributor page 😄 |
Hello,
These classes add support for java 8 streams and path streams.
I've read the framework is compatible with java 7, so i set java.version to 1.8 only for the added module. Build and tests OK for 1.8.
I hope that it will be useful.
Best regards.
Charles Fleury