Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Java 8 Stream in repository finder methods [DATAJPA-677] #1054

Closed
spring-projects-issues opened this issue Feb 20, 2015 · 4 comments

Comments

@spring-projects-issues
Copy link

@spring-projects-issues spring-projects-issues commented Feb 20, 2015

Christoph Grimmer-Dietrich opened DATAJPA-677 and commented

For a current project I need to read large amounts of entities from a repository, transform these into different ones and store those in another repository. Since a query might return several thousand entities fetching the full list is out of the question. So currently we fetch Slice after Slice inside a Spliterator and feed them into a Stream.
Of course this is just a workaround. What we actually would like to have is to get a Stream directly from the repository so we would not have to handle the chunking manually like this:

Stream<MyEntity> findByDateModifiedBetween(Timestamp start, Timestamp end);

Issue Links:

  • DATACMNS-650 Provide a CloseableIterator abstraction as a foundation for streaming of results

  • DATACMNS-890 Support infinite paging

Referenced from: pull request #136

2 votes, 6 watchers

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Feb 21, 2015

Oliver Drotbohm commented

As much as we'd like to see this feature for JPA, too, it's very unlikely to happen for one simple reason: JPA is List based by default, so we can't optimize on that internally but basically use your workaround, too. This however means that we might run into situations where entities inserted into or deleted from already read slices might cause the resulting Stream not getting elements skipped or appear multiple times in the stream. In short: we cannot generate a Stream and guarantee the user gets what she expects.

For these high-volume data access operations, would using JDBC be an option for you? You could basically use JdbcTemplate alongside a RowCallbackHandler to pipe data to a different store while reading the incoming rows

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Feb 25, 2015

Christoph Grimmer-Dietrich commented

Hello Oliver,
thanks for the response. I will discuss with the team about using JDBC directly. After Thomas' talk about XD at the JUGH we actually might consider this an option, too.

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Mar 3, 2015

Thomas Darimont commented

Please revise

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented May 6, 2016

Andrei Ivanov commented

Hi Oliver Drotbohm,
Perhaps you might be interested in the discussion of https://hibernate.atlassian.net/browse/HHH-9340 ?
I think you might be able to contribute better than me :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants