Add iterable JDBC template to process large amount of data [SPR-13900]

[spring-projects-issues]

Michael Osipov opened SPR-13900 and commented

I have a usecase where I need to write a table with millions of rows to an output stream via HTTP. Those data cannot be reasonably fetched with the current JdbcTemplate. One would need an abstracted iterator which does all the nice stuff JdbcTemplate but does not spill memory.

Fortunately, there is springjdbc-iterable but the author is not maintaining it anymore. I have requested to donate this code to Spring core to have this by default.

---

Affects: 4.2.4

Issue Links:

• Add Optional Support to JdbcTemplate [SPR-12662] #17262 Add Optional Support to JdbcTemplate
• Customize PreparedStatementCreatorFactory in getPreparedStatementCreator [SPR-16050] #20599 Customize PreparedStatementCreatorFactory in getPreparedStatementCreator

1 votes, 2 watchers

[alpet]

What is the status on this one ? Is been two years have the plans changed ?

[denniseffing]

Could this be prioritized higher for future releases?

We realized last week that Spring Data JDBC is not capable of streaming large result sets and we are encountering the first memory issues because of this.

[jhoeller]

Talking to the Spring Data team, we're aiming for an arrangement in the Spring Framework 5.3 timeframe now, with iterable methods on JdbcTemplate and NamedParameterJdbcTemplate built on a CloseableIterator or on Stream, to be picked up by Spring Data JDBC right away.

[jhoeller]

A first cut of this is ready now, introducing queryForStream methods on JdbcTemplate and NamedParameterJdbcTemplate, all with a RowMapper argument and a closeable Stream result (ideally to be used with a try-with-resources clause). Feel free to give it a try with the upcoming 5.3 snapshot, or with the 5.3 M1 release in June...