Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Base.find(String query, Object... params) is fetching the whole resultset instead of streaming it #1101

alniks opened this issue Apr 16, 2021 · 5 comments


Copy link

alniks commented Apr 16, 2021

the code is passing 0 to the method which sets fetch size:

find(RowProcessor.ResultSetType.FORWARD_ONLY, RowProcessor.ResultSetConcur.READ_ONLY, 0, query, params)

according to docs 0 is ignored:

which leads to fetching whole resultset into memory.

I would set the default value to at least 10 to save on network calls

Copy link

@alniks this is not a bug. This is a way to get an infinite number of records from a database by processing the results using RowProcessor one record at the time. If you are somehow accumulating these records and are running out of heap space, this API is not for this case. The whole idea here is not to have a limit, but rather to use a "cursor" to be able to process infinite number of records.

@ipolevoy ipolevoy removed the bug label Apr 20, 2021
Copy link

alniks commented Apr 21, 2021

@ipolevoy you are wrong: "0" is ignored by the driver and means - fetch the whole result set into memory. If you need "one record at the time" you have to pass 1. Here is the stack strace:

at java.lang.OutOfMemoryError.()V (
at[B (
at (
at ( <= here

Line 4 from the top.

Basically Base.find(RowProcessor.ResultSetType.FORWARD_ONLY, RowProcessor.ResultSetConcur.READ_ONLY, 0, query, params) is equal to Base.findAll

Limit means "how many records you are going to fetch at one time when moving cursor". 0, as I said, is ignored and means "fetch all records at one time", 1 means "fetch one record, then another one" meaning driver will make a network call to fetch each record and return it to RowProcessor. So it's better to use something like 100 to save network round trips.

Copy link

@alniks you are correct, here is a better explanation:

ipolevoy added a commit that referenced this issue Apr 21, 2021
Copy link

@alniks the last commit should take care of the issue. I will be included in the release for Java 16

Copy link

I still would not say this is a bug, since the other method allows you a full control over a fetch size:

public static RowProcessor find(RowProcessor.ResultSetType type, RowProcessor.ResultSetConcur concur, int fetchSize, String query, Object ... params) {

I'd rather call this an enhancement :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

No branches or pull requests

2 participants