Skip to content

Fix an issue with JDBC scans #47

Closed
wants to merge 2 commits into from

5 participants

@sudiptodas

When issuing scans, a JDBC driver can either use a cursor to stream the results as they are consumed or can fetch the entire result set. The postgres JDBC driver does the latter. Therefore, YCSB runs out of heap space in workloads with scans on large tables. I added a new parameter to specify a limit on the number of rows fetched. This is a hint to the driver, not a binding contract, but seems to work with postgres. I felt this is a more portable way compared to using say LIMIT clause, whose syntax varies from one DBMS to another.

The added parameter is scan.fetch.limit whose default is 10 rows and can be overridden by the -p option of the YCSB command line arguments.

Hope this is helpful.

Sudipto

@m1ch1
Collaborator
m1ch1 commented Nov 21, 2011

Hi Sudipto,

Thanks for the patch! We've had many check-ins recently, and it looks like your branch is a bit out of date. Could you pull all the changes from the master branch?

Thanks!
--Michi

@sudiptodas

Hi Michi,

I'll look into it.

Sudipto

@sudiptodas

Hi Michi,

Let me know if the above merge is what you wanted.

Sudipto

@m1ch1
Collaborator
m1ch1 commented Dec 1, 2011

Hi Sudipto,

I still see changes that I think shouldn't be included in this patch:

sudiptodas@aaa247c

Maybe it's easier if you can send me the diff for your JDBC client change. Sorry I'm still a git newbie :)

Thanks!
--Michi

@pmq pmq added a commit to pmq/YCSB that referenced this pull request Nov 27, 2012
@pmq pmq gh-47 Fix an issue with JDBC scans
JDBC scans did not instruct the driver to use a specific fetchsize,
resulting in most cases in an OOM; add support for setting fetchsize.

Also add support for disabling autocommit on the JDBC connections,
mostly because Postgres for example doesn't support setting fetchsize
without disabling autocommit.
e44e677
@pmq
pmq commented Nov 27, 2012

Hi Michi, here's a patch for this issue. I don't know if I should reopen a pull request or if this is sufficient.

Sudipto's patch won't probably work, as for this feature to work correctly on Postgres, you'd need as documented in my patch to disable auto-commit, otherwise the underlying cursor won't be present any more.

@m1ch1
Collaborator
m1ch1 commented Nov 27, 2012

Thanks Pierre, I'll review the patch some time this week.

--Michi

@m1ch1 m1ch1 pushed a commit that referenced this pull request Dec 16, 2012
@pmq pmq gh-47 Fix an issue with JDBC scans
JDBC scans did not instruct the driver to use a specific fetchsize,
resulting in most cases in an OOM; add support for setting fetchsize.

Also add support for disabling autocommit on the JDBC connections,
mostly because Postgres for example doesn't support setting fetchsize
without disabling autocommit.
7b564cc
@joey joey added a commit to joey/YCSB that referenced this pull request Feb 7, 2013
@pmq pmq gh-47 Fix an issue with JDBC scans
JDBC scans did not instruct the driver to use a specific fetchsize,
resulting in most cases in an OOM; add support for setting fetchsize.

Also add support for disabling autocommit on the JDBC connections,
mostly because Postgres for example doesn't support setting fetchsize
without disabling autocommit.
2a5c4a3
@wolfgangihloff wolfgangihloff pushed a commit that referenced this pull request Jan 21, 2015
@pmq pmq gh-47 Fix an issue with JDBC scans
JDBC scans did not instruct the driver to use a specific fetchsize,
resulting in most cases in an OOM; add support for setting fetchsize.

Also add support for disabling autocommit on the JDBC connections,
mostly because Postgres for example doesn't support setting fetchsize
without disabling autocommit.
11e736c
@busbey
Collaborator
busbey commented Jun 6, 2015

this got fixed back in commit 7b564cc

@busbey busbey closed this Jun 6, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.