Athena can only pull 999 rows at a time

[daroczig]

No description provided.

[nfultz]

Yeck, RJDBC needs to get updated to use new DBI -

I think it makes more sense to implement dbFetch here than dbGetQuery, otherwise things might work inconsistently across the package.

In the dbFetch implementation, you should be able to calculate block size a la:
with block = if(n < 0) 999 else pmin(n, 999)

and then delegate the request back to RJDBC::fetch with the smaller block size.

See also [1] and [2]

[1] https://github.com/s-u/RJDBC/blob/master/java/JDBCResultPull.java#L69
[2] https://github.com/s-u/RJDBC/blob/master/R/class.R#L328

[nfultz]

Also is this 999 limitation documented anywhere?

[nfultz]

Ah s-u/RJDBC#44

[daroczig]

I'd like to get dbGetQuery working on Athena, and as fetch has a default 2K set for the blocksize -- that is called by dbGetQuery without passing block, I don't really see any other easy way for this. This PR required more commits than I originally hoped :) But seems to work fine now.

[daroczig]

@nfultz pls feel free to tweak this further, I don't really want to start hacking dbSendQuery and fetch instead

[daroczig]

Closing in favour of #9