Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downloading table in chunks by bq_table_download fails with "Invalid unsigned integer value" #248

Closed
AlekseyBuzmakov opened this issue May 23, 2018 · 2 comments

Comments

@AlekseyBuzmakov
Copy link

commented May 23, 2018

I run the following code:

proj = "test-194315"
sql = "SELECT * FROM SberbankContestData.Transactions_native"
tb <- bq_project_query(proj, sql)

ps = 10000
max_res = ps * 10
first = bq_table_download(tb, max_results = max_res, page_size=ps)
nrow_read = max_res
chunk = bq_table_download(tb, max_results = max_res, page_size=ps, start_index = nrow_read )

However the code reports:

Downloading 100,000 rows in 10 pages.
Error: Invalid unsigned integer value: '-20000'. [invalidParameter]

The only version I found in ?bigrquery at the end of the help:

Package bigrquery version 1.0.0

@stPhena

This comment has been minimized.

Copy link

commented Aug 8, 2018

The error seems to lie in the function bq_download_page_info() which does not calculate the end positions correctly. Like in the following example (when I tried to download rows from 500000 to 1000000):

> bq_download_page_info(nrows=3e+06, max_results=1000000, page_size=10000,
                        start_index=500000)
$n_rows
[1] 1000000
$n_pages
[1] 100
$begin
  [1]  500000  510000  520000  530000  540000  550000  560000  570000  580000
 [10]  590000  600000  610000  620000  630000  640000  650000  660000  670000
 [19]  680000  690000  700000  710000  720000  730000  740000  750000  760000
 [28]  770000  780000  790000  800000  810000  820000  830000  840000  850000
 [37]  860000  870000  880000  890000  900000  910000  920000  930000  940000
 [46]  950000  960000  970000  980000  990000 1000000 1010000 1020000 1030000
 [55] 1040000 1050000 1060000 1070000 1080000 1090000 1100000 1110000 1120000
 [64] 1130000 1140000 1150000 1160000 1170000 1180000 1190000 1200000 1210000
 [73] 1220000 1230000 1240000 1250000 1260000 1270000 1280000 1290000 1300000
 [82] 1310000 1320000 1330000 1340000 1350000 1360000 1370000 1380000 1390000
 [91] 1400000 1410000 1420000 1430000 1440000 1450000 1460000 1470000 1480000
[100] 1490000
$end
  [1]  510000  520000  530000  540000  550000  560000  570000  580000  590000
 [10]  600000  610000  620000  630000  640000  650000  660000  670000  680000
 [19]  690000  700000  710000  720000  730000  740000  750000  760000  770000
 [28]  780000  790000  800000  810000  820000  830000  840000  850000  860000
 [37]  870000  880000  890000  900000  910000  920000  930000  940000  950000
 [46]  960000  970000  980000  990000 1000000 1000000 1000000 1000000 1000000
 [55] 1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000
 [64] 1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000
 [73] 1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000
 [82] 1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000
 [91] 1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000
[100] 1000000

I would say that adding an end_index parameter to functions bq_table_download() and bq_download_page_info() would solve the issue.

@hadley

This comment has been minimized.

Copy link
Member

commented Jan 22, 2019

bigrquery:::bq_download_page_info(
  nrow = 3e+06,
  max_results = 1000,
  page_size = 100,
  start_index = 500
)
#> $n_rows
#> [1] 1000
#> 
#> $n_pages
#> [1] 10
#> 
#> $begin
#>  [1]  500  600  700  800  900 1000 1100 1200 1300 1400
#> 
#> $end
#>  [1]  600  700  800  900 1000 1000 1000 1000 1000 1000

Created on 2019-01-22 by the reprex package (v0.2.1.9000)

@hadley hadley closed this in 6b15ed6 Jan 22, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.