File error (Out of disk space) during bq_table_download

[mr2dark]

I would like to run an R script which uses bigrquery to run a query to BQ and download results within an AWS Fargate task which is rather disk space constrained. There are a lot of data and when I run that script it eventually fails with the following lines in logs:

Error in file(path, open = "wb") : cannot open the connection
Calls: <Anonymous> ... bq_download_pages -> <Anonymous> -> <Anonymous> -> file
In addition: There were 50 or more warnings (use warnings() to see the first 50)

I assume that the cause of the problem is in the fact that bq_download_pages creates a set of temporary files which aren't removed after being parsed.
As a workaround (rather nasty) I do the following after a call to bq_download_pages:

# A workaround cleanup until https://github.com/r-dbi/bigrquery/issues/343 is fixed
do.call(file.remove, list(list.files(tempdir(), pattern = 'bq-[[:xdigit:]]+\\.json', full.names = TRUE)))

But I believe the proper solution will be adding before the final line of bq_download_pages the following line:

do.call(file.remove, page_paths)

[meztez]

The file haven't been parsed at this point, files are parsed inside bq_table_download, one solution would be to add it after

bigrquery/R/bq-download.R

Line 89 in 5f43228

convert_bigint(table_data, bigint)

[hadley]

Fixed by #372