Problems downloading 2GB text file

[MarkEdmondson1234]

Thread from cloudyr/cloudyr.github.io#15

@fncalderong said:

Hi, I am using googleCloudStorageRs package. I am trying to download a 2gb txt file with R but have no susses.

I tried to download the file with gcs_get_object function, but the file is in Raw format and is quite dificult to convert it.

I tried to get the URL of the file with gcs_download_url function, but when I try to download the file via URL using downloadFile from R.utils Package it download a html file (the page where google ask the credentials).

@MarkEdmondson1234 said:

Yes best report there, but it sounds like you only need to specify the .txt file extension in the name you have downloaded it to. Try renaming the file.

@fncalderong said:

when I try to run this:

gcs_get_object(objects$name[[36]], saveToDisk = "data.txt",bucket)

I get this:

Error in writeBin(bin, saveToDisk) :
  long vectors not supported yet: connections.c:4123

where
objects$name[[36]][1] "20170523/planofull_170523.txt"
thanks

[MarkEdmondson1234]

Hmm ok, this is the save function in base R failing as 2GB is too big. Will investigate to see if it can be solved.

[MarkEdmondson1234]

From ?writeBin: Only 2^31 - 1 bytes can be written in a single call (and that is the maximum capacity of a raw vector on 32-bit platforms).

@leeper is there a work around for this?

[MarkEdmondson1234]

Same issue here richfitz/remake#76

[fncalderong]

Nice!!!!....thank you so much. the reason I think is beacause gcs_get_object download it in raw format (long vector) and as you said is to big for R. But for instance if I try to download the file using Rcurl with this command:

URL <- gcs_download_url(objects$name[[36]],bucket,public=T)
library(RCurl)
getURL(URL)
download.file(URL, destfile = "data2.txt", method="wget")

I can download the file successfuly only if I set the file public (I can´t make this for security reasons).

[MarkEdmondson1234]

Hmm ok, in that case try downloading it via the gcs_download_url method, as you can set URLs that are not public but accessible via your Google account sign in.

[fncalderong]

So, in that way I must always give my credentials by web. Am I rigth?.

[fncalderong]

I meant, if i do it with a not public object, the result is a html file (a google page asking credentials).

[MarkEdmondson1234]

Yes, as we unfortunately don't have this feature yet, where you could have a signed URL that you can reach via code #54

I'll see if you can use the same auth you have from the other API calls to apply to the download that you can make via httr::GET

[MarkEdmondson1234]

Probably best solution at the moment would be to use the python gsutil tool (https://cloud.google.com/storage/docs/access-control/create-signed-urls-gsutil) , either directly or to create the signed URL, then call that URL from R. If necessary you can wrap the python tool in R wrappers if you must stick within R.

[MarkEdmondson1234]

Ok it looks like a solved issue in httr, so I'll look to implement it r-lib/httr#44

[fncalderong]

Man that would be awesome!!! I appreciate your help.

[MarkEdmondson1234]

Give that a go now from the latest version on github, its working in my local tests for small files at least.

[MarkEdmondson1234]

gcs_get_object(obj$name[[3]], bucket = "your-project", saveToDisk = "blah.txt", overwrite = TRUE)