curl_fetch_stream: return value of 'fun' not evaluated - desired behaviour?

[bart6114]

This relates to r-lib/httr#352.

If I understand correctly httr::write_stream is mostly offhanded to curl::curl_fetch_stream which does not take into account the value that is returned by the callback function in such way as defined in the httr docs.

As a reference the docs from ?httr::write_stream:

write_stream(f)

Arguments

f: Callback function. It should have a single argument, a raw vector containing the bytes recieved from the server. This will usually be 16k or less. It should return the length of bytes processed - if this is less than the input length, the function will terminate.

If the callback function's return value would actually be checked to be the length of bytes processed, which it doesn't ATM, this would allow for terminating the stream when required.

This quick hack:

curl_fetch_stream <- function(url, fun, handle = new_handle()){
  con <- .Call(R_curl_connection, url, "rb", handle, FALSE)
  on.exit(close(con))
  while(length(bin <- readBin(con, raw(), 8192L))){
    fun_eval<-fun(bin)
    if(!is.numeric(fun_eval) || fun_eval<length(bin)) break
    fun_eval
  }
  handle_response_data(handle)
}

Would allow e.g. for:

foo<-1
bar<-5

GET("https://jeroenooms.github.io/data/diamonds.json",
    write_stream(function(x) {
      print(length(x))
      foo<<-foo+1
      if(foo > bar) return(0)

      length(x)
    })
)

Or maybe I'm taking this too far and there is a much easier way to terminate the handler when wanted? 😁

[jeroen]

We can't add a feature like this in curl right now because functions already using curl_fetch_stream will likely break. There is no mention in the curl documentation that the handler should return the buffer size. So I think this is mostly a bug in httr.

As an alternative you can simply use curl's connection interface:

con <- curl("https://jeroenooms.github.io/data/diamonds.json")
open(con, "r")
while(buf <- readLines(con, n = 10)){
  do_whatever(buf)
}
close(con)

[bart6114]

Indeed, looks more like a bug / wrong documentation on the httr side.
Thanks for the alternative.