Problem with `req_body_json()` and passing jsonlite args

[juliasilge]

I am having trouble passing jsonlite args through to req_body_json():

library(jsonlite)
library(httr2)

cars <- mtcars[4:7,]

## this is what I want
toJSON(cars, dataframe = "rows")
#> [{"mpg":21.4,"cyl":6,"disp":258,"hp":110,"drat":3.08,"wt":3.215,"qsec":19.44,"vs":1,"am":0,"gear":3,"carb":1,"_row":"Hornet 4 Drive"},{"mpg":18.7,"cyl":8,"disp":360,"hp":175,"drat":3.15,"wt":3.44,"qsec":17.02,"vs":0,"am":0,"gear":3,"carb":2,"_row":"Hornet Sportabout"},{"mpg":18.1,"cyl":6,"disp":225,"hp":105,"drat":2.76,"wt":3.46,"qsec":20.22,"vs":1,"am":0,"gear":3,"carb":1,"_row":"Valiant"},{"mpg":14.3,"cyl":8,"disp":360,"hp":245,"drat":3.21,"wt":3.57,"qsec":15.84,"vs":0,"am":0,"gear":3,"carb":4,"_row":"Duster 360"}]
## this is what I do NOT want
toJSON(cars, dataframe = "columns")
#> {"mpg":[21.4,18.7,18.1,14.3],"cyl":[6,8,6,8],"disp":[258,360,225,360],"hp":[110,175,105,245],"drat":[3.08,3.15,2.76,3.21],"wt":[3.215,3.44,3.46,3.57],"qsec":[19.44,17.02,20.22,15.84],"vs":[1,0,1,0],"am":[0,0,0,0],"gear":[3,3,3,3],"carb":[1,2,1,4],"_row":["Hornet 4 Drive","Hornet Sportabout","Valiant","Duster 360"]}

## does not do what I want -- looks like `dataframe = "columns"`
request("http://httpbin.org/") %>% req_body_json(cars, dataframe = "rows") %>% req_dry_run()
#> POST / HTTP/1.1
#> Host: httpbin.org
#> User-Agent: httr2/0.1.1 r-curl/4.3.2 libcurl/7.64.1
#> Accept: */*
#> Accept-Encoding: deflate, gzip
#> Content-Type: application/json
#> Content-Length: 247
#> 
#> {"mpg":[21.4,18.7,18.1,14.3],"cyl":[6,8,6,8],"disp":[258,360,225,360],"hp":[110,175,105,245],"drat":[3.08,3.15,2.76,3.21],"wt":[3.215,3.44,3.46,3.57],"qsec":[19.44,17.02,20.22,15.84],"vs":[1,0,1,0],"am":[0,0,0,0],"gear":[3,3,3,3],"carb":[1,2,1,4]}
request("http://httpbin.org/") %>% req_body_json(cars, dataframe = "columns") %>% req_dry_run()
#> POST / HTTP/1.1
#> Host: httpbin.org
#> User-Agent: httr2/0.1.1 r-curl/4.3.2 libcurl/7.64.1
#> Accept: */*
#> Accept-Encoding: deflate, gzip
#> Content-Type: application/json
#> Content-Length: 247
#> 
#> {"mpg":[21.4,18.7,18.1,14.3],"cyl":[6,8,6,8],"disp":[258,360,225,360],"hp":[110,175,105,245],"drat":[3.08,3.15,2.76,3.21],"wt":[3.215,3.44,3.46,3.57],"qsec":[19.44,17.02,20.22,15.84],"vs":[1,0,1,0],"am":[0,0,0,0],"gear":[3,3,3,3],"carb":[1,2,1,4]}

^{Created on 2021-10-20 by the reprex package (v2.0.1)}

I think this may be the problem in #88

[hongooi73]

I think I've tracked down why this happens. req_body_json ultimately calls req_body_apply to generate the request body, which in turn calls exec(jsonlite::toJSON, data, !!!req$body$params). However, the class attribute of the data is lost by the time this happens, so the dataframe arg doesn't do anything.

req <- request("http://httpbin.org/") %>% req_body_json(cars, dataframe = "rows") 
class(req$body$data)
# [1] "list"


# need to restore 'data.frame' class for desired output
req$body$data <- as.data.frame(req$body$data)
httr2:::req_body_apply(req)

# POST http: //127.0.0.1:9359/ HTTP/1.1
# Host: httpbin.org
# User-Agent: httr2/0.1.1 r-curl/4.3 libcurl/7.64.1
# Accept: */*
# Accept-Encoding: deflate, gzip
# Proxy-Connection: Keep-Alive
# Content-Type: application/json
# Content-Length: 1144

# [{"speed":4,"dist":2},{"speed":4,"dist":10},{"speed":7,"dist":4},{"speed":7,"dist":22},
# ...

[hongooi73]

It looks like the class stripping happens in req_body_json itself, here:

httr2/R/req-body.R

Line 104 in 87011c0

data <- modify_list(req$body$data, !!!data)

Is there a reason why NSE is invoked in this line? Or indeed, why we need to combine lists?

[hadley]

The intent was to allow you to iteratively build up a request body, in the same way that you can build up a url. But this no longer seems like a good idea to me, so we could easily fix by removing the data <- modify_list(req$body$data, !!!data) line.

[hadley]

OTOH it's easy enough to fix this, and it is occasionally useful to built up a response piece-by-piece.

[hadley]

But if you have a data frame and then add a scalar, recycling will fill in the whole column, so this again seems like a bad idea. I think it's better to just replace the existing data.

[hadley]

Oooh, but the existing OAuth flows depend on this behaviour so it's better to make a small change to make this work rather than attempting to overhaul it all.