User defined cookies / headers are overwritten

[colearendt]

For the purposes of my example, I have only tested with libcurl. However, hopefully we can remedy this issue with the other download handlers as well.

Straight invocation of libcurl:

hm <- curl::new_handle()
curl::handle_setopt(hm, .list = list(httpheader = "key:value"))
curl::curl_fetch_echo("http://localhost:32776/upload", handle = hm)$headers
#>                                                    accept 
#>                                                     "*/*" 
#>                                           accept-encoding 
#>                                           "deflate, gzip" 
#>                                                      host 
#>                                         "localhost:32776" 
#>                                                       key 
#>                                                   "value" 
#>                                                user-agent 
#> "R (3.4.3 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)"
curl::handle_setopt(hm, .list = list(httpheader = "key2:value2"))

# NOTE: key:value missing
curl::curl_fetch_echo("http://localhost:32776/upload", handle = hm)$headers
#>                                                    accept 
#>                                                     "*/*" 
#>                                           accept-encoding 
#>                                           "deflate, gzip" 
#>                                                      host 
#>                                         "localhost:32776" 
#>                                                      key2 
#>                                                  "value2" 
#>                                                user-agent 
#> "R (3.4.3 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)"
curl::handle_setopt(hm, .list = list(cookielist = c("key=value;key2=value2")))

# cannot see it here, but there are key=value; key2=value2 cookies defined on this request
curl::curl_fetch_echo("http://localhost:32776/upload", handle = hm)$headers
#>                                                    accept 
#>                                                     "*/*" 
#>                                           accept-encoding 
#>                                           "deflate, gzip" 
#>                                                      host 
#>                                         "localhost:32776" 
#>                                                      key2 
#>                                                  "value2" 
#>                                                user-agent 
#> "R (3.4.3 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)"
curl::handle_setopt(hm, .list = list(httpheader = c("cookie:key=value")))

# cannot see here, but key2:value2 cookie is missing. Also, key2=value2 header missing
curl::curl_fetch_echo("http://localhost:32776/upload", handle = hm)$headers
#>                                                    accept 
#>                                                     "*/*" 
#>                                           accept-encoding 
#>                                           "deflate, gzip" 
#>                                                    cookie 
#>                                  "key=value; key2=value2" 
#>                                                      host 
#>                                         "localhost:32776" 
#>                                                user-agent 
#> "R (3.4.3 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)"

^{Created on 2020-02-14 by the reprex package (v0.3.0)}

As you can see, the httpheader setting is not additive, but overwrites. It even overwrites previously specified cookies. As a result, when we add the user's options here:

rsconnect/R/http-libcurl.R

Lines 9 to 12 in fef0bf6

	userOptions <- getOption("rsconnect.libcurl.options")
	if (is.list(userOptions)) {
	curl::handle_setopt(handle, .list = userOptions)
	}

And then don't incorporate them later:

rsconnect/R/http-libcurl.R

Line 121 in fef0bf6

curl::handle_setheaders(handle, .list = headers)

Everything the user has specified gets overwritten.

Are there any other workarounds to specify arbitrary user-defined headers / cookies via .rsconnect_profile? This is something that we have a few customers stuck at when deploying to RStudio Connect (specifically when headers / cookies need to be specified to satisfy a proxy).

[colearendt]

Spoke with @jeroen offline and libcurl (in particular) does not allow "reading" previously set headers. So to enable setting headers / cookies, we probably need to either:

• expose a way to set headers / cookies outside of options("rsconnect.libcurl.options")
• parse options("rsconnect.libcurl.options") to see which ones set headers

If we go with option 1 (which feels cleaner and may be more general across http handlers), then perhaps we should document that headers set via options("rsconnect.libcurl.options") will be overwritten.

At this point, I'm not sure whether this is a bug or feature request, but basically

users should be able to set arbitrary headers / cookies via .rsconnect_profile for use in enterprise environments

😄

[kevinushey]

It seems like the right solution would involve us building up a list of options (overlaying user-specified options as appropriate) and then passing everything into curl::handle_setopt() in one go, rather than setting things piecemeal (as we then risk overwriting user-specified values)

Alternatively, the cheap way out would be to add options like rsconnect.libcurl.headers / rsconnect.libcurl.cookies.

[colearendt]

Brilliant! That sounds like a win to me! 😄 Is it worth my investigating the other handlers as well? Ideally there'd be a way to set headers / cookies no matter which HTTP handler is chosen.

[rstub]

It looks like (deprecated) RCurl has the same problem, since it overwrites anything that is present in options$httpheaders

rsconnect/R/http-rcurl.R

Line 78 in fef0bf6

options$httpheader <- extraHeaders

Plain curl does not offer any way to set custom headers. A general rsconnect.headers / rsconnect.cookies used by all backends might make sense.