Does read_html() support using a proxy?

[magic-lantern]

I'm doing some work behind a SOCKS5 proxy. I've setup my environment such that I can update or install new packages by setting the proper environment variable (Sys.setenv(http_proxy = "socks://proxyhost:port")).

However, read_html("some url") fails due to not attempting (or failing) to use my proxy.

Does rvest support use of a proxy directly?

It does appear that you can instead pass an httr result, of the form:

read_html(httr::GET("https://cran.r-project.org/src/contrib/", httr::use_proxy("socks5://proxyhost:port")))

[jimhester]

rvest (and xml2, which is actually what read_html() is from) use httr for the connections.

You can set the global configuration options for httr (such as this) with httr::set_config().

So something like this should use the proxy configuration for all httr connections.

httr::set_config(httr::use_proxy("socks5://proxyhost:port")))

[magic-lantern]

Thanks for the clarification.

[1beb]

Hey Jim,

Can confirm that this does not work. The httr config does not get called through into read_html.

library(rvest)
library(httr)
httr::set_config(
  httr::use_proxy(
    "socks5://proxy-nl.privateinternetaccess.com", port=1080, username = "USER", password = "PASSWORD") 
)

# works for httr
> GET("https://myip.com") %>% content(as='text') %>% read_html %>% html_nodes("li") %>% html_text()
[1] "\nYour IP address is:\n\n109.201.154.197\ncopy\n\n"               
[2] "\nHost:\n172.69.55.92\n"                                          
[3] "\nRemote Port:\n33336\n"                                          
[4] "\nISP:\n\n\nAmsterdam Residential Television and Internet, LLC \n"
# rvest ignores the proxy, returning my true IP
> read_html("https://myip.com") %>% html_nodes("li") %>% html_text()
[1] "\nYour IP address is:\n\n200.121.230.93\ncopy\n\n"  "\nHost:\n198.41.231.175\n"                         
[3] "\nRemote Port:\n33762\n"                            "\nISP:\n\n\nInternet Assigned Numbers Authority \n"

I can provide user/pass for test if you need it @jimhester

[jimhester]

You are correct, looking at the code again xml2 uses curl::curl() when passed a URL. However if you call read_html(GET("https://myip.com")) than this should work.

[rcepka]

Hello,
Is there any way to incorporate read_html_live() please?
It is obviously not possible just simply replace read_html by read_html_live. In other words I am trying to use proxy with read_html_live(). Any advices please?