Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Proxy settings in Windows #1

Closed
jeroen opened this issue Nov 22, 2014 · 27 comments
Closed

Proxy settings in Windows #1

jeroen opened this issue Nov 22, 2014 · 27 comments

Comments

@jeroen
Copy link
Owner

jeroen commented Nov 22, 2014

Read proxy settings in windows to mimic setInternet2:

http://stackoverflow.com/questions/202547/how-do-i-find-out-the-browsers-proxy-settings
@hadley
Copy link
Collaborator

hadley commented Feb 20, 2015

Oh that would be awesome!

@jeroen
Copy link
Owner Author

jeroen commented Feb 27, 2015

I've implemented a function ie_proxy_info that looks up this information on windows. Thereby users or high level packages can decide for themselves if they want to use this to automatically set CURLOPT_PROXY.

@jeroen jeroen closed this as completed Feb 27, 2015
@hadley
Copy link
Collaborator

hadley commented Feb 27, 2015

What's the argument for not doing this by default?

@jeroen
Copy link
Owner Author

jeroen commented Feb 27, 2015

It's windows, it has it's own logic/configuration that might not map perfectly to libcurl. But maybe if we test it and it works well can do it by default.

@jeroen jeroen reopened this Feb 27, 2015
@jeroen
Copy link
Owner Author

jeroen commented Feb 27, 2015

So the API returns these 4 fields:

  • fAutoDetect if the proxy server is "auto detected" whatever that may mean.
  • lpszAutoConfigUrl a string with the url of the auto detect
  • lpszProxy the actual proxy url. Need to parse that to feed it into CURLOPT_PROXY, CURLOPT_PROXYPORT.
  • lpszProxyBypass a semicolon separated list of urls that you want to bypass. Need to parse and feed to CURLOPT_NOPROXY.

Libcurl also allows for setting CURLOPT_PROXYTYPE, CURLOPT_PROXYUSERNAME, CURLOPT_PROXYPASSWORDand CURLOPT_PROXYAUTH for which there is no windows equivalent I think.

@jmcphers
Copy link

jmcphers commented Mar 2, 2015

Judging from a cursory reading of the SO post linked by @jeroenooms, auto-detected proxies require a further call to WinHttpGetProxyForUrl to get the actual proxy prior to handling a request, which I'm guessing is not possible by setting CURLOPT_.

IMO, it'd be reasonable to do the following:

  • if no proxy config info is available (i.e. WinHttpGetIEProxyConfigForCurrentUser returns false), don't set any CURLOPT_ automatically
  • if a proxy is specified explicitly, use it if CURLOPT_PROXY is not set
  • if the configuration specifies that an auto-detected proxy is to be used, look up the proxy URL per request if CURLOPT_PROXY is not set -- might need to get put behind an argument or flag; some systems may have the "autodetect" flag ticked but no actual proxy requirements, and I'm not sure what the behavior is there

@jeroen
Copy link
Owner Author

jeroen commented Mar 3, 2015

I don't have enough faith in this to enable it by default. For example I noticed if the configured proxy is set to an invalid or dead server, windows will silently ignore it. I suppose this is considered a feature. However if we set an invalid proxy in libcurl, it won't connect or might even crash.

Perhaps we can have something like setInternet2 where the user can globally opt-in to automatically using the proxy.

@jeroen
Copy link
Owner Author

jeroen commented Jul 29, 2015

The 'auto proxy' just points to a piece of JavaScript that maps a URL to a proxy server: https://en.wikipedia.org/wiki/Proxy_auto-config

@jeroen
Copy link
Owner Author

jeroen commented Aug 9, 2015

This is now mostly implemented, see the announcement post. However I do not have access to a proxy server to actually test this.

@sebastian-c
Copy link

I don't know if this is relevant but in the latest R release (3.2.2):

  • (Windows only) setInternet2(TRUE) is now the default. The
    command-line option --internet2 and environment variable
    R_WIN_INTERNET2 are now ignored.

@sonicrick
Copy link

Not working on my PAC. ie_get_proxy_for_url(url) threw an error Error in ie_get_proxy_for_url(url) : ERROR_WINHTTP_UNABLE_TO_DOWNLOAD_SCRIPT

ie_proxy_info()$AutoConfigUrl returns the PAC name e.g. http://myproxypac.mydomain/proxy.pac

@jeroen
Copy link
Owner Author

jeroen commented Nov 12, 2015

Can you show us the source code of your pac file?

@sonicrick
Copy link

A shortened and half-cleaned one here. Too long to clean all the IPs so I've omitted quite a few, I hope it doesn't affect the trouble shooting.

proxypac.txt

@jeroen
Copy link
Owner Author

jeroen commented Nov 12, 2015

Mmm strange. In the topic you said initially that for ie_get_proxy_for_url() returned "proxy.<my.domain>:8080". Did you change any settings?

@sonicrick
Copy link

Apologies, apparently the error is my side (malformed value in url) causing the error. Your answer in SO worked for my pac, thanks!

@jeroen
Copy link
Owner Author

jeroen commented Feb 17, 2016

I think this is all working now. At least haven't heard complaints for a while.

@jeroen jeroen closed this as completed Feb 17, 2016
@jangorecki jangorecki mentioned this issue Apr 29, 2016
Closed
@sagar-m
Copy link

sagar-m commented Apr 20, 2017

Hi,

I'm rying to get bypass the pac proxy setup at my work to be able to download packages in R. The following gave me an error. Please advise! Greatly appreciate it.

Have attach a snapshot of my internet settings.

require("curl")

curl_with_proxy <- function(url, verbose = TRUE){
proxy <- ie_get_proxy_for_url(url)
h <- new_handle(verbose = verbose, proxy = proxy)
curl(url, handle = h)
}

con <- curl_with_proxy("https://httpbin.org/get")
readLines(con)

Sys.setenv(http_proxy_user = "plum:apple")
Sys.setenv(http_proxy = "FGProxy1:3128")

The error that was printed in the console:

  • No valid port number in proxy string (3128;FGProxy1:3128)
  • timeout on name lookup is not supported
  • Trying 10.0.4.41...
  • TCP_NODELAY set
  • connect to 10.0.4.41 port 443 failed: Connection refused
  • Failed to connect to FGProxy1 port 443: Connection refused
  • Closing connection 0
    Error in readLines(con) : Couldn't connect to server

snip

@cderv
Copy link

cderv commented Apr 26, 2017

with download.file and method = "libcurl", it it explained that we could use http_proxy environment variable with the form [user:password@]machine[:port]. and it is working.

Is their a specificity in curl package that prevent from using this ? It does not seems to work.

@jeroen
Copy link
Owner Author

jeroen commented Apr 26, 2017 •
edited

Curl uses separate varaibles for http and https proxy. Here is an example https://gist.github.com/jeroen/5127c288f8914bdb20be

@cderv
Copy link

cderv commented Apr 26, 2017 •
edited

ok Thanks for the gist. I understand it is case sensitive. It is working for me. @sagar-m you should try.

If you want me to try document this somewhere, let me know.

@sagar-m
Copy link

sagar-m commented Apr 27, 2017

Hi @cderv , not sure I followed. Please explain how I can resolve the error. Thank you so much.

@cderv
Copy link

cderv commented May 22, 2017

As shown in the gist, configure environnement variables in R. It will be understand by libcurl. You just need to know you proxy url. curl::ie_get_proxy_for_url() could help.

Sys.setenv(http_proxy = "[scheme://][user:password@]machine[:port]")
Sys.setenv(HTTPS_PROXY= "[scheme://][user:password@]machine[:port]")

Replace bracket parts with your specific case. for me, [scheme://] is http://

Try configure your proxy and run these lines to test

# http test
readLines(base::url("http://httpbin.org/get", method = "libcurl"))
readLines(curl::curl("http://httpbin.org/get"))

# https test
readLines(base::url("https://httpbin.org/get", method = "libcurl"))
readLines(curl::curl("https://httpbin.org/get"))

@sagar-m
Copy link

sagar-m commented Jun 12, 2017 via email

@jeroen
Copy link
Owner Author

jeroen commented Jun 12, 2017

@sagar-m from your output it looks like things are working. What error do you get?

@sagar-m
Copy link

sagar-m commented Jun 12, 2017 via email

@jeroen
Copy link
Owner Author

jeroen commented Jun 12, 2017

You are using an old version of R. You can solve the problem by upgrading to R 3.4. If you don't want to upgrade, try this:

install.packages("uroot", method = "libcurl")

@sagar-m
Copy link

sagar-m commented Jun 12, 2017 via email

@mpadge mpadge mentioned this issue Dec 5, 2017
Closed
@stevengj stevengj mentioned this issue Mar 6, 2019
Merged
@rlmoore0 rlmoore0 mentioned this issue Aug 19, 2019
Closed
@ldecicco-USGS ldecicco-USGS mentioned this issue Dec 27, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@hadley @jeroen @jmcphers @sebastian-c @cderv @sonicrick @sagar-m