httpclient package can't make SSL requests over an HTTP proxy #4520

Closed
RyanMarcus opened this Issue Jul 26, 2016 · 1 comment

Projects

None yet

2 participants

@RyanMarcus
Contributor

Here's a simple reproducer (assuming you have the https_proxy environmental variable set):

import httpclient
import os
import uri

var sysProxy:Proxy = nil
if os.existsEnv("https_proxy"):
  let httpsProxyStr = os.getEnv("https_proxy").string
  let parsed = uri.parseUri(httpsProxyStr)
  sysProxy = newProxy($parsed, nil)
  echo("Have an HTTPS proxy!")

echo(get("http://github.com", proxy=sysProxy))

I'm using uri to parse the thing and then turn it back into a string because that's what Nimble does in its getProxy method. It also won't work if you specify the proxy explicitly as a string.

Output:

[ryan@rmarcus-zbook-arch nimble_web]$ nim c -r -d:ssl main.nim 
Hint: used config file '/etc/nim.cfg' [Conf]
Hint: system [Processing]
Hint: main [Processing]
Hint: httpclient [Processing]
Hint: net [Processing]
Hint: nativesockets [Processing]
Hint: os [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: math [Processing]
Hint: times [Processing]
Hint: posix [Processing]
Hint: sets [Processing]
Hint: hashes [Processing]
Hint: etcpriv [Processing]
Hint: openssl [Processing]
Hint: uri [Processing]
Hint: strtabs [Processing]
Hint: base64 [Processing]
Hint: mimetypes [Processing]
Hint: random [Processing]
Hint: httpcore [Processing]
Hint: tables [Processing]
Hint: asyncnet [Processing]
Hint: asyncdispatch [Processing]
Hint: oids [Processing]
Hint: endians [Processing]
Hint: macros [Processing]
Hint: heapqueue [Processing]
Hint: queues [Processing]
Hint: selectors [Processing]
Hint: epoll [Processing]
CC: nimble_web_main
CC: stdlib_system
Hint:  [Link]
Hint: operation successful (34649 lines compiled; 0.779 sec total; 62.008MiB; Debug Build) [SuccessX]
Have an HTTPS proxy!
Traceback (most recent call last)
main.nim(12)             main
httpclient.nim(506)      getContent
Error: unhandled exception: 400 Bad Request [HttpRequestError]
Error: execution of an external program failed: '/home/ryan/nimble_web/main '
[ryan@rmarcus-zbook-arch nimble_web]$ 

wget and curl both contact and work through the proxy as expected.

Using strace, it looks like Nim is just sending a standard request to the proxy, as if the webpage was HTTP. Instead, it should be sending an HTTP CONNECT request, I believe.

[ryan@rmarcus-zbook-arch nimble_web]$ strace -f -e trace=network -s 10000 ./main
Have an HTTPS proxy!
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
socket(AF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
connect(4, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
socket(AF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
connect(4, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.50.85")}, 16) = 0
sendto(4, "\5\214\1\0\0\1\0\0\0\0\0\0\tproxy-ccy\7houston\2hp\3com\0\0\1\0\1", 42, MSG_NOSIGNAL, NULL, 0) = 42
recvfrom(4, "\5\214\201\200\0\1\0\1\0\6\0\6\tproxy-ccy\7houston\2hp\3com\0\0\1\0\1\300\f\0\1\0\1\0\0\10\264\0\4\20\330\353\24\300\36\0\2\0\1\0\2g\350\0\6\3ns6\300\36\300\36\0\2\0\1\0\2g\350\0\6\3ns5\300\36\300\36\0\2\0\1\0\2g\350\0\6\3ns1\300\36\300\36\0\2\0\1\0\2g\350\0\6\3ns4\300\36\300\36\0\2\0\1\0\2g\350\0\6\3ns2\300\36\300\36\0\2\0\1\0\2g\350\0\6\3ns3\300\36\300j\0\1\0\1\0\0\5\246\0\4\17\333\221\f\300\216\0\1\0\1\0\0\6\207\0\4\17\333\240\f\300\240\0\1\0\1\0\0\6l\0\4\17\323\300\f\300|\0\1\0\1\0\0\5\255\0\4\17\313\340\16\300X\0\1\0\1\0\0\6\177\0\4\17\303\300%\300F\0\1\0\1\0\0\6\201\0\4\17\303\320\f", 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.50.85")}, [16]) = 262
connect(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("16.216.235.20")}, 16) = 0
sendto(3, "GET http://github.com HTTP/1.1\r\nHost: github.com\r\nUser-Agent: Nim httpclient/0.14.2\r\nProxy-Authorization: basic \r\n\r\n", 116, MSG_NOSIGNAL, NULL, 0) = 116
recvfrom(3, "HTTP/1.1 301 Moved Permanently\r\nContent-length: 0\r\nLocation: https://github.com/\r\nCache-Control: proxy-revalidate\r\nProxy-Connection: Keep-Alive\r\nConnection: Keep-Alive\r\nAge: 0\r\nDate: Tue, 26 Jul 2016 21:49:53 GMT\r\n\r\n", 4000, 0, NULL, NULL) = 216
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.50.85")}, 16) = 0
sendto(4, "\221Q\1\0\0\1\0\0\0\0\0\0\tproxy-ccy\7houston\2hp\3com\0\0\1\0\1", 42, MSG_NOSIGNAL, NULL, 0) = 42
recvfrom(4, "\221Q\201\200\0\1\0\1\0\6\0\6\tproxy-ccy\7houston\2hp\3com\0\0\1\0\1\300\f\0\1\0\1\0\0\10\264\0\4\20\330\353\24\300\36\0\2\0\1\0\2g\350\0\6\3ns5\300\36\300\36\0\2\0\1\0\2g\350\0\6\3ns3\300\36\300\36\0\2\0\1\0\2g\350\0\6\3ns6\300\36\300\36\0\2\0\1\0\2g\350\0\6\3ns4\300\36\300\36\0\2\0\1\0\2g\350\0\6\3ns2\300\36\300\36\0\2\0\1\0\2g\350\0\6\3ns1\300\36\300\240\0\1\0\1\0\0\5\246\0\4\17\333\221\f\300\216\0\1\0\1\0\0\6\207\0\4\17\333\240\f\300X\0\1\0\1\0\0\6l\0\4\17\323\300\f\300|\0\1\0\1\0\0\5\255\0\4\17\313\340\16\300F\0\1\0\1\0\0\6\177\0\4\17\303\300%\300j\0\1\0\1\0\0\6\201\0\4\17\303\320\f", 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.50.85")}, [16]) = 262
connect(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("16.216.235.20")}, 16) = 0
sendto(3, "GET https://github.com/ HTTP/1.1\r\nHost: github.com\r\nUser-Agent: Nim httpclient/0.14.2\r\nProxy-Authorization: basic \r\n\r\n", 118, MSG_NOSIGNAL, NULL, 0) = 118
recvfrom(3, "HTTP/1.1 400 Bad Request\r\nCache-Control: no-cache\r\nPragma: no-cache\r\nContent-Type: text/html; charset=utf-8\r\nProxy-Connection: close\r\nConnection: close\r\nContent-Length: 691\r\n\r\n<HTML><HEAD>\n<TITLE>Request Error</TITLE>\n</HEAD>\n<BODY>\n<FONT face=\"Helvetica\">\n<big><strong></strong></big><BR>\n</FONT>\n<blockquote>\n<TABLE border=0 cellPadding=1 width=\"80%\">\n<TR><TD>\n<FONT face=\"Helvetica\">\n<big>Request Error (invalid_request)</big>\n<BR>\n<BR>\n</FONT>\n</TD></TR>\n<TR><TD>\n<FONT face=\"Helvetica\">\nYour request could not be processed. Request could not be handled\n</FONT>\n</TD></TR>\n<TR><TD>\n<FONT face=\"Helvetica\">\nThis could be caused by a misconfiguration, or possibly a malformed request.\n</FONT>\n</TD></TR>\n<TR><TD>\n<FONT face=\"Helvetica\" SIZE=2>\n<BR>\nFor assistance, contact your network support team.\n</FONT>\n</TD></TR>\n</TABLE>\n</blockquote>\n</FONT>\n</BODY></HTML>\n", 4000, 0, NULL, NULL) = 867
Traceback (most recent call last)
main.nim(12)             main
httpclient.nim(506)      getContent
Error: unhandled exception: 400 Bad Request [HttpRequestError]
+++ exited with 1 +++
[ryan@rmarcus-zbook-arch nimble_web]$ 

As you can see from above, I'm behind an HP corporate proxy that I imagine is configured just like many other corporate proxies.

Nimble, when running nimble refresh, will throw the same error.

@dom96 dom96 added the Stdlib label Jul 26, 2016
@RyanMarcus
Contributor
RyanMarcus commented Jul 27, 2016 edited

I've got a patch for this. I'm working on testing it now. Essentially, it just involved reorganizing and adding a small amount of code to request* in httpclient.nim.

Here's my branch: https://github.com/RyanMarcus/Nim

I'm running the tests now to make sure I didn't break anything.

@dom96 dom96 closed this in #4523 Jul 30, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment