HTTPS request not working with proxy

[chowey]

I have a squid proxy on my local computer (127.0.0.1:3128). It is configured to require no authentication. If I try to use an https.request, I get an error like:

{ [Error: socket hang up] code: 'ECONNRESET' }

The following code reproduces this issue (provided of course that you have a squid proxy set up on port 3128):

var https = require('https');

var options = {
  hostname: '127.0.0.1',
  port: 3128,
  path: 'https://github.com/',
  headers: {
    Host: 'github.com',
    'User-Agent': 'Node.js/0.6.6'
  },
  method: 'GET'
};

var req = https.request(options, function (res) {
  console.log('statusCode: ', res.statusCode);
  console.log('headers: ', res.headers);

  res.on('data', function (chunk) {
    process.stdout.write(chunk);
  });
}).on('error', function (e) {
  console.error(e);
}).end();

It is definitely an error with node itself. If the problem were with squid, it would just result in responses with statusCode 501.

I am running v0.6.6 on Windows 7.

[koichik]

I think that you should use CONNECT method with HTTP (not HTTPS) if you want to use end-to-end HTTPS through a proxy. Unfortunately, Node does not support it well.

[chowey]

Okay, thanks for the pointer.

I played some more with this. I made a simple tcp echo server, which just passes along requests to the squid proxy, like so:

var net = require('net');

net.createServer(function (upstream) {
  var downstream = net.connect(3128, function () {
    upstream.on('data', function (data) {
      console.log(data.toString());
      downstream.write(data);
    });
    upstream.on('end', function () {
      downstream.end();
    });
  });
  downstream.on('data', function (data) {
    upstream.write(data);
  });
  downstream.on('end', function () {
    upstream.end();
  });
}).listen(3000);

This logs the request so I can see what request headers my browser sends for https through a proxy. All the browser does is send the following headers (for https://www.google.com):

CONNECT www.google.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:9.0.1) Gecko/20100101 Firefox/9.0.1
Proxy-Connections: keep-alive
Host: www.google.com

Logically then I could reproduce these in node like so:

var http = require('http');

var options = {
  hostname: '127.0.0.1',
  port: 3128,
  path: 'www.google.com:443',
  headers: {
    'User-Agent': 'Node.js/0.6.6',
    'Proxy-Connections': 'keep-alive',
    'Host': 'www.google.com'
  },
  method: 'CONNECT'
};

var req = http.request(options, function (res) {
  console.log('statusCode: ', res.statusCode);
  console.log('headers: ', res.headers);

  res.on('data', function (chunk) {
    process.stdout.write(chunk);
  });
}).on('error', function (e) {
  console.error(e);
}).end();

It gives me a statusCode 411 from squid now. Per squid, it received the following headers:

CONNECT / HTTP/1.1
User-Agent: Node.js/0.6.6
Proxy-Connections: keep-alive
Host: www.google.com
Transfer-Encoding: chunked

It seems to me that the path got truncated from www.google.com:443 to just /.

Why is node nuking my path for the CONNECT method?

[koichik]

@chowey - Probably, squid displays a path without host/port. Actually, Node sends valid request line:

CONNECT www.google.com:443 HTTP/1.1

The problem may be Transfer-Encoding. You can prevent it:

-    'Host': 'www.google.com'
+    'Host': 'www.google.com',
+    `Content-Length`: 0

result:

$ ./node /tmp/2474.js
statusCode:  200
headers:  {}

I think that Node should not add Transfer-Encoding if CONNECT method is used. And should provide a socket to the user like Upgrade. Then, TLS and HTTPS modules should be able to accept the socket from the user... We need a lot of work :-<

[chowey]

I figured you might use options.createConnection to provide the custom socket. But in the https module:

78  options.createConnection = createConnection;

they just overwrite any createConnection I provide. It isn't like that for the http module.

If they did something instead like:

78  options.createConnection = options.createConnection || createConnection;

then I could provide a custom socket. My createConnection formula might be something like:

function createConnection(port, host, options) {
  var socket = net.connect(port, host, function () {
    socket.write('CONNECT www.google.com:443 HTTP/1.1\r\n' +
        'User-Agent: Node.js/0.6.6\r\n' +
        'Proxy-Connections: keep-alive\r\n' +
        'Host: www.google.com\r\n\r\n',
    function () {
      return socket;
    });
  });
}

(probably this function needs more work, but you get the idea.)

[koichik]

I have noticed that http-parser handles CONNECT method request as a Upgrade request. So, we can handle it using 'upgrade' listener not 'request' listener on http.Server.

[koichik]

@bnoordhuis - Can you review koichik/node@47adb2e and koichik/node@a635acb?

[bnoordhuis]

@koichik: I don't really like the concept, it adds a lot of extra API for a relatively minor issue. What are the alternatives?

[koichik]

@bnoordhuis - Thanks for the suggestion, how about like this:

var req = https.get({
  host: 'github.com',
  ...
  agent: new https.Agent({ // HTTPS
    httpTunnel: {          // over HTTP
      host: 'localhost',
      port: 3128
    }
  }
}, function(res) {
  ...
});

Or, should not Agent support tunneling?

[bnoordhuis]

That looks much simpler.

should not Agent support tunneling?

I'm undecided.

Yes, because HTTP is a first class citizen in Node.

No, because tunneling is complex. You start with simple CONNECT support, next thing you know you'll need to add various authentication methods, SOCKS, DNS-over-SOCKS, workarounds for proxy bugs, etc.

@isaacs, @piscisaureus, @ry: What do you think?

[chowey]

If I can add my $0.02, if the authentication methods, etc. etc. can be done in userland, then they can be done in userland.

The problem with CONNECT support was it could not be done in userland because the node core's http/https modules did not really permit it.

I don't know if the Agent support is really too much "fluff" above the node core, but I don't think it will create a snowball effect of needing to support proxy bugs, etc. Because all that complexity stuff, if the core API is solid and adaptable, can be pushed off to userland.

[koichik]

@chowey - Sorry for the delay, with current master you can use HTTPS via proxy like this:

var http = require('http');
var https = require('https');

var connectReq = http.request({ // establishing a tunnel
  host: 'localhost',
  port: 3128,
  method: 'CONNECT',
  path: 'github.com:443',
}).on('connect', function(res, socket, head) {
  // should check res.statusCode here
  var req = https.get({
    host: 'github.com',
    socket: socket, // using a tunnel
    agent: false    // cannot use a default agent
  }, function(res) {
    res.setEncoding('utf8');
    res.on('data', console.log);
  });
}).end();

So, Agent support may not be MUST...

[chowey]

Wow, excellent.

[isaacs]

Here's my $0.02:

Patch @mikeal's request lib to support this kind of tunneling with the same API that it uses for http->http or https->https proxying. I would like this to Just Work for npm anyway, and I think he'd probably accept a pull req to make it work.

If the requests for this go away, great. If something is blocked there, then we can discuss how to address that.

The benefit of doing it in userland is that we can have the feature much sooner and experiment on it faster.

For 0.7, we can discuss the best api to pull in proxying, if necessary. I quite like request's approach of doing:

request.get({ url: "http://blahblah.com/", proxy: "http://the-proxy:2314" })

Maybe we could add a "proxy" member to the config object that gets passed to http.request and https.request. But that can't land sooner than 0.7, of course, and it might even be a good idea to put it off until 0.8 if we uncover issues with it.

[mikeal]

yes, i'll accept a pull request that adds support for https proxies.

On Jan 13, 2012, at January 13, 201210:47 AM, Isaac Z. Schlueter wrote:

Here's my $0.02:

Patch @mikeal's request lib to support this kind of tunneling with the same API that it uses for http->http or https->https proxying. I would like this to Just Work for npm anyway, and I think he'd probably accept a pull req to make it work.

If the requests for this go away, great. If something is blocked there, then we can discuss how to address that.

The benefit of doing it in userland is that we can have the feature much sooner and experiment on it faster.

For 0.7, we can discuss the best api to pull in proxying, if necessary. I quite like request's approach of doing:

request.get({ url: "http://blahblah.com/", proxy: "http://the-proxy:2314" })

Maybe we could add a "proxy" member to the config object that gets passed to http.request and https.request. But that can't land sooner than 0.7, of course, and it might even be a good idea to put it off until 0.8 if we uncover issues with it.

---

Reply to this email directly or view it on GitHub:
#2474 (comment)

[tjanczuk]

@koichik Is this feature going to be interagted into v0.6.x?

[koichik]

@isaacs, @mikeal - Thanks, some blockers have already fixed in c1a63a9, 08a91ac and 7dffbaf. The CONNECT method works now. Okay, Let's move Agent support to userland. I will send a PR to @mikeal's request or create a new module.

@tjanczuk - No, because the patches change some behavior, especially the event emitted by a CONNECT method was changed into 'connect' from 'request' (client side) or 'upgrade' (server side).

Closing.

[chowey]

Is it just me or is the current master failing to build?

[koichik]

@chowey - Please try:

$ make distclean && ./configure && make

[chowey]

Alas I am using Winblows 7 with VC++ Express.

[chowey]

I get the following error:

NameError: name 'node_use_isolates' is not defined while evaluating condition 'node_use_isolates=="true"' in node.gyp while trying to load node.gyp

What's this isolates business all about?

[chowey]

No worries, found the solution in #2452:

python configure --without-isolates, then vcbuild

[lucasguaru]

I was wondering if the proxy settings should be hard coded.
Maybe where I'm developing I need to use proxy but when deploying it, it will no longer be necessary.
In this case, would be better if I set up proxy config when starting the application:
node -proxy-config index.js

[Vikasg7]

I have this same issue with my proxies (buycheapproxies.org). I was never able to use them with nodejs with the code that worked perfectly with other proxies (fineproxy.org). I tried @koichik code above but got socket hung up when used www.mysite.com:443 as path in the CONNECT request but when I use mysite.com:443 (ie. without www), requests gets successful but I get a different version of the website that doesn't match with what it looks in the browser. Same proxies work perfectly fine in the chrome and curl but not in the node.js. My Node.js version is 7.x.
Not sure if it makes difference but bugcheapproxies (not working) have different ports like 51822, 51323 but fineproxy(working) have same port that is 8085.
The website I am trying to access is on https. I was using request module originally but have tested the core https and http modules as well.

Can somebody guide in the correct direction ?