Problems proxying HTTPS through an HTTP proxy #1065

jason0x43 · 2020-01-23T15:10:43Z

Intern 4.x and 5-pre may have problems proxying HTTPS connections through an HTTP proxy. This axios PR suggests that some configuration may be required to achieve this.

Verify that HTTPS connections through HTTP proxies are failing, and if so, provide a way to tell Intern to tell Axios that a proxy supports HTTPS.

jason0x43 · 2020-01-23T15:44:51Z

A fix for this should also be back ported to https://github.com/theintern/common.

references theintern/intern#1065

jason0x43 · 2020-01-23T15:51:00Z

@theintern/common@2.4.0-beta.0 has been published with a possible fix. To test it, npm install @theintern/common@2.4.0-beta.0 and manually remove node_modules/intern/node_modules/@theintern/common, then try running tests.

jason0x43 · 2020-01-23T21:41:25Z

This is likely related to theintern/digdug#65

Resolve the remote environments (resolving aliases, normalizing environment descriptors) in resolveConfig. Defer parameter aliasing (e.g., ensuring both browser and browserName are present) until after environment resolution so we don't accidentally create too many environment permutations. resolves theintern#1065 resolves theintern#1063

amrtn · 2020-01-28T13:16:22Z

Hi, here are detailed answers to the questions you posted on gitter

I was hoping the Axios proxy protocol flag would help out here. So, just to be sure I know what's going on...

You're trying to make requests to https URLs through an http proxy

That's correct

The proxy requires authentication

Using @theintern/common@0.2.3 works through a local http proxy (and you're still making requests to https URLs)

Yes, common@0.2.3 works with the local http proxy and making requests to https URLS. This is my actual test:

Configure intern to use local Fiddler (http://localhost:8888 that needs no authentication)

"tunnelOptions": {
  "proxy": "http://localhost:8888"
}

Execute

$ rm -rf node_modules
$ npm install
$ npm ls @theintern/common
dojo-intern-example@0.2.0 D:\AMO\dev\learn\intern\intern-examples\dojo-example
`-- intern@4.8.0
  +-- @theintern/common@0.2.3 
  +-- @theintern/digdug@2.4.3
  | `-- @theintern/common@0.2.3
  `-- @theintern/leadfoot@2.3.0
    `-- @theintern/common@0.2.3

$ npm test

Intern performs this http request and executes the tests successfully.

curl -k -i --raw -o 0.dat "https://selenium-release.storage.googleapis.com/3.141/selenium-server-standalone-3.141.59.jar" -H "Accept: application/json, text/plain, */*" -H "User-Agent: axios/0.19.2" -H "host: selenium-release.storage.googleapis.com" -H "Connection: close"

With 0.2.3, making an HTTPS request through the authenticating HTTP proxy gives you an error like Error: [GET https://selenium-release.storage.googleapis.com/3.5/selenium-server-standalone-3.5.2.jar] socket hang up

That's correct. This is my actual test:

Configure intern to use the external proxy

"tunnelOptions": {
  "proxy": "http://domain%5Cusername:password@proxy.zzz.com:3128"
}

Execute

$ rm -rf node_modules
$ npm install
$ npm ls @theintern/common
dojo-intern-example@0.2.0 D:\AMO\dev\learn\intern\intern-examples\dojo-example
`-- intern@4.8.0
  +-- @theintern/common@0.2.3 
  +-- @theintern/digdug@2.4.3
  | `-- @theintern/common@0.2.3
  `-- @theintern/leadfoot@2.3.0
    `-- @theintern/common@0.2.3

$ npm test
> dojo-intern-example@0.2.0 test D:\AMO\dev\learn\intern\intern-examples\dojo-example
> intern

Listening on localhost:9000 (ws 9001)
(ノಠ益ಠ)ノ彡┻━┻
Error: socket hang up
  at createHangUpError @ _http_client.js:323:15
  at Socket.socketOnEnd @ _http_client.js:426:23
  at Socket.emit @ events.js:203:15
  at endReadableNT @ _stream_readable.js:1145:12
  at process._tickCallback @ internal\process\next_tick.js:63:19

npm ERR! Test failed.  See above for more details.

Using @theintern/common@0.2.4-beta.0doesn't work through the local http proxy or the external one

With 0.2.4-beta.0, making an HTTPS request through the authenticating HTTP proxy gives you an error like Error: Client network socket disconnected before secure TLS connection was established

That's correct. This is my actual test:

Configure intern to use the external (authenticating) proxy

"tunnelOptions": {
  "proxy": "http://domain%5Cusername:password@proxy.zzz.com:3128"
}

Add "@theintern/common": "^0.2.4-beta.0", to the devDependencies in package.json

$ rm -rf node_modules
$ npm install
$ rm -rf node_modules\intern\node_modules\@theintern\common\
$ rm -rf node_modules\@theintern\digdug\node_modules\@theintern\common
$ rm -rf node_modules\@theintern\leadfoot\node_modules\@theintern\common
$ npm ls @theintern/common
dojo-intern-example@0.2.0 D:\AMO\dev\learn\intern\intern-examples\dojo-example
+-- @theintern/common@0.2.4-beta.0 
`-- intern@4.8.0
  +-- UNMET DEPENDENCY @theintern/common@0.2.3
  +-- @theintern/digdug@2.4.3
  | `-- UNMET DEPENDENCY @theintern/common@0.2.3
  `-- @theintern/leadfoot@2.3.0
    `-- UNMET DEPENDENCY @theintern/common@0.2.3

npm ERR! missing: @theintern/common@0.2.3, required by intern@4.8.0
npm ERR! missing: @theintern/common@0.2.3, required by @theintern/digdug@2.4.3
npm ERR! missing: @theintern/common@0.2.3, required by @theintern/leadfoot@2.3.0

$ npm test
> dojo-intern-example@0.2.0 test D:\AMO\dev\learn\intern\intern-examples\dojo-example
> intern

Listening on localhost:9000 (ws 9001)
(ノಠ益ಠ)ノ彡┻━┻
Error: Client network socket disconnected before secure TLS connection was established
  at TLSSocket.onConnectEnd @ _tls_wrap.js:1095:19
  at Object.onceWrapper @ events.js:286:20
  at TLSSocket.emit @ events.js:203:15
  at endReadableNT @ _stream_readable.js:1145:12
  at process._tickCallback @ internal\process\next_tick.js:63:19

npm ERR! Test failed.  See above for more details.

With 0.2.4-beta.0 and a local non-autheticating HTTP proxy, making an HTTPS request gives you an error like Error: write EPROTO 13732:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:c:\ws\deps\openssl\openssl\ssl\record\ssl3_record.c:332:

That's correct. This is my actual test:

Configure intern to use the local proxy

"tunnelOptions": {
  "proxy": "http://localhost:8888"
}

Add "@theintern/common": "^0.2.4-beta.0", to the devDependencies in package.json

$ rm -rf node_modules
$ npm install
$ rm -rf node_modules\intern\node_modules\@theintern\common\
$ rm -rf node_modules\@theintern\digdug\node_modules\@theintern\common
$ rm -rf node_modules\@theintern\leadfoot\node_modules\@theintern\common
$ npm ls @theintern/common
dojo-intern-example@0.2.0 D:\AMO\dev\learn\intern\intern-examples\dojo-example
+-- @theintern/common@0.2.4-beta.0 
`-- intern@4.8.0
  +-- UNMET DEPENDENCY @theintern/common@0.2.3
  +-- @theintern/digdug@2.4.3
  | `-- UNMET DEPENDENCY @theintern/common@0.2.3
  `-- @theintern/leadfoot@2.3.0
    `-- UNMET DEPENDENCY @theintern/common@0.2.3

npm ERR! missing: @theintern/common@0.2.3, required by intern@4.8.0
npm ERR! missing: @theintern/common@0.2.3, required by @theintern/digdug@2.4.3
npm ERR! missing: @theintern/common@0.2.3, required by @theintern/leadfoot@2.3.0

$ npm test
> dojo-intern-example@0.2.0 test D:\AMO\dev\learn\intern\intern-examples\dojo-example
> intern

Listening on localhost:9000 (ws 9001)
(ノಠ益ಠ)ノ彡┻━┻
Error: write EPROTO 18592:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:c:\ws\deps\openssl\openssl\ssl\record\ssl3_record.c:332:

  at WriteWrap.afterWrite [as oncomplete] @ net.js:788:14

npm ERR! Test failed.  See above for more details.

No requests are logged in Fiddler.

jason0x43 · 2020-01-28T16:23:03Z

Using Intern 4.8, common 0.2.3, and mitmproxy running on http://localhost:8080, test runs fail with

Error: Download server returned status code 400 for https://chromedriver.storage.googleapis.com/79.0.3945.36/chromedriver_mac64.zip
  @ node_modules/src/Tunnel.ts:400:21
  @ node_modules/@theintern/common/index.js:670:29
  at processTicksAndRejections @ internal/process/task_queues.js:97:5

The mitm log indicates a protocol error:

  -> HTTP protocol error in client request: Invalid request scheme: https

It looks like Axios may be making a request to https://<proxyhost> rather than http://proxyhost.

jason0x43 · 2020-01-28T16:41:22Z

Or that could just be a problem with mitm proxy. Proxies are kind of a pain. 😄

jason0x43 · 2020-01-28T19:04:02Z

Well, after making a small modification to mitmproxy (it's a bit picky about URL schemas) by commenting out this block, connections to https addresses go through when using @theintern/common@0.2.3.

jason0x43 · 2020-01-28T19:05:00Z

That doesn't actually do much to solve the problem, but it suggests that the issue is with specific proxy configurations rather than proxy support being generally broken.

Resolve the remote environments (resolving aliases, normalizing environment descriptors) in resolveConfig. Defer parameter aliasing (e.g., ensuring both browser and browserName are present) until after environment resolution so we don't accidentally create too many environment permutations. resolves theintern#1065 resolves theintern#1063

Resolve the remote environments (resolving aliases, normalizing environment descriptors) in resolveConfig. Defer parameter aliasing (e.g., ensuring both browser and browserName are present) until after environment resolution so we don't accidentally create too many environment permutations. resolves #1065 resolves #1063

jason0x43 added bug Something that's not working as intended domain-tunnels Related to Intern's webdriver tunnels (formerly Dig Dug) and removed domain-tunnels Related to Intern's webdriver tunnels (formerly Dig Dug) labels Jan 23, 2020

jason0x43 added a commit to theintern/common that referenced this issue Jan 23, 2020

Set proxy protocol for https requests

d11b8b2

references theintern/intern#1065

jason0x43 added the repo-common Resolution applies to @theintern/common label Jan 23, 2020

jason0x43 mentioned this issue Jan 24, 2020

dig dug is not connecting with sauce labs correctly when behind http proxy with authentication theintern/digdug#17

Closed

jason0x43 added the repo-digdug Resolution applies to @theintern/digdug label Jan 24, 2020

jason0x43 mentioned this issue Jan 24, 2020

Downloading Selenium and Drivers via HTTPS through Proxy fails theintern/digdug#65

Closed

jason0x43 added priority-medium This should get done, but it's not a high priority effort-low Should be pretty quick labels Jan 24, 2020

jason0x43 added effort-medium This may take a couple of days and removed effort-low Should be pretty quick labels Jan 28, 2020

jason0x43 added this to To do in Development Feb 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problems proxying HTTPS through an HTTP proxy #1065

Problems proxying HTTPS through an HTTP proxy #1065

jason0x43 commented Jan 23, 2020

jason0x43 commented Jan 23, 2020

jason0x43 commented Jan 23, 2020

jason0x43 commented Jan 23, 2020

amrtn commented Jan 28, 2020 •

edited

Loading

jason0x43 commented Jan 28, 2020

jason0x43 commented Jan 28, 2020

jason0x43 commented Jan 28, 2020

jason0x43 commented Jan 28, 2020

Problems proxying HTTPS through an HTTP proxy #1065

Problems proxying HTTPS through an HTTP proxy #1065

Comments

jason0x43 commented Jan 23, 2020

jason0x43 commented Jan 23, 2020

jason0x43 commented Jan 23, 2020

jason0x43 commented Jan 23, 2020

amrtn commented Jan 28, 2020 • edited Loading

jason0x43 commented Jan 28, 2020

jason0x43 commented Jan 28, 2020

jason0x43 commented Jan 28, 2020

jason0x43 commented Jan 28, 2020

amrtn commented Jan 28, 2020 •

edited

Loading