Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add noproxy configuration #2873

Closed
wants to merge 1 commit into from
@vvision

When fetching, check if the url requested must use a proxy (if present) or not.
This allows package installation from lan without having to change/disable the proxy parameter each time.

"noproxy" is a string containing hostnames. When fetching a package from one of these hostnames, the specifed proxy won't be used.
So, this add a way to specify a list of hostnames that will not ever go through a proxy.

Note: Most relevant addition can be found in lib/utils/fetch.js.

@vvision vvision referenced this pull request in npm/npmconf
Closed

Add noproxy configuration #6

@isaacs
Owner

I'm not interested in taking on mocha/chai as a dependency. There's already a test framework that npm uses (tap) and a bunch of packages in test/packages that will be installed and have their test scripts run. This should not be a 50,000 line change.

@vvision

You're right. I think it's much better now.
I've removed all the useless stuff I had added: mocha, chai and express.
Furthermore, I've put my test npm-test-noproxy in test/packages.
Contrary to one of the commit, I'm not using tap anymore.

@VirgileD

+1 (and more)

lib/utils/fetch.js
((15 lines not shown))
- , proxy: proxy
- , strictSSL: npm.config.get("strict-ssl")
- , ca: remote.host === regHost ? npm.config.get("ca") : undefined
- , headers: { "user-agent": npm.config.get("user-agent") }}
- var req = request(opts)
- req.on("error", function (er) {
- fstr.emit("error", er)
- })
- req.pipe(fstr)
- return req;
+ , proxy: proxy
+ , strictSSL: npm.config.get("strict-ssl")
+ , ca: remote.host === regHost ? npm.config.get("ca") : undefined
+ , headers: { "user-agent": npm.config.get("user-agent") }}
+
+ var req = request(opts)
@isaacs Owner
isaacs added a note

Why is this indentation changed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@isaacs
Owner

This is good for the npm bit. However there are two other pieces:

  1. the config needs to be added to https://github.com/isaacs/npmconf with a default and type.
  2. in https://github.com/isaacs/npm-registry-client, it needs to respect the value in lib/request.js, so that registry urls will respect NO_PROXY if the registry is on that list.
@feugy

:+1:

Please include this pull request.
We daily use dependencies that are accessible only within your society network, and have to change npm configuration very frequently (which is a real pain under windows because of its security management).

It will be very handy to have this configuration options !

Thank you

@vvision

Concerning npmconf, I think this pull request now make sense: npm/npmconf#6.

Associated Pull Request;
npm/npmconf#13
npm/npm-registry-client#11

@vvision vvision referenced this pull request in npm/npm-registry-client
Closed

Add noproxy configuration #11

@Filirom1

I am under a proxy at work and I can say that this pull request works like a charm.

Here is what I did to use it:

[~]> git clone https://github.com/vvision/npm
[~/npm]> npm install -g
[~/npm]> npm config -g set noproxy priv.repo
@vvision vvision referenced this pull request in npm/npmconf
Closed

Add noproxy configuration #13

@vvision

I've made a clean new pull request concerning the configuration needed in npmconf.
Is it good for npmconf and npm-registry-client?

@VirgileD

still using the fork, would be easier if integrated in the main npm package.
thx++

@pouicr

+4

It's so annoying to change configuration every time and not be able to easily install something that requires both priv and pub dependencies...

@domenic
Collaborator

This seems pretty reasonable. Sorry @vvision for us taking so long to get to this.

Do you think it has addressed all the feedback? Can you rebase on master and squash into a single commit (no merge commits)?

@vvision

I think it should now address all the feedback.

I wasn't familiar with rebasing but it seems I've managed to remove merge commits.
I must admit it's more readable with a single commit.

lib/utils/fetch.js
@@ -73,9 +73,12 @@ function makeRequest (remote, fstr, headers) {
"Auth required and none provided. Please run 'npm adduser'"))
}
- var proxy
- if (remote.protocol !== "https:" || !(proxy = npm.config.get("https-proxy"))) {
- proxy = npm.config.get("proxy")
+ var proxy = null
+ var noproxy = npm.config.get("noproxy") ? npm.config.get("noproxy") : ""
+ if(noproxy.search(remote.hostname) === -1) {
@domenic Collaborator
domenic added a note

.search is not a good idea here, because it will convert its argument to a regular expression (so e.g. if remote.hostname is foo.bar.com, and noproxy contains fooxbarxcom.com, it will match). indexOf is better.

@vvision
vvision added a note

You're right. This should work better:

var proxy = null
var noproxy = npm.config.get("noproxy") ? 
  npm.config.get("noproxy").replace(/\s/g, "").split(",") : []
if(noproxy.indexOf(remote.hostname) === -1) {

Like that, it's possible to have multiple noproxy url in the conf like "foo.bar.com, example.com".

@domenic Collaborator
domenic added a note

I think space-separated is probably better than comma-separated? But I am not sure what these sorts of things usually use.

@vvision
vvision added a note

I think we can keep comma here.
I'm using GNU Wget manual about environment variables concerning proxies as a reference.

no_proxy
This variable should contain a comma-separated list of domain extensions proxy should not be used for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@domenic
Collaborator

LGTM but I cannot merge due to #3948; tagging as ready-to-merge

@vvision

Added the modification we discussed in the last outdated diff.

@PaulMougel

Any news?

@Filirom1

Please merge, this pull request will make my life easier :)

@luk-

We should be able to merge this now if everything passes with it.

@rlidwka

There is nothing wrong with noproxy config line, but no_proxy environment variable is kinda standard, and it is defined a bit differently. Not as a list of hosts, but a list of domain suffixes.

from wget manual:

no_proxy
This variable should contain a comma-separated list of domain extensions proxy should not be used for. For instance, if the value of no_proxy is `.mit.edu', proxy will not be used to retrieve documents from MIT.

I recently tried to implement this in another package, and it's a bit more tricky than this one.

So, creating different implementation of the same thing is a bit undesirable.

@goddabuzz

We could implement the Java notation and rename it to a more npm standard name non-proxy-hosts

See: http://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html

http.nonProxyHosts (default: localhost|127.*|[::1])

Indicates the hosts that should be accessed without going through the proxy. Typically this defines internal hosts. The value of this property is a list of hosts, separated by the '|' character. In addition the wildcard character '' can be used for pattern matching. For example -Dhttp.nonProxyHosts=”.foo.com|localhost” will indicate that every hosts in the foo.com domain and the localhost should be accessed directly even if a proxy server is specified.

The default value excludes all common variations of the loopback address.

@othiym23 othiym23 added review and removed ready-to-merge! labels
@gaboom

++1

@vvision vvision referenced this pull request in npm/npm-registry-client
Open

Add noproxy configuration. #69

@gfinger

I would urgently need this feature. We live behind a company firewall and have some referenced modules external, needing the proxy, others internal, which must not use the proxy. Can anybody roughly estimate how long it will take to have a no_proxy feature in npm?

@mattiasrunge

@gfinger if it is urgent you could patch it in yourself I think. I have been running a patched version of NPM for a year now at my company which also has some internal servers and some external.

in node-0.10.31/lib/node_modules/npm/lib/utils/fetch.js replace the makeRequest function with this:

function makeRequest (remote, fstr, headers) {
  remote = url.parse(remote)
  log.http("GET", remote.href)
  regHost = regHost || url.parse(npm.config.get("registry")).host

  if (remote.host === regHost && npm.config.get("always-auth")) {
    remote.auth = new Buffer( npm.config.get("_auth")
                            , "base64" ).toString("utf8")
    if (!remote.auth) return fstr.emit("error", new Error(
      "Auth required and none provided. Please run 'npm adduser'"))
  }

  // no_proxy patch https://github.com/isaacs/npm/pull/2873
  var proxy = null
  var noproxy = npm.config.get("noproxy") ? 
    npm.config.get("noproxy").replace(/\s/g, "").split(",") : []
  if(noproxy.indexOf(remote.hostname) === -1) {
    if (remote.protocol !== "https:" || !(proxy = npm.config.get("https-proxy"))) {
      proxy = npm.config.get("proxy")
    }
  }

  var sessionToken = npm.registry.sessionToken
  if (!sessionToken) {
    sessionToken = crypto.randomBytes(8).toString("hex")
    npm.registry.sessionToken = sessionToken
  }

  var ca = remote.host === regHost ? npm.config.get("ca") : undefined
  var opts = { url: remote
             , proxy: proxy
             , strictSSL: npm.config.get("strict-ssl")
             , rejectUnauthorized: npm.config.get("strict-ssl")
             , ca: ca
             , headers:
               { "user-agent": npm.config.get("user-agent")
               , "npm-session": sessionToken
               , referer: npm.registry.refer
               }
             }
  var req = request(opts)
  req.on("error", function (er) {
    fstr.emit("error", er)
  })
  req.pipe(fstr)
  return req
}

And in node-0.10.31/etc/npmrc:

noproxy="my.internal.server.com,my.other.internal.server.com"
strict-ssl=false

And in our environment we have:

http_proxy := http://www-proxy.internal.com:8080
no_proxy := internal.com

I guess the no_proxy in the environment might be unused, but for completeness we have it for other apps.

@othiym23 othiym23 added the enterprise label
@gfinger

Thanks for your answer. I'm on v0.10.31 but there is no fetch.js in the path you specified. I found it under this location

/usr/local/lib/node_modules/npm/node_modules/npm-registry-client//lib/fetch.js

When I substituted the makeRequest function with your code above I got a lot of errors when running npm update. It seems not to fit to the rest of the file. But based on your suggestion I will try to fix the existing file in a minimal invasive way.

Still I wonder, when on can expect to get this functionality in the released npm. The original request is two years old. Why can the fix not be transferred to the master branch of npm?

@vvision

Indeed, the makeRequest function moved to npm-registry-client and proxy related settings can be found in lib/initialize.js.

So I now have a new ongoing pull request on npm/npm-registry-client#69.

It would be great if @othiym23 could provide us with some feedbacks.

@othiym23
Owner

@vvision I only have so much time. ;)

@isaacs and I walked through what we think is the best way to get this functionality into npm. While the functionality is implemented in npm-registry-client, the configuration is handled in npm and npmconf, so they'd need updates as well. @isaacs suggested – and I agree – that we should follow the precedent established by wget and curl, and do exactly what they're doing (which means using the same name they do as well, which in this case is no_proxy (wget) / NO_PROXY (curl)).

The most important suggestion we have is this: this work should actually be pushed onto request, because of the way request deals with redirects. If you have a request going through a proxy, and it encounters a redirect, request will handle that transparently. In the case where the original host is not on the no_proxy list, but the redirect target is, if request isn't no_proxy aware, it'll just naïvely redirect. This is an edge case, but there are large internet companies where this is done.

If you file an issue / submit a PR on request for this functionality (i.e. it takes a noProxy option that is a JavaScript array of hostnames to not match), it is likely to be speedily integrated. Then getting it into npm itself is a pretty straightforward matter of bumping the request version and making sure the option gets passed to request in npm-registry-client, and doing the necessary configuration handling, which would happen here and in npmconf.

Good luck and happy hunting! This would be a great feature to have in npm!

@gfinger

As far as I understand the node coding, the request module is indeed the best place to include the proxy handling. This way all usages of the request module would profit from it.

It would be cute, if the request could just evaluate the settings of the http_proxy, https_proxy and np_proxy variables from the environment. Having the proxy settings in the npm-configuration as it is currently is definitely not optimal, as it redundantly duplicates settings maintained elswhere.

@othiym23
Owner

@gfinger We discussed that, because that was my thought as well, but for other reasons npm itself needs to know about the no-proxy list, so the behavior would have to be overrideable from outside request.

There's also the matter of request suddenly sprouting an implicit environmental dependency, which has the potential to surprise people, but that's request's concern, not ours. ;)

@vvision

@othiym23 Thanks!

@tauren

request/request#1096

request@2.45.0 was recently updated to include no_proxy support, and npm@2.1.4 has it as a dependency. This issue can probably be closed.

@othiym23
Owner

@tauren request@2.45.0 was included in npm@2.1.5, not npm@2.1.4, so it's not in the latest stable version of npm, but you're right that this is now fixed in npm. I'll try to get to your PR shortly, but that won't land until npm@2.1.6, so it will be a week and a half or so until everything's copacetic in stable npm. Either way, thanks to everyone for their patience!

@othiym23 othiym23 closed this
@othiym23 othiym23 removed the review label
@tauren

@othiym23 thanks for clarifying the version, that was a typo on my part. I look forward to npm@2.1.6.

@othiym23
Owner

See the PR – there's a fairly substantial amount of work to be done on it, and I could really use your help if we're to land this this week.

@gratex

Has anyone tried this feature?

It does not work for me. I have tried to debug it, and I have found that the code which is responsible for handling NO_PROXY, is never called:

NO_PROXY is handled in getProxyFromURI method in request/request.js, which is called only if
proxy property is not defined in passed options:

if(!self.hasOwnProperty('proxy')) {
    self.proxy = getProxyFromURI(self.uri)
}

(If proxy property is not set, it will be be read from ENV variable as well as no_proxy)

But npm passes options to requset, which always contains proxy property; see npm-registry-client/lib/initialize.js

The proxy property is set to value defined in configuration or defaults to ENV variable or NULL; see lib/config/defaults.js.

@othiym23
Owner

@gratex npm-registry-client could work around this (by only adding a proxy property to the options when there is a proxy or https-proxy option configured in npm), but this looks like an overly restrictive check in request. if (!self.proxy) ... would be what I would expect it to do. Nevertheless, if you want npm-registry-client to work around this behavior, file an issue over there, and that work can be landed with @tauren's work, assuming we get that in landable shape soon. (I personally think this should be fixed in request as well as npm.)

@gratex gratex referenced this pull request in npm/npm-registry-client
Closed

Send proxy property to request only if configured #82

@tauren

I agree that the check in request seems overly restrictive, but I just discovered that when proxy === null, request forces no proxy to be used. This is why if (!self.proxy) ... is not used.

At this point, I'm not sure where it should be fixed. Maybe doing it in npm-registry-client would be best for now. I'm hoping @othiym23 and @nylen might have some ideas on a good solution.

@othiym23 othiym23 referenced this pull request from a commit in npm/npm-registry-client
@othiym23 othiym23 request will handle proxy by itself (fixes 82) e2880c0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 30, 2013
  1. @vvision

    Added noproxy configuration

    vvision authored
This page is out of date. Refresh to see the latest.
View
7 doc/misc/npm-config.md
@@ -483,6 +483,13 @@ Any "%s" in the message will be replaced with the version number.
The node version to use when checking package's "engines" hash.
+### noproxy
+
+* Default: `NO_PROXY` or `no_proxy` environment variable, or "null"
+* Type: String
+
+Hostnames which won't use proxy when being requested.
+
### npat
* Default: false
View
10 lib/utils/fetch.js
@@ -73,9 +73,13 @@ function makeRequest (remote, fstr, headers) {
"Auth required and none provided. Please run 'npm adduser'"))
}
- var proxy
- if (remote.protocol !== "https:" || !(proxy = npm.config.get("https-proxy"))) {
- proxy = npm.config.get("proxy")
+ var proxy = null
+ var noproxy = npm.config.get("noproxy") ?
+ npm.config.get("noproxy").replace(/\s/g, "").split(",") : []
+ if(noproxy.indexOf(remote.hostname) === -1) {
+ if (remote.protocol !== "https:" || !(proxy = npm.config.get("https-proxy"))) {
+ proxy = npm.config.get("proxy")
+ }
}
var opts = { url: remote
View
17 test/packages/npm-test-noproxy/README
@@ -0,0 +1,17 @@
+Test noproxy configuration.
+
+
+We create a server and a proxy.
+Server listens on localhost:80.
+Proxy listens on localhost:8080.
+The proxy redirects all requests to /proxy on the server.
+On the server, /proxy sends back proxy.tar.gz .
+When server is directly requested, it answers with noproxy.tar.gz .
+
+
+So we perform 2 tests, both with proxy equal to "http://localhost:8080".
+-A test is performed with noproxy equal to "null". In this case, npm
+ installs the package named "proxy.tar.gz" because the proxy is used.
+-In the other test, noproxy equal "localhost, example.com".
+ Since localhost is part of noproxy, request is made directly
+ to the server and proxy is ignored.
View
1  test/packages/npm-test-noproxy/noproxy-package/README.md
@@ -0,0 +1 @@
+Package for npm-test-noproxy.
View
6 test/packages/npm-test-noproxy/noproxy-package/package.json
@@ -0,0 +1,6 @@
+{
+ "author": "vvision",
+ "name": "test-noproxy-test",
+ "description": "Dummy package for noproxy configuration tests",
+ "version": "10.10.10"
+}
View
BIN  test/packages/npm-test-noproxy/noproxy.tar.gz
Binary file not shown
View
7 test/packages/npm-test-noproxy/package.json
@@ -0,0 +1,7 @@
+{ "name":"npm-test-noproxy"
+, "version" : "1.0.0"
+, "scripts" :
+ { "install" : "node test.js"
+ , "test" : "node test.js"
+ }
+}
View
1  test/packages/npm-test-noproxy/proxy-package/README.md
@@ -0,0 +1 @@
+Package for npm-test-noproxy.
View
6 test/packages/npm-test-noproxy/proxy-package/package.json
@@ -0,0 +1,6 @@
+{
+ "author": "vvision",
+ "name": "test-proxy-test",
+ "description": "Dummy package for noproxy configuration tests",
+ "version": "7.7.7"
+}
View
BIN  test/packages/npm-test-noproxy/proxy.tar.gz
Binary file not shown
View
109 test/packages/npm-test-noproxy/test.js
@@ -0,0 +1,109 @@
+var npm = require("npm")
+ , npmconf = require("npmconf")
+ , assert = require("assert")
+ , path = require('path')
+ , request = require('request')
+ , fs = require('graceful-fs')
+ , http = require('http')
+ , nopt = require('nopt')
+ , log = require("npmlog")
+
+ , configDefs = npmconf.defs
+ , shorthands = configDefs.shorthands
+ , types = configDefs.types
+ , conf = nopt(types, shorthands)
+ //We create a server and a proxy (see README).
+ , server = http.createServer(function(req, res){
+ res.statusCode = 200
+ if(req.url == '/proxy') {
+ // opens file in read, with a stream
+ fs.createReadStream(__dirname+'/proxy.tar.gz').on('error', function(err){
+ // something goes wrong while reading
+ log.warn(err)
+ res.statusCode = 404
+ res.end()
+ }).pipe(res)
+ // read data are piped into the response
+ } else {
+ fs.createReadStream(__dirname+'/noproxy.tar.gz').on('error', function(err){
+ log.warn(err)
+ res.statusCode = 404
+ res.end()
+ }).pipe(res)
+ }
+ })
+ , proxy = http.createServer(function (req, res) {
+ res.statusCode = 200
+ var url = 'http://localhost:80/proxy'
+ var x = request(url)
+ req.pipe(x)
+ x.pipe(res)
+ })
+
+npm.load(conf, function (er, npm) {
+ if(er) log.warn(er)
+
+ //Set proxy configuration, launch server and proxy
+ var initialize = function (cb) {
+ npm.config.set('proxy', 'http://localhost:8080')
+ server.listen(80, 'localhost', function () {
+ proxy.listen(8080, 'localhost', cb)
+ })
+ }
+
+ //Tests
+ initialize(function () {
+ assert.equal('http://localhost:8080', npm.config.get('proxy'))
+ //Checking the route for server and proxy
+ request.get("http://localhost:80/test", function (err, res, body) {
+ assert.equal(res.statusCode, 200)
+ request.get("http://localhost:80/proxy", function (err, res2, body) {
+ assert.equal(res2.statusCode, 200)
+ request.get("http://localhost:8080/test", function (err, res3, body) {
+ assert.equal(res3.statusCode, 200)
+ makeNoProxyTest(function () {
+ makeProxyTest(function () {
+ closeServer(server)
+ closeServer(proxy)
+ })
+ })
+ })
+ })
+ })
+ })
+
+ //Use npm with noproxy
+ var makeNoProxyTest = function (cb) {
+ //Set noproxy configuration
+ npm.config.set('noproxy', 'localhost , example.com')
+ assert.equal('localhost , example.com', npm.config.get('noproxy'))
+ //Install from localhost:80/test without proxy
+ npm.commands.install(['http://localhost:80/test'], function () {
+ //Let's check if the package directory exists in node_modules
+ fs.exists('../test-noproxy-test', function (exists) {
+ assert.equal(exists, true)
+ cb()
+ })
+ })
+ }
+
+ //Use npm with proxy
+ var makeProxyTest = function (cb) {
+ //Set proxy configuration
+ npm.config.set('noproxy', 'null')
+ assert.equal('null', npm.config.get('noproxy'))
+ //Install from localhost:80/test with proxy
+ npm.commands.install(['http://localhost:80/test'], function () {
+ //Let's check if the package directory exists in node_modules
+ fs.exists('../test-proxy-test', function (exists) {
+ assert.equal(exists, true)
+ cb()
+ })
+ })
+ }
+
+ var closeServer = function (s) {
+ s.close()
+ }
+
+})
Something went wrong with that request. Please try again.