cache isn't working via npm install behind proxy #30

Hotell · 2014-04-03T11:45:59Z

Hi, I'm trying to setup npm_lazy on corporate server ( runs on centOS ) but without any luck.
I've configured the proxy settings in config file.

When I run npm_lazy:

[root@ITAP11 ~] npm_lazy -c ./npm_lazy.config.js
app info npm_lazy at 0.0.0.0 port 9000
app info npm_lazy cache directory: /root/.npm_lazy
app info Using http://171.27.1.254:3128 to proxy http requests.
app info Using https://171.27.1.254:3128 to proxy https requests.

now when i run npm install --registry http://localhost:9000 lodash
this is what I get:

npm http 502 http://localhost:9000/lodash
npm ERR! registry error parsing json
npm ERR! SyntaxError: Unexpected token <
npm ERR! <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
npm ERR! <html><head>
npm ERR! <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
npm ERR! <title>ERROR: The requested URL could not be retrieved</title>
npm ERR! <style type="text/css"><!--
npm ERR!  /*
npm ERR!  Stylesheet for Squid Error pages
npm ERR!  Adapted from design by Free CSS Templates
npm ERR!  http://www.freecsstemplates.org
npm ERR!  Released for free under a Creative Commons Attribution 2.5 License
npm ERR! */
...
...
npm ERR! System Linux 2.6.32-431.5.1.el6.centos.plus.x86_64
npm ERR! command "node" "/usr/bin/npm" "i" "--registry" "http://localhost:9000/" "lodash"
npm ERR! cwd /root/tmp
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.4.6
npm ERR! type unexpected_token
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /root/tmp/npm-debug.log
npm ERR! not ok code 0

but when I run request via wget, it works and correctly saves package to ~/.npm_lazy/
wget http://localhost:9000/lodash

any ideas?

thanks in advance

The text was updated successfully, but these errors were encountered:

mixu · 2014-04-03T16:15:30Z

Do you mean wget http://registry.npmjs.org/lodash works? I'm not sure why there would be a need to use a proxy for localhost but I'm not familiar with your setup,

In any case can you run the wget that works with the -d option which will print request and response headers and post them here? It's probably a difference between the headers npm_lazy is sending vs wget and if wget works then that's encouraging.

Hotell · 2014-04-04T07:21:36Z

Well all our local servers are behind a firewall so when you wanna access internet you have to go through proxy, that's why.

about wget. Nope, I meant calling npm_lazy server, which should check if that package is in npm_lazy cache. If not it calls http://registry.npmjs.org/lodash. So I'm calling wget localhost:9000/lodash to "try" if npm_lazy server is working.

here is the response with -d option that you've requested.

 wget -d localhost:9000/lodash
DEBUG output created by Wget 1.12 on linux-gnu.

--2014-04-04 09:19:02--  http://localhost:9000/lodash
Resolving localhost... ::1, 127.0.0.1
Caching localhost => ::1 127.0.0.1
Connecting to localhost|::1|:9000... Closed fd 3
failed: Connection refused.
Connecting to localhost|127.0.0.1|:9000... connected.
Created socket 3.
Releasing 0x0000000002107390 (new refcount 1).

---request begin---
GET /lodash HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: */*
Host: localhost:9000
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Date: Fri, 04 Apr 2014 07:19:02 GMT
Connection: close

---response end---
200 OK
Length: unspecified
Saving to: “lodash.1”

    [ <=>                                                                                      ] 69,989      --.-K/s   in 0.001s

Closed fd 3
2014-04-04 09:19:02 (111 MB/s) - “lodash.1” saved [69989]

thanks

mixu · 2014-04-04T07:41:13Z

ok, thanks for the clarification. So the issue is that npm_lazy hangs when trying to connect externally in response to your request against localhost. Can you run wget -d http://registry.npmjs.org/lodash and post it here. If that works correctly (check that the file is as expected) through your proxy then I should be able to compare the npm_lazy code with whatever works in your environment.

I am assuming .wgetrc is already configured with the proxy info but if wget -d http://registry.npmjs.org/lodash doesn't work could you figure out the correct wget settings that work for external requests and then post an example of a working set of request headers?

Hotell · 2014-04-04T08:19:15Z

wget -d registry.npmjs.org/lodash
DEBUG output created by Wget 1.12 on linux-gnu.

--2014-04-04 10:15:41--  http://registry.npmjs.org/lodash
Connecting to 171.27.1.254:3128... connected.
Created socket 3.
Releasing 0x0000000001990bd0 (new refcount 0).
Deleting unused 0x0000000001990bd0.

---request begin---
GET http://registry.npmjs.org/lodash HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: */*
Host: registry.npmjs.org

---request end---
Proxy request sent, awaiting response...
---response begin---
HTTP/1.0 200 OK
Age: 3399
Via: 1.1 varnish, 1.1 varnish, 1.0 localhost (squid/3.1.6), 1.0 171.27.1.254 (McAfee Web Gateway 7.4.1.1.0.17024)
Date: Fri, 04 Apr 2014 08:15:41 GMT
ETag: "AJSV8HHNVCYLT3DO196P8YWMI"
Vary: Accept
Server: CouchDB/1.5.0 (Erlang OTP/R14B04)
X-Cache: MISS, MISS
X-Cache: MISS from localhost
X-Timer: S1396595942.354797840,VS0,VS42,VE223,VE3399068
X-Served-By: cache-v41-ASH, cache-am72-AMS
Content-Type: application/json
X-Cache-Hits: 0, 0
Accept-Ranges: bytes
Cache-Control: max-age=1
Content-Length: 70809
X-Cache-Lookup: MISS from localhost:8080
Proxy-Connection: Close

---response end---
200 OK
Length: 70809 (69K) [application/json]
Saving to: “lodash”

100%[===================================================================================================================>] 70,809      --.-K/s   in 0.1s

Closed fd 3
2014-04-04 10:15:41 (632 KB/s) - “lodash” saved [70809/70809]

mixu · 2014-04-04T09:03:12Z

Nothing looks obviously wrong in the headers. Now that I think of it, are you using the npm proxy setting right now on that server? What does: npm config get proxy and npm config get https-proxy say?

Because it would make a lot of sense that npm install --registry http://localhost:9000 lodash would not work IF npm itself is trying to access localhost:9000 through a proxy - which, now that I think of it is what the output looks like e.g. a squid generated page for a local http request.

it looks like this stuff also uses environment variables: https://www.npmjs.org/doc/misc/npm-config.html so maybe try:

HTTP_PROXY= HTTPS_PROXY= npm config get proxy
HTTP_PROXY= HTTPS_PROXY= npm config get https-proxy

those should both return null. If that works then: HTTP_PROXY= HTTPS_PROXY= npm install --registry http://localhost:9000 lodash might work as expected (e.g. no proxy via npmrc, no proxy via env vars, and please use npm_lazy)

Edit: it would kind of make sense that npm does not special case localhost requests when doing proxying, as having the client and the registry on the same machine is not a common case.

Hotell · 2014-04-04T09:32:13Z

ha! works. man, you're the best. Thanks so much! 🎉

summary

yeah I've had proxy config in .npmrc, and also in env variables

so I've commented that out ( so installing packages will be possible only via npm_lazy )
.npmrc

;proxy = http://171.27.1.254:3128/
;https-proxy = https://171.27.1.254:3128/
strict-ssl = false
;registry = http://registry.npmjs.org/
registry = http://localhost:9000

npm_lazy.config.js

 proxy: {
    'http' : 'http://my-proxy-ip:port',
    'https' : 'https://my-proxy-ip:port'
}

installing packages:
HTTP_PROXY= HTTPS_PROXY= npm install lodash

mixu · 2014-04-04T09:51:12Z

awesome, I'll add a note to the readme about this. thanks for your help in debugging this!

kozmaz87 · 2015-04-24T17:11:57Z

This is all well and good until you install a package like imagemin. Which brings in pngquant, which goes to github via https and cannot download stuff.

⚠ connect ETIMEDOUT
⚠ jpegtran pre-build test failed
ℹ compiling from source
⚠ connect ETIMEDOUT
⚠ optipng pre-build test failed
ℹ compiling from source
⚠ Request to https://raw.github.com/imagemin/pngquant-bin/v3.0.0/vendor/linux/x64/pngquant failed
⚠ pngquant pre-build test failed
ℹ compiling from source
⚠ connect ETIMEDOUT
⚠ gifsicle pre-build test failed
ℹ compiling from source
✖ Error: connect ETIMEDOUT
at errnoException (net.js:904:11)
at Object.afterConnect as oncomplete
✖ Error: connect ETIMEDOUT
at errnoException (net.js:904:11)
at Object.afterConnect as oncomplete
✖ Error: connect ETIMEDOUT
at errnoException (net.js:904:11)
at Object.afterConnect as oncomplete
✖ Error: connect ETIMEDOUT
at errnoException (net.js:904:11)
at Object.afterConnect as oncomplete

I am afraid it is not possible to run npm_lazy on localhost because of the utterly stupid way npm handles requests to localhost. There is not a single use-case in this quadrant of our galaxy where requests to localhost aka 127.0.0.1 is supposed to be sent off to a proxy.

mixu closed this as completed Apr 4, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cache isn't working via npm install behind proxy #30

cache isn't working via npm install behind proxy #30

Hotell commented Apr 3, 2014

mixu commented Apr 3, 2014

Hotell commented Apr 4, 2014

mixu commented Apr 4, 2014

Hotell commented Apr 4, 2014

mixu commented Apr 4, 2014

Hotell commented Apr 4, 2014

mixu commented Apr 4, 2014

kozmaz87 commented Apr 24, 2015

cache isn't working via npm install behind proxy #30

cache isn't working via npm install behind proxy #30

Comments

Hotell commented Apr 3, 2014

mixu commented Apr 3, 2014

Hotell commented Apr 4, 2014

mixu commented Apr 4, 2014

Hotell commented Apr 4, 2014

mixu commented Apr 4, 2014

Hotell commented Apr 4, 2014

summary

mixu commented Apr 4, 2014

kozmaz87 commented Apr 24, 2015