npm shrinkwrap results in ECONNRESET errors with 3rd Party Registry (Follow-up)

[simonua]

This is a follow-up to issue 11117.

@iarna, I was able to reproduce the ECONNRESET errors against our ProGet instance and captured everything with silly logging turned on. The attached log file has been scrubbed for our ProGet URL (the organization has been replaced with "contoso") and the username ("foouser"), but everything else should be original.

I was not able to reproduce the 404s, but if I encounter that behavior again, I will either attach or create a new issue.

We make more than 1,000 resource requests for modules as part of the package we are installing. Not to put the cart before the horse, but if this indeed turns out to be a throttling issue, I don't suppose there's much we can do with ProGet at this time, and the two options we may have would be to a) wait until a throttling feature is added to the npm client, or b) use npm private modules or a different registry, correct?

Thanks very much again for checking into this!

npm-silly-scrubbed.txt

[simonua]

Is there a way to see what specific request an ECONNRESET pertains to? I don't see it in silly logging. Looking into this further on the IIS / ProGet side, I see 200s returned for all requests. I realize this is drifting away from npm a little, but any additional information pertaining to what requests fail would be appreciated.

[ro0gr]

We have the same issue using proget and shrinkwrap.
Unfortunatelly I've failed to find a better solution than removing all the 'resolved' fields from npm—shrinkwrap.json. ng2 clean—shrinkwrap may be usefull to automate this task.

[othiym23]

Looking into this further on the IIS / ProGet side, I see 200s returned for all requests.

That implies to me that the ECONNRESETs are happening at the network layer on the ProGet side, before the HTTP handler even sees them. Something that might help is throwing a proxy into the mix, or using a tool like crapify to cut down the concurrency a little bit. Because of how shrinkwrap is designed, it bypasses the local cache, and also requests basically all of the package tarballs at once (which is why removing the resolved fields from npm-shrinkwrap.json works, although that also weakens the guarantee that shrinkwrap makes about the current install producing the same tree as when `npm shrinkwrap was run).

npm will retry on ECONNRESET, and that log makes it look like the install mostly succeeds, but then fails on a cb called twice error, which is a catchall error that is much rarer in npm@>=2, and these days points to an inconsistency in the retry logic. This is a known issue, but it's worth bringing up because it implies that the ECONNRESET errors aren't necessarily a blocker if somebody can extirpate whatever's calling the callback repeatedly. It's not ideal, because having to retry requests will slow down the install, but it should still complete.

Is there a way to see what specific request an ECONNRESET pertains to?

Unfortunately, given the way HTTP request errors bubble through the system (and given the way Node creates those error messages), it's a lot harder than you'd think to get that information on there. It's undeniably useful, so at some point the CLI team would like to put some work into making those error messages more useful, but it's not something we're going to have time to get to soon.

we may have would be to a) wait until a throttling feature is added to the npm client

This is on the roadmap for this year, although realistically it's probably going to be 2016Q2 at the earliest before the team has the cycles available to do the necessary work. That said, give crapify a shot – you can run it on the build machines and configure it like any other transparent HTTP proxy, and some of npm's enterprise customers have had luck with it in situations like these.

Because I don't think there's anything specific here to how npm interacts with ProGet, and because the bugs that you've encountered are being tracked elsewhere, I'm going to close this issue. Let us know if you need further help getting the CLI working with your network + registry setup. Thanks for the followup!