Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] 429 Too Many Requests #836

Closed
raphaelyancey opened this issue Feb 17, 2020 · 266 comments
Closed

[BUG] 429 Too Many Requests #836

raphaelyancey opened this issue Feb 17, 2020 · 266 comments

Comments

@raphaelyancey
Copy link

@raphaelyancey raphaelyancey commented Feb 17, 2020

What / Why

I'm getting E429 error

When

  • Everytime I run npm ci (since today at least)

Where

  • npm public registry

Current Behavior

  • The npm ci command returns E429 error (Too Many Requests) and doesn't complete packages installation

Steps to Reproduce

  • npm ci

Expected Behavior

  • It should install packages
@44kia244
Copy link

@44kia244 44kia244 commented Feb 17, 2020

Same here, but with npm -g install @vue/cli.

npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/lowdb/-/lowdb-1.0.0.tgz

@hubertsmusz
Copy link

@hubertsmusz hubertsmusz commented Feb 17, 2020

#805 similar isseue - random 403 or 429

@MatejBalantic
Copy link

@MatejBalantic MatejBalantic commented Feb 17, 2020

Having same issue on our pipelines. Responses vary between 403 Forbidden and 420 Too Many Requests

@kurrestahlberg
Copy link

@kurrestahlberg kurrestahlberg commented Feb 17, 2020

We see this in any of our CI tasks running in AWS

Step 8/11 : RUN npm ci
 ---> Running in 87051ac87a51
npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/@types/xxhashjs/-/xxhashjs-0.2.1.tgz
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-02-17T11_49_35_151Z-debug.log
The command '/bin/sh -c npm ci' returned a non-zero code: 1
ERROR: Job failed: exit code 1

@m33ch
Copy link

@m33ch m33ch commented Feb 17, 2020

Also for me on bamboo build:

error	17-feb-2020 12:49:46	npm ERR! code E429
error	17-feb-2020 12:49:46	npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/@snyk/configstore/-/configstore-3.2.0-rc1.tgz

@quite
Copy link

@quite quite commented Feb 17, 2020

Centralized infrastructure :~(
(waiting for the post-mortem, but not holding my breath...)

@raphaelyancey
Copy link
Author

@raphaelyancey raphaelyancey commented Feb 17, 2020

It'd be useful to have a list of (verified) public registry mirrors. I found some but I can't trust them.

@pwright08
Copy link

@pwright08 pwright08 commented Feb 17, 2020

Same, both locally and on Circle CI

@Kellym-Kainos
Copy link

@Kellym-Kainos Kellym-Kainos commented Feb 17, 2020

Also seeing the same using Circle CI and locally

npm ERR! code E429 npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/camelcase/-/camelcase-2.1.1.tgz

@jimhooker2002
Copy link

@jimhooker2002 jimhooker2002 commented Feb 17, 2020

I'm seeing errors like..

"The owner of this website (registry.npmjs.org) has banned you temporarily from accessing this website"

and

"You are being rate limited"

I'm guessing this is all related?

@ymynem
Copy link

@ymynem ymynem commented Feb 17, 2020

We are also having this issue when deploying on Heroku.

npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/strip-indent/-/strip-indent-1.0.1.tgz

@bitqueenbr
Copy link

@bitqueenbr bitqueenbr commented Feb 17, 2020

Having the same errors when deploying on heroku.

@codyfy
Copy link

@codyfy codyfy commented Feb 17, 2020

same here with AWS CodeBuild and npm i -g aws-cdk

npm ERR! code E429
--
28 | npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/aws-cdk

general server issue?

@esrgrlk
Copy link

@esrgrlk esrgrlk commented Feb 17, 2020

I also have same problem
npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/silent-error

@vrease
Copy link

@vrease vrease commented Feb 17, 2020

Same here when installing packages locally.
Sweden.

npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/lodash

@mattandrews
Copy link

@mattandrews mattandrews commented Feb 17, 2020

Yep, I'm seeing this on Travis too for npm audit:

npm ERR! audit Your configured registry (https://registry.npmjs.org/) may not support audit requests, or the audit endpoint may be temporarily unavailable.

npm ERR! audit The server said:

Access denied | registry.npmjs.org used Cloudflare to restrict access
You are being rate limited
The owner of this website (registry.npmjs.org) has banned you temporarily from accessing this website.

@Franchi05
Copy link

@Franchi05 Franchi05 commented Feb 17, 2020

Same thing happening over here. Getting the error when doing npm update

@wiseelf
Copy link

@wiseelf wiseelf commented Feb 17, 2020

17-Feb-2020 11:47:48 npm ERR! code E429
17-Feb-2020 11:47:48 npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/@babel/plugin-transform-block-scoping/-/plugin-transform-block-scoping-7.8.3.tgz

Same issue here. We are using bamboo ci. Own installation.
The file is accessible from the server itself:

--2020-02-17 11:59:28--  https://registry.npmjs.org/@babel/plugin-transform-block-scoping/-/plugin-transform-block-scoping-7.8.3.tgz
Resolving registry.npmjs.org (registry.npmjs.org)... 104.16.17.35, 104.16.24.35, 104.16.26.35, ...
Connecting to registry.npmjs.org (registry.npmjs.org)|104.16.17.35|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6735 (6.6K) [application/octet-stream]
Saving to: 'plugin-transform-block-scoping-7.8.3.tgz'

plugin-transform-block-scoping-7.8.3.tgz                 100%[================================================================================================================================>]   6.58K  --.-KB/s    in 0s

2020-02-17 11:59:28 (95.5 MB/s) - 'plugin-transform-block-scoping-7.8.3.tgz' saved [6735/6735]```

@dtmzr
Copy link

@dtmzr dtmzr commented Feb 17, 2020

Facing this issue too, is this a global thing or maybe region related? We just had something similar last year in Germany.

@dorsharon
Copy link

@dorsharon dorsharon commented Feb 17, 2020

Same here running on Gitlab CI

@simonkarman
Copy link

@simonkarman simonkarman commented Feb 17, 2020

Same here in The Netherlands. (AWS Codebuild from Ireland)

@serggend
Copy link

@serggend serggend commented Feb 17, 2020

Russia to

@Volem
Copy link

@Volem Volem commented Feb 17, 2020

Istanbul here

@NickThompson1993
Copy link

@NickThompson1993 NickThompson1993 commented Feb 17, 2020

This appears to be a Cloudflare related issue to the registry.npmjs.org site.

got the following html response on update:

<!DOCTYPE html>
npm ERR! <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
npm ERR! <!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
npm ERR! <!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
npm ERR! <!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
npm ERR! <head>
npm ERR! <title>Access denied | registry.npmjs.org used Cloudflare to restrict access</title>
npm ERR! <meta charset="UTF-8" />
npm ERR! <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
npm ERR! <meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
npm ERR! <meta name="robots" content="noindex, nofollow" />
npm ERR! <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
npm ERR! <link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
npm ERR! <!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
npm ERR! <style type="text/css">body{margin:0;padding:0}</style>
npm ERR!
npm ERR!
npm ERR! <!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/zepto.min.js"></script><!--<![endif]-->
npm ERR! <!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/cf.common.js"></script><!--<![endif]-->
npm ERR!
npm ERR!
npm ERR!
npm ERR! </head>
npm ERR! <body>
npm ERR!   <div id="cf-wrapper">
npm ERR!     <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
npm ERR!     <div id="cf-error-details" class="cf-error-details-wrapper">
npm ERR!       <div class="cf-wrapper cf-header cf-error-overview">
npm ERR!         <h1>
npm ERR!           <span class="cf-error-type" data-translate="error">Error</span>
npm ERR!           <span class="cf-error-code">1015</span>
npm ERR!           <small class="heading-ray-id">Ray ID: REDACTED &bull; 2020-02-17 11:26:27 UTC</small>
npm ERR!         </h1>
npm ERR!         <h2 class="cf-subheadline">You are being rate limited</h2>
npm ERR!       </div><!-- /.header -->
npm ERR!
npm ERR!       <section></section><!-- spacer -->
npm ERR!
npm ERR!       <div class="cf-section cf-wrapper">
npm ERR!         <div class="cf-columns two">
npm ERR!           <div class="cf-column">
npm ERR!             <h2 data-translate="what_happened">What happened?</h2>
npm ERR!             <p>The owner of this website (registry.npmjs.org) has banned you temporarily from accessing this website.</p>
npm ERR!           </div>
npm ERR!
npm ERR!
npm ERR!         </div>
npm ERR!       </div><!-- /.section -->
npm ERR!
npm ERR!       <div class="cf-error-footer cf-wrapper">
npm ERR!   <p>
npm ERR!     <span class="cf-footer-item">Cloudflare Ray ID: <strong>REDACTED</strong></span>
npm ERR!     <span class="cf-footer-separator">&bull;</span>
npm ERR!     <span class="cf-footer-item"><span>Your IP</span>: REDACTED</span>
npm ERR!     <span class="cf-footer-separator">&bull;</span>
npm ERR!     <span class="cf-footer-item"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>
npm ERR!
npm ERR!   </p>
npm ERR! </div><!-- /.error-footer -->
npm ERR!
npm ERR!
npm ERR!     </div><!-- /#cf-error-details -->
npm ERR!   </div><!-- /#cf-wrapper -->
npm ERR!
npm ERR!   <script type="text/javascript">
npm ERR!   window._cf_translation = {};
npm ERR!
npm ERR!
npm ERR! </script>
npm ERR!
npm ERR! </body>
npm ERR! </html>

@CormacWhite-rsc
Copy link

@CormacWhite-rsc CormacWhite-rsc commented Feb 17, 2020

Same issue happening with AWS Codebuild us-east-1. Was broken locally up til about 30 minutes ago but back working now (locally from Ireland)

@merc1er
Copy link

@merc1er merc1er commented Feb 17, 2020

This appears to be a Cloudflare related issue to the registry.npmjs.org site.

Is there any mirror not using cloudflare?

@tschannik
Copy link

@tschannik tschannik commented Feb 17, 2020

Same problem! Build pipelines are failing :(

@jmeade11
Copy link

@jmeade11 jmeade11 commented Feb 17, 2020

Same: npm ERR! code E429

@ghost
Copy link

@ghost ghost commented Feb 17, 2020

That's it. Internet is done. Good bye everyone.

@andreeleuterio
Copy link

@andreeleuterio andreeleuterio commented Feb 17, 2020

Andre from npm security here. Our content delivery partner has informed us they’ve implemented a fix. and we are continuing to monitor the situation. You can find more information in our status page: https://status.npmjs.org/.

Closing the issue but if you have any issue please contact support@npmjs.com

@rahulshinde19
Copy link

@rahulshinde19 rahulshinde19 commented Feb 17, 2020

tenor

@SargoDarya
Copy link

@SargoDarya SargoDarya commented Feb 17, 2020

@aeleuterio Any chance we can get a post-mortem on this?

@cathzetjo
Copy link

@cathzetjo cathzetjo commented Feb 17, 2020

Working now!
OMG. Dont do that again! Ever! )))))

@gokselb
Copy link

@gokselb gokselb commented Feb 17, 2020

It's not working again. I guess, issue has not been fixed by your partners.

@danielytics
Copy link

@danielytics danielytics commented Feb 17, 2020

@cmcarey it is never a bad thing to smile and laugh, it is was never wasting the time thing, it will improve your productivity actually, memes and jokes make us a human instead of just working machines, even Detroit become a human man!

Memes and jokes drown out the conversation that may be vital to fixing whatever the issue actually is, though.

@uncoooloj
Copy link

@uncoooloj uncoooloj commented Feb 17, 2020

Working now 😓

@CraigMason
Copy link

@CraigMason CraigMason commented Feb 17, 2020

Memes and jokes drown out the conversation that may be vital to fixing whatever the issue actually is, though.

Indeed, I had to scroll through about 200 memes just to see the actual status update from NPM.

@PurHur
Copy link

@PurHur PurHur commented Feb 17, 2020

Just for your info: yarn was able to load the packages without triggering the rate limit :1)

yarn is great! (and safe)

@buro9
Copy link

@buro9 buro9 commented Feb 17, 2020

Hello and profuse apologies from Cloudflare, a post-mortem of sorts directly in your issue comments.

I am the engineering manager for the DDoS protection team and this morning at 11:06 UTC we tweaked a rule that affected one of our signals. The signal relates to the HTTP referer header, and we have a piece of code that looks at invalid referer headers. In this case we tweaked it to include not just "obvious garbage" but "anything that does not conform to the HTTP specification"... i.e. is the referer a URI? If not then it contributes to knowledge about bad traffic.

So... why did this impact npmjs.org? It turns out that a lot of NPM traffic sends the referer as "install" which is invalid according to the HTTP specification. As NPM is also a heavily trafficked site this resulted in the DDoS systems picking this up and treating the traffic as a HTTP flood and determining that a rate-limit should be applied.

When we noticed that NPM was seeing an increase in HTTP 429s (as seen on Twitter) we contacted NPM and started an internal investigation. As soon as we identified the root cause we reverted the change, which was at 13:00 UTC.

We'll note that NPM and 1 other site use the referer for purposes outside the HTTP spec and we'll update our systems to ensure that this does not happen again. Additionally we'll improve our monitoring around changes of this nature so that we can discover impact sooner and roll back automatically.

@CraigMason
Copy link

@CraigMason CraigMason commented Feb 17, 2020

Thanks for the explaination @buro9

Hopefully you will have some explicit tests for NPM, given the importance to the developer community.

We (and many others I'm sure) were unable to deploy a number of projects for 2 hours this morning, during EU business hours. This should also serve as a reminder for us all to have better continuity measures in place for when these events happen.

@NicolasGoeddel
Copy link

@NicolasGoeddel NicolasGoeddel commented Feb 17, 2020

In my opinion, it would be best to make sure that the requests from the NPM installer comply with the HTTP specification.

@nc9
Copy link

@nc9 nc9 commented Feb 17, 2020

In my opinion, it would be best to make sure that the requests from the NPM installer comply with the HTTP specification.

Referer should be blank, installer should be user agent

@MLDMoritz
Copy link

@MLDMoritz MLDMoritz commented Feb 17, 2020

Thanks, I am able to download all my 5464950 dependencies every 15min for another build again.

@ahmadnassri
Copy link

@ahmadnassri ahmadnassri commented Feb 17, 2020

@buro9 we would appreciate it if you respond to our tickets and our internal slack comms, before posting to a public issue, we still have not gotten a post-mortem report for the last two outages.

as for pointing at HTTP specifications, considering this behaviour has been in place for years, I would ask to review what change was pushed in CF today that caused this sudden "compliance with HTTP Specification" result?

I'll ask again to please follow up with our open tickets and report back to to us on the post-mortem for the last two outages, we'd rather learn about this from you directly, than seeing it in an issue on github ..

@uap-universe
Copy link

@uap-universe uap-universe commented Feb 17, 2020

Hello and profuse apologies from Cloudflare,

I don't think you have to apologize. npm obviously messed up the referrer field and you did nothing wrong. Just because it accidentally worked out in the past doesn't mean it should stay that way. Who shall guarantee that something like this doesn't happen again in the future, because someone is respecting the spec?

@MLDMoritz
Copy link

@MLDMoritz MLDMoritz commented Feb 17, 2020

npm obviously messed up the referrer field and you did nothing wrong. Just because it accidentally worked out in the past doesn't mean it should stay that way.

That's called BC breaks and should not occur in the same 'version'.

@uap-universe
Copy link

@uap-universe uap-universe commented Feb 17, 2020

npm obviously messed up the referrer field and you did nothing wrong. Just because it accidentally worked out in the past doesn't mean it should stay that way.

That's called BC breaks and should not occur in the same 'version'.

Yeah I give you that point. But hopefully the decision will not be "it stays forever that way and everyone must comply".

@datenwolf
Copy link

@datenwolf datenwolf commented Feb 17, 2020

(…)
we tweaked a rule that affected one of our signals. The signal relates to the HTTP referer header, and we have a piece of code that looks at invalid referer headers.
(…)
So... why did this impact npmjs.org? It turns out that a lot of NPM traffic sends the referer as "install" which is invalid according to the HTTP specification.

Doesn't the Referer header permit the use of relative / partial URIs? https://tools.ietf.org/html/rfc7231#section-5.5.2

5.5.2.  Referer

   The "Referer" [sic] header field allows the user agent to specify a
   URI reference for the resource from which the target URI was obtained
   (i.e., the "referrer", though the field name is misspelled).  A user
   agent MUST NOT include the fragment and userinfo components of the
   URI reference [RFC3986], if any, when generating the Referer field
   value.

     Referer = absolute-URI / partial-URI

Without probing the server for existence of the URI, how to you distinguish arbitrary, urlencoded text from an actual partial-URI?

If I follow down the deductions of the specification then an Referer header install at first glance could be perfectly valid:

https://tools.ietf.org/html/rfc7230#section-2.7

relative-part = <relative-part, see [RFC3986], Section 4.2>
partial-URI   = relative-part [ "?" query ]

https://tools.ietf.org/html/rfc3986#section-4.2

      relative-ref  = relative-part [ "?" query ] [ "#" fragment ]

      relative-part = "//" authority path-abempty
                    / path-absolute
                    / path-noscheme
                    / path-empty

https://tools.ietf.org/html/rfc3986#section-3.3

path-noscheme = segment-nz-nc *( "/" segment )
segment       = *pchar
segment-nz    = 1*pchar
segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims / "@" )
                    ; non-zero-length segment without any colon ":"

pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"

So if I'm not terribly misreading the specification, I have to wonder: Why the heck has that referrer header been treated as invalid? Does Cloudflare check the existence of that URI upon request (and then caches it), or is this yet again a shoot fast-and-loose "whoopsie"?

@uap-universe
Copy link

@uap-universe uap-universe commented Feb 17, 2020

@datenwolf syntactically it should be possible to have a partial URI like install, but how about this, a few lines below your quoted text:

If the target URI was obtained from a source that does not have its
own URI [...], the user agent MUST either exclude the
Referer field or send it with a value of "about:blank".

This imho applies here.

This doesn't, however, answer your last question. But it seems that the topic is a bit more complex and splits into two parts: 1) is npm conforming to the http spec, and 2) does CF use reliable detection rules. And maybe, just maybe, the answer to both questions is: no. But I leave this to others to discuss. I just wanted to point out that prematurely apologizing for "something" may leave potential bugs unfixed, so the wording might have been a bit unfortunate towards a real "solution" to the problem - whatever it is.

@mkkhedawat
Copy link

@mkkhedawat mkkhedawat commented Feb 17, 2020

Instead of implementing block mode the very first day, It should have been in observe mode for some time.

@datenwolf
Copy link

@datenwolf datenwolf commented Feb 17, 2020

@datenwolf syntactically it should be possible to have a partial URI like ìnstall`, but how about this, a few lines below your quoted text:

If the target URI was obtained from a source that does not have its
own URI [...], the user agent MUST either exclude the
Referer field or send it with a value of "about:blank".

This imho applies here.

Yes, it does. Maybe I've not made it clear enough, that I'm fully aware of this. But that's entirely besides the point.

As you've already pointed out

(…) it seems that the topic is a bit more complex and splits into two parts: 1) is npm conforming to the http spec, and 2) does CF use reliable detection rules. (…)

and I already pointed out, that without an explicit check for the existence of the Referer URI, the validity of the header is impossible to decide.

Does npm violate the specification? Sure.

Can CF correctly detect that? Only by performing an explicit check of the URI. Does CF perform that check? I don't know… yet (but I might set up a testing ground, just for that).

However, without further information I just have to assume that CF again did as CF does and for ill-founded reasons broke a part of the Internet… yet again.

@isaacs
Copy link
Contributor

@isaacs isaacs commented Feb 20, 2020

It's worth keeping in mind in all this that there's very little that a CDN can do which will meaningfully improve their service and which does not carry the risk of "breaking a part of the internet". The nature of the service that CF provides, and their well-deserved popularity as one to provide that service, means that they are basically always playing with fire, and likely to upset quite a lot of people when they make otherwise forgivable and well-intended mistakes.

Having done this npm thing for quite some time now, I have no shortage of sympathy for their position, and I believe that it is inappropriate to heap scorn on them for this. They're doing a great job, protecting npm (and thus, the whole JS community) from a lot of bad actors and outages, and making all of our builds much faster and more reliable. We love and appreciate Cloudflare. So let's be kind here.

That being said, I do believe that npm is neither in violation of the letter nor the spirit of the relevant HTTP specifications in our use of the Referer header to indicate the command that caused a given request to be made. I hope that anyone following this discussion find the following flood of HTTP pedantry useful or at least enjoyable. If that's not your kind of thing, please go do something else, you won't have a good time reading this :)

CF deployed a change that treated unusual use of HTTP headers as a heuristic to be marked as a malicious request. When dealing primarily with traffic from web browsers, the Referer header will generally always be either missing, about:blank, or a fully-qualified URL. So a header like Referer: install is weird.

However: The specification says "URI". It does not say "URL". It certainly does not say "Fully qualified URL". Given the typically meticulous usage of "URI" vs "URL" in IETF documentation, the hair-splitting discussions that often ensue over matters like this, and the fact that Referer first appeared in rfc1945 (though it was in use long before then), and that the HTTP specs have kept "URI" through multiple rounds of updates and revisions, I have to conclude that the intent here is for Referer to be a URI rather than required to be a URL per se, as well as the letter of the spec.

A URI and a URL are not the same thing. Both the linked RFCs have been updated and obsoleted (in part) since their inception by subsequent RFCs, and I strongly encourage anyone still reading at this point to follow the links and learn about how Uniform Resource Location and Identification standards have changed and expanded in subtle and interesting ways over the years.

The bottom line is this: the HTTP Referer header does not need to be a Uniform Resource Locator, but rather a Uniform Resource Identifier. There is no requirement that this identifier use a well-known URI scheme, or that it be a full form rather than relative. The only limitation placed on it is that (a) it must be a URI, and (b) if the request is satisfying a direct user request that does not have any kind of identifier, such as typing a URL into an address bar, then it must be omitted.

A URI explicitly does not need to be locatable, fetchable, or resolveable by any given network agent, or over any given protocol.

So, install is an identifier for the thing which the user interacted with that caused the request to be made. They didn't type the url to the packument or tarball into an address bar, they typed npm install, and it in turn fetched the resource via HTTPS. In order to resolve that resource, it had to make some HTTP requests. No scheme is provided, but none is required. "install" is a compact series of characters identifying a resource. It's a scheme-less (ie, not fully-qulified) URI.

As URI semantics and syntax are defined by their scheme, it is impossible, in absence of a scheme, to say that install isn't valid. For example, tel:+12345678901 is a valid URI (and a valid URL), but http:+12345678901 is not. In order to know if +12345678901 is a valid partial URI, you'd have to know the scheme. Its locatability would depend on the particulars of the telephone systems of northeastern Ohio, USA. If a telephone call to an automated system at that number triggered an HTTP request to be made, it would be perfectly appropriate for that HTTP request to include a Referer header of +12345678901. If the server expects to get requests from such a telephone system, it can infer the scheme based on the context.

That is exactly what happens with the npm client and the npm registry. It sends a Referer header containing the command being run. (When the command contains positional arguments, anything containing / or \ is redacted, as this might be a private path, url, or git repo.) This is semantically and syntactically a correct and appropriate use of the HTTP Referer header in such a non-browser context, and it is my sincere belief that, after 30 years of revision, analysis, and review by the IETF, through multiple versions of this specification, if this was not the intent of the Referer specification, it would've said "URL" rather than "URI".

And just to reiterate, I certainly don't think Cloudflare is a bad actor here, and I'm disappointed to see how quick so many people have been to "pick sides" as if it's npm vs Cloudflare in a battle over this. We were affected by a mistake they made, but we're sometimes going to be affected by mistakes that they make, because we're their customer, and of course they're going to make mistakes from time to time, because humans and machines aren't perfect. That's just how the world is. By and large, we're very happy with the response we got, and we've all improved our monitoring and response systems in light of this hiccup.

@reschke
Copy link

@reschke reschke commented Feb 21, 2020

FWIW, the "referer" is not defined as URI. See the spec: https://greenbytes.de/tech/webdav/rfc7231.html#rfc.section.5.5.2. It's a URI reference. "install" would be interpreted as path, relative to the request URI.

@isaacs
Copy link
Contributor

@isaacs isaacs commented Feb 21, 2020

@reschke Even on that reading, it's still perfectly valid.

That is: from https://registry.npmjs.org/foo, with Referer: install, the fully resolved Referer would be https://registry.npmjs.org/install, a valid URI. From https://registry.npmjs.org/foo/-/foo-1.2.3.tgz, it'd be https://registry.npmjs.org/foo/-/install, also a valid URI.

Even if I'm reading the spec more broadly than intended, it's certainly not a "violation" of the spec to use Referer in this way, and it's a (completely forgivable!) mistaken overreach to block or rate-limit requests that include Referer headers that are not fully-qualified URLs.

In light of this, however, it probably would be worthwhile to put a scheme on the Referer headers that the npm cli sends. We'd have to research this to see if that makes it more or less likely that requests will be mangled by proxies. Another option, of course, is to accept that some proxies will just be overeager in filtering resulting in slightly less ideal data, but use a custom header with a more definitive meaning, like npm-command: install instead. We do this for the npm-session header to group requests together, and have found cases where we don't get this custom header, even though the user-agent implies that it is a "real" npm client (or at least, is claiming to be).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet