Add authentication for Libraries.io-based badges, run [Libraries Bower] #7080
Conversation
calebcartwright
added
service-badge
| get decrementedUsesRemaining() { | ||
| return this._usesRemaining - 1 | ||
| } | ||
|
|
There was a problem hiding this comment.
The rationale/necessity for this is explained in the inline comments in the getRateLimitFromHeaders function in the new librariesio-api-provider class
| const withPooling = tokens.length > 1 | ||
| Object.assign(this, { | ||
| baseUrl, | ||
| withPooling, | ||
| globalToken: tokens[0], | ||
| defaultRateLimit, | ||
| }) |
There was a problem hiding this comment.
Opted to stick with simplicity in the event of a single token, to avoid dealing with all the pooling when there isn't really a pool
| if (this.withPooling) { | ||
| this.standardTokens = new TokenPool({ batchSize: 45 }) | ||
| tokens.forEach(t => this.standardTokens.add(t, {}, defaultRateLimit)) | ||
| } | ||
| } |
There was a problem hiding this comment.
Arbitrary number selected for the batch size. I don't think it matters all that much what the actual value is, so don't feel too strongly about it.
| getRateLimitFromHeaders({ headers, token }) { | ||
| // The Libraries.io API does not consistently provide the rate limiting headers. | ||
| // In some cases (e.g. package/version not founds) it won't include any of these headers, | ||
| // and the `retry-after` header is only provided _after_ the rate limit has been exceeded | ||
| // and requests are throttled. | ||
| // | ||
| // https://github.com/librariesio/libraries.io/issues/2860 | ||
|
|
||
| // The standard rate limit is 60/requests/minute, so fallback to that default | ||
| // if the header isn't present. | ||
| // https://libraries.io/api#rate-limit | ||
| const rateLimit = headers['x-ratelimit-limit'] || this.defaultRateLimit | ||
|
|
||
| // If the remaining header is missing, then we're in the 404 response phase, and simply | ||
| // subtract one from the `usesRemaining` count on the token, since the 404 responses do count | ||
| // against the rate limits. | ||
| const totalUsesRemaining = | ||
| headers['x-ratelimit-remaining'] || token.decrementedUsesRemaining | ||
|
|
||
| // The `retry-after` header is only present post-rate limit excess, and contains the value in | ||
| // seconds the client needs to wait before the limits are reset. | ||
| // Our token pools internally use UTC-based milliseconds, so we perform the conversion | ||
| // if the header is present to ensure the token pool has the correct value. | ||
| // If the header is absent, we just use the current timestamp to | ||
| // advance the value to _something_ | ||
| const retryAfter = headers['retry-after'] | ||
| const nextReset = Date.now() + (retryAfter ? retryAfter * 1000 : 0) | ||
|
|
||
| return { | ||
| rateLimit, | ||
| totalUsesRemaining, | ||
| nextReset, | ||
| } | ||
| } |
There was a problem hiding this comment.
I hope the inline comments provide sufficient explanation for what's happening here. I have opened an upstream issue in the Libraries.io repo (link included in the comments) to see if we can get some consistent response header behavior, but otherwise felt like we had to deal with the reality of the present.
There was a problem hiding this comment.
I think it is unlikely we'll see much movement on that so agree we just have to deal with the messyness 🙁
| token.update(totalUsesRemaining, nextReset) | ||
| } | ||
|
|
||
| async requestAsPromise(request, url, options = {}) { |
There was a problem hiding this comment.
I kept the same name as the corresponding function in the GitHub API Provider, but this is using the same fetcher (got based) as the standard request routes, and is async/await based. Open to bikeshedding on the name, I left it as-is for lazy copy/paste reasons, don't have any real opinions on the name
There was a problem hiding this comment.
Naming things is hard, but I think we can do better here.
The github requestAsPromise function is taking an instance of request (as in literally the library request)
shields/core/base-service/base.js
Line 448 in 14f64d1
which is callback-based and wraps it as a promise
shields/services/github/github-api-provider.js
Lines 202 to 213 in 23c0406
That is a name and concept we need to remove in an upcoming PR to advance #4655 but given this function is not taking an instance of request and not wrapping it as a promise, lets give it a more descriptive name. Now that we've made up our minds on which HTTP client we're going to use, maybe that makes things easier.
There was a problem hiding this comment.
I don't think there's any need for function names to reflect the names of libs consumed (directly or indirectly), so thoughts on any of these and/or any suggestions?
sendRequestmakeRequestrequestsendcallgetgetLibrariesIODatafetch
There was a problem hiding this comment.
I think fetch(requestFetcher, url, options) probably makes the most sense to me if we can use fetch without clashing with base service.
There was a problem hiding this comment.
renamed in e331c2e
|
| const response = await request(`${baseUrl}${url}`, mergedOptions) | ||
| if (this.withPooling) { | ||
| if (response.res.statusCode === 401) { | ||
| this.invalidateToken(token) | ||
| } else if (response.res.statusCode < 500) { | ||
| this.updateToken({ token, url, res: response.res }) | ||
| } | ||
| } | ||
| return response |
There was a problem hiding this comment.
My reading of how we've set up the actual http request issuing code with got is that this will only reject/throw in the event of major issue (no connectivity) and not on standard http response errors. In such a major issue type of scenario, I don't think it would be safe/valid to attempt to update the token pooling info, so I've not wrapped this in any kind of try catch. I think the call making infrastructure already handles anything that would be caught here, but let me know if a catch is needed with something like an explicit error conversion
There was a problem hiding this comment.
Yeah the fetch function we're calling here (unhelpfully the param is called request 🙃 - see other comment about naming) is fundamentally calling this internally:
shields/core/base-service/got.js
Lines 48 to 65 in f62dbed
throwHttpErrors = false (so non-2XX status codes can be handled at the service layer) and wraps any error got throws as a InvalidResponse or Inaccessible (which can then be caught cleanly) so I think we can just call it here - same as in any other service class really.
| @@ -86,6 +86,7 @@ private: | |||
| jenkins_pass: 'JENKINS_PASS' | |||
| jira_user: 'JIRA_USER' | |||
| jira_pass: 'JIRA_PASS' | |||
| librariesio_tokens: 'LIBRARIESIO_TOKENS' | |||
There was a problem hiding this comment.
opted to just stick an s on the end to, along with the docs, convey this supports accepting multiple tokens. I'm not sure we have any prior art with this type of value beyond the various *origins vars though, so let me know if folks have any preferred alternatives
core/server/server.js
Outdated
| this.librariesioConstellation = new LibrariesIoConstellation({ | ||
| service: publicConfig.services.librariesio, | ||
| private: privateConfig, | ||
| }) |
There was a problem hiding this comment.
This constellation class is a borderline no-op. It originally looked a bit more like the corresponding GitHub class, but after the decision earlier today to skip persistence it was dramatically simplified. I've opted to keep the class as somewhat of an abstraction/keeping the libraries-specific instantiations inside there, but let me know if folks think it'd be better to just deal with the api provider instantiation inline here in the server.
| import Joi from 'joi' | ||
| import { anyInteger, nonNegativeInteger } from '../validators.js' | ||
| import { BaseJsonService } from '../index.js' | ||
|
|
||
| // API doc: https://libraries.io/api#project | ||
| const projectSchema = Joi.object({ | ||
| platform: Joi.string().required(), | ||
| dependents_count: nonNegativeInteger, | ||
| dependent_repos_count: nonNegativeInteger, | ||
| rank: anyInteger, | ||
| }).required() | ||
|
|
||
| function createRequestFetcher(context, config) { | ||
| const { sendAndCacheRequest, librariesIoApiProvider } = context | ||
|
|
||
| return async (url, options) => | ||
| await librariesIoApiProvider.requestAsPromise( | ||
| sendAndCacheRequest, | ||
| url, | ||
| options | ||
| ) | ||
| } | ||
|
|
||
| export default class LibrariesIoBase extends BaseJsonService { | ||
| constructor(context, config) { | ||
| super(context, config) | ||
| this._requestFetcher = createRequestFetcher(context, config) | ||
| } | ||
|
|
||
| async fetchProject({ platform, scope, packageName }) { | ||
| return this._requestJson({ | ||
| schema: projectSchema, | ||
| url: `/${encodeURIComponent(platform)}/${ | ||
| scope ? encodeURIComponent(`${scope}/`) : '' | ||
| }${encodeURIComponent(packageName)}`, | ||
| errorMessages: { 404: 'package not found' }, | ||
| }) | ||
| } | ||
| } |
There was a problem hiding this comment.
I personally tend to prefer the reusable helper/fetcher type functions over extending the inheritance hierarchy, but it felt cleaner/more straightforward to me to use a base class for setting up the request fetcher on construction in a base class.
There was a problem hiding this comment.
There are 2 main entities in libraries.io - projects and repos:
- https://libraries.io/api#project
- https://libraries.io/api#repository
which have different schemas so - https://libraries.io/api/npm/babel (project) and
- https://libraries.io/api/github/babel/babel (repo)
return quite differently structured responses (although that isn't totally obvious from the routes - its a wierd API).
Most of our badges query projects but we do have one or two query the repo endpoints.
This class is called LibrariesIoBase but it is really best thought of as LibrariesIoProjectBase. If we made LibrariesIoBase just
class LibrariesIoBase extends BaseJsonService {
constructor(context, config) {
super(context, config)
this._requestFetcher = createRequestFetcher(context, config)
}
}and kept projectSchema/fetchProject seperate would you see any downside to that? It seems more conceptually right (although at this point the chance of adding new libraries.io badges seems somewhat remote).
There was a problem hiding this comment.
seperate would you see any downside to that?
My thinking was more about encapsulation and usage patterns. For service classes that need to make requests to the Libraries.io API, they must use the custom request fetcher, and therefore must extend this base class. If we have the common, borderline ubiquitous, function(s) and arguments (schema(s)) separated in a separate module, then that requires each individual service class to be sure to import and use from both modules, but it's still possible for a developer/class to speciously just import the function/schema without extending the base class.
I feel like the standalone modules with importable functions/objects/etc. are particularly useful/preferable when we want them to be usable regardless of the inheritance hierarchy, which isn't applicable in the Librarioes.io based cases.
Yes, it's true that adding the schema and function to the base class means the base class will contain things that aren't strictly used by every extending child class. However, that's pretty common throughout the rest of our own service class hierarchy and in my experience OOP in general.
although at this point the chance of adding new libraries.io badges seems somewhat remote
I actually think this PR increases the odds of us at least having the ability to use the service for more, now that we have a more well defined and expansive request footprint with authenticated requests using a pool. I don't know whether that hypothetical would necessarily manifest as new badges with net-new data points, or as an alternative source for existing types of badges
| t.create('license not declared') | ||
| .get('/bootstrap.json') | ||
| .intercept(nock => | ||
| nock('https://libraries.io') | ||
| .get('/api/bower/bootstrap') | ||
| .reply(200, { normalized_licenses: [] }) | ||
| ) | ||
| .expectBadge({ label: 'license', message: 'missing' }) | ||
|
|
There was a problem hiding this comment.
Our Bower badges had a few such mocked requests. With the hard requirement that we have the API Keys enabled in CI (we always exceed the rate limit with our ~20+ tests without auth), trying to mock these becomes increasingly difficult.
We've used some nock tricks elsewhere (e.g. skipWhen) that I considered, but I actually found these mocked tests were either duplicative/unnecessary tests of a render helper (as was the case here) and/or the test scenario would be better covered with unit tests.
| const data = await this.fetch({ packageName }) | ||
| const includePrereleases = queryParams.include_prereleases !== undefined | ||
| const version = this.constructor.transform(data, includePrereleases) | ||
|
|
||
| return renderVersionBadge({ version }) |
There was a problem hiding this comment.
transformation logic separated out here to make it easier to use standard unit tests, because we were using mocked service tests to validate this transformation logic before.
| t.create('Version label should be `no releases` if no stable version') | ||
| .get('/v/bootstrap.json') | ||
| .intercept(nock => | ||
| nock('https://libraries.io') | ||
| .get('/api/bower/bootstrap') | ||
| .reply(200, { normalized_licenses: [], latest_stable_release: null }) | ||
| ) | ||
| .expectBadge({ label: 'bower', message: 'no releases' }) | ||
|
|
||
| t.create('Version label should be `no releases` if no pre-release') | ||
| .get('/v/bootstrap.json?include_prereleases') | ||
| .intercept(nock => | ||
| nock('https://libraries.io') | ||
| .get('/api/bower/bootstrap') | ||
| .reply(200, { normalized_licenses: [], latest_release_number: null }) | ||
| ) | ||
| .expectBadge({ label: 'bower', message: 'no releases' }) | ||
|
|
There was a problem hiding this comment.
Covered with new unit tests, mocked tests removed for the aforementioned reasons
| const scope = nock('https://libraries.io/api') | ||
| // This ensures that the expected credentials are actually being sent with the HTTP request. | ||
| // Without this the request wouldn't match and the test would fail. | ||
| .get(`/bower/bootstrap?api_key=${fakeApiKey}`) |
There was a problem hiding this comment.
For whatever reason, I was unable to get nock to intercept this request when I using the options arg (e.g. get('....', { qs: ... }). As is often the case for me with nock it was probably a pebkac issue, but I didn't feel like trying to battle it
calebcartwright
force-pushed
the
librariesio-auth
branch
from
baab63c
to
4790cef
Compare
| token.update(totalUsesRemaining, nextReset) | ||
| } | ||
|
|
||
| async requestAsPromise(request, url, options = {}) { |
There was a problem hiding this comment.
Naming things is hard, but I think we can do better here.
The github requestAsPromise function is taking an instance of request (as in literally the library request)
shields/core/base-service/base.js
Line 448 in 14f64d1
which is callback-based and wraps it as a promise
shields/services/github/github-api-provider.js
Lines 202 to 213 in 23c0406
That is a name and concept we need to remove in an upcoming PR to advance #4655 but given this function is not taking an instance of request and not wrapping it as a promise, lets give it a more descriptive name. Now that we've made up our minds on which HTTP client we're going to use, maybe that makes things easier.
core/server/server.js
Outdated
| @@ -242,6 +244,11 @@ class Server { | |||
| private: privateConfig, | |||
| }) | |||
|
|
|||
| this.librariesioConstellation = new LibrariesIoConstellation({ | |||
| service: publicConfig.services.librariesio, | |||
There was a problem hiding this comment.
As far as I can see, there is no services.librariesio key in publicConfigSchema
There was a problem hiding this comment.
good catch. this is a leftover from the original route i'd taken of using a backing persistence for the tokens, with some of the associated (non-secret) config items going under this key
There was a problem hiding this comment.
fixed in e42648f
| const scope = nock('https://libraries.io/api') | ||
| // This ensures that the expected credentials are actually being sent with the HTTP request. | ||
| // Without this the request wouldn't match and the test would fail. | ||
| .get(`/bower/bootstrap?api_key=${fakeApiKey}`) |
| } catch (e) { | ||
| console.error('Unable to select next Libraries.io token from pool') | ||
| log.error(e) | ||
| return |
There was a problem hiding this comment.
If we get here, this function is going to return undefined instead of a response object and throw a UnhandledPromiseRejectionWarning. I think we need to wrap this in an exception we can catch. log.error() should handle logging the error to sentry for us.
There was a problem hiding this comment.
Are you thinking we need to define a new error type of ShieldsRuntimeError and throw it here?
There was a problem hiding this comment.
It doesn't necessarily have to be a brand new exception, but ultimately we should throw it as a ShieldsRuntimeError of some description. ImproperlyConfigured might be a good fit?
There was a problem hiding this comment.
addressed in 8c912ba
| import Joi from 'joi' | ||
| import { anyInteger, nonNegativeInteger } from '../validators.js' | ||
| import { BaseJsonService } from '../index.js' | ||
|
|
||
| // API doc: https://libraries.io/api#project | ||
| const projectSchema = Joi.object({ | ||
| platform: Joi.string().required(), | ||
| dependents_count: nonNegativeInteger, | ||
| dependent_repos_count: nonNegativeInteger, | ||
| rank: anyInteger, | ||
| }).required() | ||
|
|
||
| function createRequestFetcher(context, config) { | ||
| const { sendAndCacheRequest, librariesIoApiProvider } = context | ||
|
|
||
| return async (url, options) => | ||
| await librariesIoApiProvider.requestAsPromise( | ||
| sendAndCacheRequest, | ||
| url, | ||
| options | ||
| ) | ||
| } | ||
|
|
||
| export default class LibrariesIoBase extends BaseJsonService { | ||
| constructor(context, config) { | ||
| super(context, config) | ||
| this._requestFetcher = createRequestFetcher(context, config) | ||
| } | ||
|
|
||
| async fetchProject({ platform, scope, packageName }) { | ||
| return this._requestJson({ | ||
| schema: projectSchema, | ||
| url: `/${encodeURIComponent(platform)}/${ | ||
| scope ? encodeURIComponent(`${scope}/`) : '' | ||
| }${encodeURIComponent(packageName)}`, | ||
| errorMessages: { 404: 'package not found' }, | ||
| }) | ||
| } | ||
| } |
There was a problem hiding this comment.
There are 2 main entities in libraries.io - projects and repos:
- https://libraries.io/api#project
- https://libraries.io/api#repository
which have different schemas so - https://libraries.io/api/npm/babel (project) and
- https://libraries.io/api/github/babel/babel (repo)
return quite differently structured responses (although that isn't totally obvious from the routes - its a wierd API).
Most of our badges query projects but we do have one or two query the repo endpoints.
This class is called LibrariesIoBase but it is really best thought of as LibrariesIoProjectBase. If we made LibrariesIoBase just
class LibrariesIoBase extends BaseJsonService {
constructor(context, config) {
super(context, config)
this._requestFetcher = createRequestFetcher(context, config)
}
}and kept projectSchema/fetchProject seperate would you see any downside to that? It seems more conceptually right (although at this point the chance of adding new libraries.io badges seems somewhat remote).
| const response = await request(`${baseUrl}${url}`, mergedOptions) | ||
| if (this.withPooling) { | ||
| if (response.res.statusCode === 401) { | ||
| this.invalidateToken(token) | ||
| } else if (response.res.statusCode < 500) { | ||
| this.updateToken({ token, url, res: response.res }) | ||
| } | ||
| } | ||
| return response |
There was a problem hiding this comment.
Yeah the fetch function we're calling here (unhelpfully the param is called request 🙃 - see other comment about naming) is fundamentally calling this internally:
shields/core/base-service/got.js
Lines 48 to 65 in f62dbed
throwHttpErrors = false (so non-2XX status codes can be handled at the service layer) and wraps any error got throws as a InvalidResponse or Inaccessible (which can then be caught cleanly) so I think we can just call it here - same as in any other service class really.
Resolves #6179 - would advise reviewing by commit
Our Lirbaries.io-based badges have been problematic for a long time. In the past it was advantageous for us to not authenticate requests as we were somehow actually getting a higher rate limit with the anonymous requests. However, that changed some time back which is why the service tests have been failing for months and the badges have been flaky to say the least.
According to the response headers on anonymous requests, the rate limit is 30/minute (compared to 60/minute with authentication). However, each anonymous request seems to use at least 5 points, so in practice it ends up being more like only a handful of requests per minute.
This PR adds an API provider type pattern similar to, but much simpler than, what we do with our GitHub requests, wherein a single token can be specified or multiple tokens can be used to fill a rotating pool. We have had past discussions about just doing the simple/standard auth route, however, we did have concerns that a single token would be insufficient during heavier traffic periods with our existing load patterns for these badges. Additionally, I could envision some growth/traffic increase for these badges once we get them stabilized (e.g. some folks have left Shields over the issue with these badges, and could return/future users could stick once we're able to offer these consistently).
As discussed on today's ops meeting, this implementation allows multiple tokens to be provided via the standard config instead of having them persisted in/pulled from Redis.
I've already wired up the CI environment with a token (using a bot account I own), and myself and a couple other maintainers will donate our Libraries.io tokens in our prod environment.