Enable personal token usage for CubeStore download script #4264
Description
Is your feature request related to a problem? Please describe.
We have an issue on our CI pipeline when executing yarn install and randomly the installation failed due to Github API rate limit being exceeded. The octokit lib suggests using a personal token but we can't set it into download script.
Describe the solution you'd like
It would be good, if download.ts checks an (e.g.) CUBEJS_GH_API_TOKEN and uses it for GH API requests.
Do you accept PR solving this issue with something similar solution?:
async function fetchRelease(version: string) {
const client = new Octokit({
auth: process.env.CUBEJS_GH_API_TOKEN,
request: {
agent: await getHttpAgentForProxySettings(),
}
});
const { data } = await client.request('GET /repos/{owner}/{repo}/releases/tags/{tag}', {
owner: 'cube-js',
repo: 'cube.js',
tag: `v${version}`
});
return data;
}Describe alternatives you've considered
I don't know any alternatives, because the scripts running as a post-install script, so only environment variables could work.
Additional context
Original error message:
+ yarn install
yarn install v1.22.18
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
[4/5] Linking dependencies...
warning "@cubejs-backend/cubestore-driver > @cubejs-backend/query-orchestrator > moment-range@4.0.2" has unmet peer dependency "moment@>= 2".
warning "moleculer-apollo-server > graphql-tools > @graphql-tools/relay-operation-optimizer > relay-compiler@12.0.0" has incorrect peer dependency "graphql@^15.0.0".
warning "@cubejs-backend/native@0.29.22" is missing a bundled dependency "@mapbox/node-pre-gyp". This should be reported to the package maintainer.
[5/5] Building fresh packages...
error /opt/atlassian/pipelines/agent/build/node_modules/@cubejs-backend/cubestore: Command failed.
Exit code: 1
Command: node bin/post-install
Arguments:
Directory: /opt/atlassian/pipelines/agent/build/node_modules/@cubejs-backend/cubestore
Output:
Cube.js Cube Store Installer ---------------------------------------
RequestError [HttpError]: API rate limit exceeded for 52.54.90.98. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)
at /opt/atlassian/pipelines/agent/build/node_modules/@octokit/request/dist-src/fetch-wrapper.js:68:27
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at fetchRelease (/opt/atlassian/pipelines/agent/build/node_modules/@cubejs-backend/cubestore/js-wrapper/src/download.ts:26:20)
at Object.downloadBinaryFromRelease (/opt/atlassian/pipelines/agent/build/node_modules/@cubejs-backend/cubestore/js-wrapper/src/download.ts:63:19)
at /opt/atlassian/pipelines/agent/build/node_modules/@cubejs-backend/cubestore/js-wrapper/src/post-install.ts:12:7 {
status: 403,
response: {
url: 'https://api.github.com/repos/cube-js/cube.js/releases/tags/v0.29.21',
status: 403,
headers: {
'access-control-allow-origin': '*',
'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-RateLimit-Used, X-RateLimit-Resource, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset',
connection: 'close',
'content-length': '277',
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
'content-security-policy': "default-src 'none'; style-src 'unsafe-inline'",
'content-type': 'application/json; charset=utf-8',
date: 'Wed, 23 Mar 2022 21:25:16 GMT',
'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
server: 'Varnish',
'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
'x-content-type-options': 'nosniff',
'x-frame-options': 'deny',
'x-github-media-type': 'github.v3; format=json',
'x-github-request-id': '1FBE:4350:114B0A:2D53E9:623B903C',
'x-ratelimit-limit': '60',
'x-ratelimit-remaining': '0',
'x-ratelimit-reset': '1648071552',
'x-ratelimit-resource': 'core',
'x-ratelimit-used': '60',
'x-xss-protection': '1; mode=block'
},
data: {
message: "API rate limit exceeded for 52.54.90.98. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
documentation_url: 'https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting'
}
},
request: {
method: 'GET',
url: 'https://api.github.com/repos/cube-js/cube.js/releases/tags/v0.29.21',
headers: {
accept: 'application/vnd.github.v3+json',
'user-agent': 'octokit-core.js/3.5.1 Node.js/16.14.2 (linux; x64)'
},
request: { agent: undefined, hook: [Function: bound bound register] }
}
}
Need some help? -------------------------------------
Ask this question in Cube.js Slack: https://slack.cube.dev/
Post an issue: https://github.com/cube-js/cube.js/issues