-
-
Notifications
You must be signed in to change notification settings - Fork 151
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Split the registry into registry, web, and storage services #135
Conversation
package.json
Outdated
@@ -31,8 +31,8 @@ | |||
"lint-cli": "cd cli; npm run lint", | |||
"lint-fix": "prettier --write '**/*.js'", | |||
"lint-registry": "cd services/registry; npm run lint", | |||
"postinstall": "for d in cli services/registry services/workers services/common/boltzmann; do cd $d; npm i; cd -; done", | |||
"postinstall": "for d in cli services/registry services/workers services/web services/common/boltzmann; do cd $d; npm i; cd -; done", | |||
"start": "./misc/start.sh", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly I'm just thinking about how much work is ahead re-implementing run-scripts.
return function mw(next) { | ||
return async function inner(context) { | ||
const request = context.request; | ||
context.id = request.headers[requestIdHeader] || uuid.v1(); | ||
context.id = request.headers[requestIdHeader] || `${host}_${uuid.v1()}`; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aha. Excellent.
services/web/NOTES
Outdated
deleteToken({for: user, valueHashes: [String]}) -> { count } | ||
resolveCLISession({session, value}) | ||
fetchCLISession({session}) -> { description } | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is, I note, the minimal useful web feature set. I think it's smart of you to avoid things like package visualization for now.
fork.get('/signup', redirectAuthenticated(signup)), | ||
fork.post('/signup', redirectAuthenticated(signupAction)), | ||
fork.get('/tokens', seasurf(redirectUnauthenticated(tokens))), | ||
fork.post('/tokens', seasurf(redirectUnauthenticated(handleTokenAction))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, putting the endpoints into registry for now is good. You might consider namespacing them. I think that'll be good even after we pull things into a second service.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ideally this lands at /settings/:username/tokens
, but I'm going to leave that for another PR.
services/web/handlers/auth.js
Outdated
|
||
context.session.set('banner', 'Successfully deleted 1 token.'); | ||
context.session.set('banner', `Successfully deleted ${count} token${count === 1 ? '' : 's'}.`); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the thin edge of the wedge with ternaries
@@ -0,0 +1,74 @@ | |||
{ | |||
"name": "entropic-web", | |||
"description": "community package manager but for spiders", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🕷
} else { | ||
bole.output({ level: 'info', stream: process.stdout }); | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I note that 90% of this is boilerplate. We should make a generic wrapper.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My go at code golfing the wrapper:
Wrapper
function boltzStarter(router, middles, boltzmann) {
return boltzmann.make(router, middles.map(middle => require(middle)));
}
Usage
const myMiddles = [
'boltzmann/middleware/logger',
'boltzmann/middleware/flush-request',
'boltzmann/middleware/requestid',
'boltzmann/middleware/redis',
'boltzmann/middleware/storage-api',
'./middleware/session',
];
boltzStarter(router, myMiddles, boltzmann).listen(
process.env.PORT,
'0.0.0.0'
);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, I commented above but it looks like y'all are already on it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I made a ticket for that here so it doesn't get lost
9da4900
to
f239e24
Compare
a298fe4
to
8a06e47
Compare
@@ -0,0 +1,100 @@ | |||
storageApi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey look, it's almost a complete list of everything an entropic can do!
@@ -0,0 +1,207 @@ | |||
'use strict' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both the web gateway and registry gateway can use this client to talk to storage.
}) { | ||
this.url = url | ||
this.requestId = requestId | ||
this.userAgent = `${hostname}(${userAgent})` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wrap up the incoming requestId and userAgent (so we know which host is sending which user-agent where, in logs.)
Should I add x-forwarded-for
here as well, so remote address logging works correctly in storage-api?
body: rawBody ? rawBody : body ? JSON.stringify(body) : null | ||
}) | ||
|
||
if (!response.ok) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The fetch api is so nice.
return this._parsedUrl | ||
} | ||
this._parsedUrl = parse(this.request.url, true) | ||
return this._parsedUrl |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Be lazy about URL parsing.
'maintainer.invite.package_dne': `Unknown package: "${invitee}".`, | ||
'maintainer.invite.already_accepted': `Namespace "${invitee}" is already a member.`, | ||
'maintainer.invite.already_declined': `Namespace "${invitee}" has declined this invite.` | ||
}[err.code] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The error mapping pattern emerges. It's almost worth making a decorator for this and throwing the error, I think. This is where we should hang i18n mapping as well, potentially.
const hash = crypto | ||
.createHash('sha256') | ||
.update(bearer + process.env.SESSION_SECRET) | ||
.digest('base64') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The registry now caches auth in redis for a span of time, so we hash it.
"culture-ships": "~1.0.0", | ||
"dotenv": "~8.0.0", | ||
"escape-html": "^1.0.3", | ||
"find-my-way": "~2.0.1", | ||
"graceful-fs": "^4.1.15", | ||
"is-email-maybe": "^1.0.1", | ||
"markdown": "^0.5.0", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO: review all service deps and shrink to the minimum set for each service.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO another day, perhaps
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
writing up issue
This turned out to be much bigger than expected! (It's good to get this out of the way now, though, because it's just going to move more cheese as time goes on.) Things left to do:
|
|
||
const hostname = os.hostname() | ||
|
||
const e = encodeURIComponent |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think of wrapping this in a function? If any future refactors need to happen it could make it easier.
const encode = (str) => encodeURIComponent(str)
|
||
module.exports = class Client { | ||
constructor ({ | ||
url = process.env.STORAGE_API_URL, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we assert
, throw
, or warn here if that env var isn't defined?
@@ -185,9 +300,10 @@ module.exports = class Client { | |||
message = parsed.message || body | |||
code = parsed.code || code | |||
} catch { | |||
; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm sure you're probably working on this now, but commenting just in case. Empty line / no caught error
'namespace_members.active': true, | ||
'namespace_members.user_id': context.user.id | ||
}) | ||
.catch(Namespace.objects.NotFound, () => null); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we add a log here? #202 took me a bit to track down 😅
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should appear a lot noisier in the logs now after adding dev-mode traces to error responses.
Awesome work! I thought I would try to help PR review and leave some comments. I'll try to test this branch out at some point to see if I can turn up any 🪲 |
@@ -1,5 +1,7 @@ | |||
'use strict'; | |||
|
|||
const { parse } = require('url'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
url.parse
is deprecated, should this use new URL
instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah yeah. new URL
has the (uncomfortable in this case) prereq that you have to give it two args to prevent it from throwing if the first arg is an "incomplete" url – we probably should move over to it, but as a follow up PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Noting to self that I should write up that task as an issue and tag it as a good starter issue.
page: Number(context.url.query.page) || 0, | ||
status: context.url.query.status | ||
}).then( | ||
xs => [null, xs], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this. Haskell influence?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Haha, thanks! I got it from looking at Golang a while back, tbh.
services/registry/handlers/users.js
Outdated
namespace, | ||
host | ||
}).then( | ||
xs => [null, xs], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be worth pulling this into an either
util?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would be great once this lands!
'use strict'; | ||
|
||
const { Response } = require('node-fetch'); | ||
const { markdown } = require('markdown'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This particular markdown lib is very abandoned. It might make sense to use something like markdown-it, which is both maintained and super pluggable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was waiting to use marky-markdown
, which is a Known Good package for rendering markdown; however it brings along oniguruma so I wanted to wait until we had separated package publishes out into a worker process.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a ticket so this doesn't get lost.
OK, I'm calling it – this PR is ready to merge, I think. Brace yourself: I anticipate that it's going to break stuff and things will be a bit bumpy for a little while. Here's what I've done:
I've tested:
I have not:
URL CHANGES
This eliminates all urls with the word why is it important to move all this cheese nowMoving this cheese:
|
"start": "docker-compose up", | ||
"test": "for d in cli services/registry; do cd $d; npm t; cd -; done" | ||
"test": "for d in cli services/{registry,workers,web,common/boltzmann}; do cd $d; npm t; cd -; done" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use bash in anger.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This package.json is a makefile, to be quite honest.
getObject({algo, digest}) // promise for a stream | ||
|
||
|
||
response.rawjson(str) // for enemies |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Leaving this for now, will clean up afterward. Sectumsempra!
|
||
if (isDev()) { | ||
err.trace = new Error().stack; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we're in dev mode, add a trace
property to all outgoing errors. This makes it easier to track down errors when they happen: the registry client will log traces coming from storage
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thoughtful!
const r = new Response(JSON.stringify(err), { status, headers }); | ||
return r; | ||
} | ||
|
||
error.coded = coded; | ||
function coded(code, ...args) { | ||
return error(Object.assign(new Error(code), { code }), ...args); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For storage-layer use. Takes computer-friendly dotted, lowercase error codes.
function authneeded(message, status = 401, extraHeaders = {}) { | ||
const headers = new Headers({ | ||
'www-authenticate': 'bearer', | ||
'content-type': 'application/json', | ||
...extraHeaders | ||
}); | ||
if (typeof message === 'string') { | ||
message = { error: message }; | ||
message = { message, code: 'authneeded' }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Standardize a bit on message
(which is used in response.error
), while we're at it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am good on standardizing on this. Fallback English message + code as a localization switch point. I note that at some point we'll want to pull error codes into some kind of known constants list, for re-use.
); | ||
} | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Provider data lives in web
now.
@@ -96,6 +59,8 @@ module.exports = class Authentication { | |||
} | |||
}; | |||
|
|||
const User = require('./user'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Had to shuffle requires because of 🎶 circular imports 🎶
@@ -34,6 +34,11 @@ module.exports = class Token { | |||
this.active = active; | |||
} | |||
|
|||
toJSON() { | |||
const { user: _0, id: _1, user_id: _2, ...meta } = this; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Omitting properties by destructure+splat.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fiendish!
fork.get('/signup', redirectAuthenticated(signup)), | ||
fork.post('/signup', redirectAuthenticated(signupAction)), | ||
fork.get('/tokens', seasurf(redirectUnauthenticated(tokens))), | ||
fork.post('/tokens', seasurf(redirectUnauthenticated(handleTokenAction))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ideally this lands at /settings/:username/tokens
, but I'm going to leave that for another PR.
<!doctype html> | ||
<html> | ||
<body> | ||
<h1><marquee>WELCOME TO ENTROPIC</marquee></h1> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a package registry on the move!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had some comments but nothing that needs action in this PR.
docker: | ||
- image: circleci/node:latest | ||
- image: circleci/postgres:latest | ||
|
||
working_directory: ~/repo/services/registry | ||
working_directory: ~/repo/services/storage |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ayup.
command: npm start | ||
networks: | ||
- entropic | ||
# ports are explicitly hidden, this is an internal service only. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good note.
"start": "docker-compose up", | ||
"test": "for d in cli services/registry; do cd $d; npm t; cd -; done" | ||
"test": "for d in cli services/{registry,workers,web,common/boltzmann}; do cd $d; npm t; cd -; done" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This package.json is a makefile, to be quite honest.
this.logger = logger; | ||
} | ||
|
||
async getProviders() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Chris. have you... have you slept in the last week? I feel super-lazy now.
This is mostly mechanical work, I realize. It makes me understand why people want things like Swagger, because code generation is so tempting. (Resist the urge!)
` | ||
.trim() | ||
.split(/\s+/) | ||
.join(''), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The lengths you're going to to avoid long lines!
Something in me always twitches when I see this pattern in your code, because my own urges toward tidiness do not see the long lines, but they do see the work that gets repeated every time the function is called.
return response.json({ | ||
objects, | ||
next: hasNext, | ||
prev: hasPrev |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
which we'll eventually need to punch through to the gateway. unless it is already and I failed to notice
// `context.user` return a promise for the currently | ||
// authenticated user (if any.) | ||
// | ||
// TODO: Someday use JWTs, maybe. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Grumpy sound.
@@ -30,6 +30,11 @@ module.exports = class User { | |||
this.active = active; | |||
} | |||
|
|||
toJSON() { | |||
const { tfa_secret: _0, backup_codes: _1, id: _2, ...meta } = this; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good choices to omit.
@@ -34,6 +34,11 @@ module.exports = class Token { | |||
this.active = active; | |||
} | |||
|
|||
toJSON() { | |||
const { user: _0, id: _1, user_id: _2, ...meta } = this; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fiendish!
context.session.set('banner', 'Successfully deleted 1 token.'); | ||
context.session.set( | ||
'banner', | ||
`Successfully deleted ${count} token${count === 1 ? '' : 's'}.` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is what an acceptable ternary looks like
memorize this chris
don't nest them
no stop don't
* fix(cli): allow command name to be placed after flags * chore(cli): refactor whoami command to shared utils * feat(cli): store username per registry when logging in * feat(cli): split `invitations` cmd into packages/namespaces * feat(cli): use --namespace/--package in invite cmd * feat(cli): add --namespace/--package flags to decline cmd * feat(cli): add `ds accept` * fix(cli): update routing to match new spec from #135
Status: it's ready!