Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split the registry into registry, web, and storage services #135

Merged
merged 14 commits into from
Jun 9, 2019

Conversation

chrisdickinson
Copy link
Collaborator

@chrisdickinson chrisdickinson commented Jun 4, 2019

Status: it's ready!

package.json Outdated
@@ -31,8 +31,8 @@
"lint-cli": "cd cli; npm run lint",
"lint-fix": "prettier --write '**/*.js'",
"lint-registry": "cd services/registry; npm run lint",
"postinstall": "for d in cli services/registry services/workers services/common/boltzmann; do cd $d; npm i; cd -; done",
"postinstall": "for d in cli services/registry services/workers services/web services/common/boltzmann; do cd $d; npm i; cd -; done",
"start": "./misc/start.sh",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly I'm just thinking about how much work is ahead re-implementing run-scripts.

return function mw(next) {
return async function inner(context) {
const request = context.request;
context.id = request.headers[requestIdHeader] || uuid.v1();
context.id = request.headers[requestIdHeader] || `${host}_${uuid.v1()}`;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha. Excellent.

deleteToken({for: user, valueHashes: [String]}) -> { count }
resolveCLISession({session, value})
fetchCLISession({session}) -> { description }

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is, I note, the minimal useful web feature set. I think it's smart of you to avoid things like package visualization for now.

fork.get('/signup', redirectAuthenticated(signup)),
fork.post('/signup', redirectAuthenticated(signupAction)),
fork.get('/tokens', seasurf(redirectUnauthenticated(tokens))),
fork.post('/tokens', seasurf(redirectUnauthenticated(handleTokenAction)))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, putting the endpoints into registry for now is good. You might consider namespacing them. I think that'll be good even after we pull things into a second service.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this lands at /settings/:username/tokens, but I'm going to leave that for another PR.


context.session.set('banner', 'Successfully deleted 1 token.');
context.session.set('banner', `Successfully deleted ${count} token${count === 1 ? '' : 's'}.`);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the thin edge of the wedge with ternaries

@@ -0,0 +1,74 @@
{
"name": "entropic-web",
"description": "community package manager but for spiders",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🕷

} else {
bole.output({ level: 'info', stream: process.stdout });
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I note that 90% of this is boilerplate. We should make a generic wrapper.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My go at code golfing the wrapper:

Wrapper

function boltzStarter(router, middles, boltzmann) {
  return boltzmann.make(router, middles.map(middle => require(middle)));
}

Usage

const myMiddles = [
  'boltzmann/middleware/logger',
  'boltzmann/middleware/flush-request',
  'boltzmann/middleware/requestid',
  'boltzmann/middleware/redis',
  'boltzmann/middleware/storage-api',
  './middleware/session',
];

boltzStarter(router, myMiddles, boltzmann).listen(
  process.env.PORT,
  '0.0.0.0'
);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I commented above but it looks like y'all are already on it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a ticket for that here so it doesn't get lost

@@ -0,0 +1,100 @@
storageApi
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey look, it's almost a complete list of everything an entropic can do!

@@ -0,0 +1,207 @@
'use strict'
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both the web gateway and registry gateway can use this client to talk to storage.

}) {
this.url = url
this.requestId = requestId
this.userAgent = `${hostname}(${userAgent})`
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrap up the incoming requestId and userAgent (so we know which host is sending which user-agent where, in logs.)

Should I add x-forwarded-for here as well, so remote address logging works correctly in storage-api?

body: rawBody ? rawBody : body ? JSON.stringify(body) : null
})

if (!response.ok) {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fetch api is so nice.

return this._parsedUrl
}
this._parsedUrl = parse(this.request.url, true)
return this._parsedUrl
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Be lazy about URL parsing.

'maintainer.invite.package_dne': `Unknown package: "${invitee}".`,
'maintainer.invite.already_accepted': `Namespace "${invitee}" is already a member.`,
'maintainer.invite.already_declined': `Namespace "${invitee}" has declined this invite.`
}[err.code]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error mapping pattern emerges. It's almost worth making a decorator for this and throwing the error, I think. This is where we should hang i18n mapping as well, potentially.

const hash = crypto
.createHash('sha256')
.update(bearer + process.env.SESSION_SECRET)
.digest('base64')
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The registry now caches auth in redis for a span of time, so we hash it.

"culture-ships": "~1.0.0",
"dotenv": "~8.0.0",
"escape-html": "^1.0.3",
"find-my-way": "~2.0.1",
"graceful-fs": "^4.1.15",
"is-email-maybe": "^1.0.1",
"markdown": "^0.5.0",
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: review all service deps and shrink to the minimum set for each service.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO another day, perhaps

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

writing up issue

@chrisdickinson
Copy link
Collaborator Author

chrisdickinson commented Jun 7, 2019

This turned out to be much bigger than expected! (It's good to get this out of the way now, though, because it's just going to move more cheese as time goes on.)

Things left to do:

  • Finish wiring up the client
  • Fill out the remaining "TODO" api endpoints from the client (things we were previously glomming together with the registry/web endpoints before, like token creation)
  • Do a error enumeration pass for now, pass errors through and leave TODO statements in an effort to not bite off more than I can chew with this PR.
  • Do a logging pass through:
    • the registry
    • the website
    • the storage layer


const hostname = os.hostname()

const e = encodeURIComponent
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think of wrapping this in a function? If any future refactors need to happen it could make it easier.

const encode = (str) => encodeURIComponent(str)


module.exports = class Client {
constructor ({
url = process.env.STORAGE_API_URL,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we assert, throw, or warn here if that env var isn't defined?

@@ -185,9 +300,10 @@ module.exports = class Client {
message = parsed.message || body
code = parsed.code || code
} catch {
;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sure you're probably working on this now, but commenting just in case. Empty line / no caught error

'namespace_members.active': true,
'namespace_members.user_id': context.user.id
})
.catch(Namespace.objects.NotFound, () => null);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a log here? #202 took me a bit to track down 😅

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should appear a lot noisier in the logs now after adding dev-mode traces to error responses.

@olingern
Copy link
Contributor

olingern commented Jun 8, 2019

Awesome work! I thought I would try to help PR review and leave some comments. I'll try to test this branch out at some point to see if I can turn up any 🪲

@@ -1,5 +1,7 @@
'use strict';

const { parse } = require('url');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

url.parse is deprecated, should this use new URL instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah. new URL has the (uncomfortable in this case) prereq that you have to give it two args to prevent it from throwing if the first arg is an "incomplete" url – we probably should move over to it, but as a follow up PR.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noting to self that I should write up that task as an issue and tag it as a good starter issue.

page: Number(context.url.query.page) || 0,
status: context.url.query.status
}).then(
xs => [null, xs],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this. Haskell influence?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haha, thanks! I got it from looking at Golang a while back, tbh.

namespace,
host
}).then(
xs => [null, xs],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be worth pulling this into an either util?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be great once this lands!

'use strict';

const { Response } = require('node-fetch');
const { markdown } = require('markdown');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This particular markdown lib is very abandoned. It might make sense to use something like markdown-it, which is both maintained and super pluggable.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was waiting to use marky-markdown, which is a Known Good package for rendering markdown; however it brings along oniguruma so I wanted to wait until we had separated package publishes out into a worker process.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a ticket so this doesn't get lost.

@chrisdickinson chrisdickinson marked this pull request as ready for review June 8, 2019 06:48
@chrisdickinson
Copy link
Collaborator Author

OK, I'm calling it – this PR is ready to merge, I think. Brace yourself: I anticipate that it's going to break stuff and things will be a bit bumpy for a little while.

Here's what I've done:

  • www.js has been split out of registry into services/web.
  • Most of the registry code has moved back up into a new (private) service called storage.
    • The intent is that this will not be visible to the outside world.
  • web and registry are responsible for trading external authentication (session cookies and authorization headers, respectively) for internal authentication (the surprisingly versatile bearer: <username> header.)
  • registry caches authentication results in redis.
  • membership (of users to namespaces) and maintainership (of packages by namespaces) URLs have been restructed but the CLI has not caught up. See URL CHANGES below.
  • docker-compose.yml contains all the new services.
  • Added response.error.coded which accepts a string (a la namespace.members.bearer_unauthed), designed to create a disjoint between storage layer error cases and registry layer error message display. (This means we can start pushing i18n concerns into the registry layer!)

I've tested:

  • logging in both via services running locally and through docker
  • publishing a new package
  • testing whoami
  • manually curling a few list endpoints – packages, maintainerships, memberships.

I have not:

  • updated the documentation (for another PR!)
  • added registry tests (but wow we need them! all of the existing tests – save invitations, which were deleted due to massive upheaval – moved from registry to storage)
  • audited the dependencies of each service. It's possible (likely!) that services are depending on things they aren't using.

URL CHANGES

GET     /v1/users/user/:user/memberships[?pending=true]                list membership
POST    /v1/users/user/:user/memberships/invitations/:namespace@:host  accept invitation
DELETE  /v1/users/user/:user/memberships/invitations/:namespace@:host  decline invitatation

GET     /v1/namespaces/namespace/:namespace@:host/maintainerships[?pending=true]            list maintainerships
POST    /v1/namespaces/namespace/:namespace@:host/maintainerships/:namespace@:host/:package accept invite
DELETE  /v1/namespaces/namespace/:namespace@:host/maintainerships/:namespace@:host/:package decline invite

GET     /v1/namespaces/namespace/:namespace@:host/members[?pending=true] list members
POST    /v1/namespaces/namespace/:namespace@:host/members/:user          invite user
DELETE  /v1/namespaces/namespace/:namespace@:host/members/:user          remove member / rescind invite

GET     /v1/packages/package/:namespace@:host/maintainers[?pending=true] list maintainers
POST    /v1/packages/package/:namespace@:host/maintainers/:name@:host    invite maintainer
DELETE  /v1/packages/package/:namespace@:host/maintainers/:name@:host    remove maintainer / rescind invite

This eliminates all urls with the word invite in them, because we don't so much have invites as "pending" maintainerships and memberships; and the user/namespace is free to leave if they choose to do so. The CLI should be updated to reflect these changes! This can be done in a subsequent PR.

why is it important to move all this cheese now

Moving this cheese:

  • Gives people a service in which to build all the website functionality in a connected way. (If we end up going with Next.js, we may need yet another service that lives outside of the private network and uses web as an API gateway/backend-for-frontend. That's okay, though – we've already gone from 1 service to 3. 3 to 4 will be much easier.)
  • Lets us separate expensive processes (like publishing and syncing!) into a worker process.

"start": "docker-compose up",
"test": "for d in cli services/registry; do cd $d; npm t; cd -; done"
"test": "for d in cli services/{registry,workers,web,common/boltzmann}; do cd $d; npm t; cd -; done"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use bash in anger.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This package.json is a makefile, to be quite honest.

getObject({algo, digest}) // promise for a stream


response.rawjson(str) // for enemies
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this for now, will clean up afterward. Sectumsempra!


if (isDev()) {
err.trace = new Error().stack;
}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're in dev mode, add a trace property to all outgoing errors. This makes it easier to track down errors when they happen: the registry client will log traces coming from storage.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughtful!

const r = new Response(JSON.stringify(err), { status, headers });
return r;
}

error.coded = coded;
function coded(code, ...args) {
return error(Object.assign(new Error(code), { code }), ...args);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For storage-layer use. Takes computer-friendly dotted, lowercase error codes.

function authneeded(message, status = 401, extraHeaders = {}) {
const headers = new Headers({
'www-authenticate': 'bearer',
'content-type': 'application/json',
...extraHeaders
});
if (typeof message === 'string') {
message = { error: message };
message = { message, code: 'authneeded' };
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Standardize a bit on message (which is used in response.error), while we're at it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am good on standardizing on this. Fallback English message + code as a localization switch point. I note that at some point we'll want to pull error codes into some kind of known constants list, for re-use.

);
}
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Provider data lives in web now.

@@ -96,6 +59,8 @@ module.exports = class Authentication {
}
};

const User = require('./user');
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had to shuffle requires because of 🎶 circular imports 🎶

@@ -34,6 +34,11 @@ module.exports = class Token {
this.active = active;
}

toJSON() {
const { user: _0, id: _1, user_id: _2, ...meta } = this;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Omitting properties by destructure+splat.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fiendish!

fork.get('/signup', redirectAuthenticated(signup)),
fork.post('/signup', redirectAuthenticated(signupAction)),
fork.get('/tokens', seasurf(redirectUnauthenticated(tokens))),
fork.post('/tokens', seasurf(redirectUnauthenticated(handleTokenAction)))
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this lands at /settings/:username/tokens, but I'm going to leave that for another PR.

<!doctype html>
<html>
<body>
<h1><marquee>WELCOME TO ENTROPIC</marquee></h1>
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a package registry on the move!

@chrisdickinson chrisdickinson changed the title feat: WIP slicing www out of registry Split the registry into registry, web, and storage services Jun 8, 2019
Copy link
Collaborator

@ceejbot ceejbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had some comments but nothing that needs action in this PR.

docker:
- image: circleci/node:latest
- image: circleci/postgres:latest

working_directory: ~/repo/services/registry
working_directory: ~/repo/services/storage
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ayup.

command: npm start
networks:
- entropic
# ports are explicitly hidden, this is an internal service only.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good note.

"start": "docker-compose up",
"test": "for d in cli services/registry; do cd $d; npm t; cd -; done"
"test": "for d in cli services/{registry,workers,web,common/boltzmann}; do cd $d; npm t; cd -; done"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This package.json is a makefile, to be quite honest.

this.logger = logger;
}

async getProviders() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chris. have you... have you slept in the last week? I feel super-lazy now.

This is mostly mechanical work, I realize. It makes me understand why people want things like Swagger, because code generation is so tempting. (Resist the urge!)

`
.trim()
.split(/\s+/)
.join(''),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lengths you're going to to avoid long lines!

Something in me always twitches when I see this pattern in your code, because my own urges toward tidiness do not see the long lines, but they do see the work that gets repeated every time the function is called.

return response.json({
objects,
next: hasNext,
prev: hasPrev
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which we'll eventually need to punch through to the gateway. unless it is already and I failed to notice

// `context.user` return a promise for the currently
// authenticated user (if any.)
//
// TODO: Someday use JWTs, maybe.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grumpy sound.

@@ -30,6 +30,11 @@ module.exports = class User {
this.active = active;
}

toJSON() {
const { tfa_secret: _0, backup_codes: _1, id: _2, ...meta } = this;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good choices to omit.

@@ -34,6 +34,11 @@ module.exports = class Token {
this.active = active;
}

toJSON() {
const { user: _0, id: _1, user_id: _2, ...meta } = this;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fiendish!

context.session.set('banner', 'Successfully deleted 1 token.');
context.session.set(
'banner',
`Successfully deleted ${count} token${count === 1 ? '' : 's'}.`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is what an acceptable ternary looks like
memorize this chris
don't nest them
no stop don't

@ceejbot ceejbot merged commit 0be1a86 into master Jun 9, 2019
@ceejbot ceejbot deleted the chris/spin-a-web branch June 9, 2019 14:51
This was referenced Jun 9, 2019
@olingern olingern mentioned this pull request Jun 9, 2019
ceejbot pushed a commit that referenced this pull request Jun 10, 2019
* fix(cli): allow command name to be placed after flags

* chore(cli): refactor whoami command to shared utils

* feat(cli): store username per registry when logging in

* feat(cli): split `invitations` cmd into packages/namespaces

* feat(cli): use --namespace/--package in invite cmd

* feat(cli): add --namespace/--package flags to decline cmd

* feat(cli): add `ds accept`

* fix(cli): update routing to match new spec from #135
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants