Skip to content

Commit

Permalink
persona arch: wordsmith
Browse files Browse the repository at this point in the history
  • Loading branch information
lloyd committed Jun 15, 2012
1 parent 64517f6 commit ac58c72
Showing 1 changed file with 39 additions and 34 deletions.
73 changes: 39 additions & 34 deletions posts/2012-06-15-persona-architectural-changes.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
title: Persona Architectural Changes

<abstract>
The [Persona Sign-In service][], by Mozilla, is a service that lets
The [Persona Login service][] lets
web developers implement seamless login via email address with a
trivial amount of code. It is written in NodeJS and runs on hardware
paid for by mozilla. The service is deployed as several distinct
trivial amount of code. It is written in NodeJS and is supported
by Mozilla. The service is deployed as several distinct
NodeJS processes. Recently we've added a new process to the service,
and this short post will describe what's changed and why.

Expand All @@ -19,28 +19,28 @@ who build, test, deploy, and maintain Persona.
The Persona deployment previously consisted of the following processes
with said responsibilities:

* **browserid** - The main entry point for all web requests, this process
was responsible for handling all *read only* api requests, handling
static file serving, and forwarding api requests that need either access
to our private "domain" key (certificate signing requests), or api
requests which require write access to the database.
* **browserid** - The main entry point for all web requests, handles
all *read only* API requests, static file serving, and
forwards API requests that need either access
to our private "domain" key (certificate signing requests), or those
which require write access to the database.
* **keysigner** - The only process in the system that requires read access
to the domain key, and handles key signing requests.
to the domain key to handle key signing requests.
* **verifier** - A process that performs verification of assertions
* **dbwriter** - A process that handles all api requests which require
* **dbwriter** - A process that handles all API requests which require
write access to the database.

In addition to these node.js processes, we have quite a bit of infrastructure
In addition to these NodeJS processes, we have quite a bit of infrastructure
for high availability in our layout as well. Roughly, here's what
BrowserID looked like:
Persona looked like:

<picture>

## Adding `router`
## Adding **router**

As you can see in the description above, the **browserid** process had
a large set of diverse responsibilities. Repeatedly in load testing
we have seen that process cpu bound and is the current bottlneck of
we have seen that process CPU bound and is the current bottleneck of
the system. Further, we have begun work implementing countermeasures
to keep the service available during malicious network attacks. Both
for performance purposes and to create a place to run countermeasures,
Expand All @@ -49,20 +49,20 @@ we've introduced a new process that subsumes some responsibilities of

The new process and responsibility breakdown is as follows:

* **router** - The main entry point for all web requests, this process
is responsible for forwarding api requests to the remaining processes.
* **browserid** - This process is responsible for handling all
*read only* api requests, handling static file serving, and forwards
* **router** - The main entry point for all web requests, responsible
for forwarding API requests.
* **browserid** - Responsible for handling all
*read only* API requests, static file serving, and forwarding
to the keysigner when certification is required.
* **keysigner** - The only process in the system that requires read access
to the domain key, and handles key signing requests.
to the domain key to execute key-signing requests.
* **verifier** - A process that performs verification of assertions
* **dbwriter** - A process that handles all api requests which require
* **dbwriter** - A process that handles all API requests which require
write access to the database.

As you can see, the responsibility shift is extremely minor in this first
push to introduce the process. We have not yet introduced the
countermeasures mentioned above, nor have we removed all of orthogonal
push to introduce **router**. We have not yet introduced the
countermeasures mentioned above, nor have we moved all of orthogonal
responsibilities of the **browserid** process. These changes are planned
for future trains (with 2 week development cycles, the only way to
roll is incrementally).
Expand All @@ -74,7 +74,7 @@ The new software deployment looks roughly like this:
As you might expect, a very small part of the Persona deployment is
actually about NodeJS: of equal importance are the mechanisms that start
the programs, display statistics and health, and monitor them to allow us
to proactively discover problems before they surface. The next several
to pro-actively discover problems before they impact users. The next several
sections run through the new deployment architecture from these different
perspectives.

Expand All @@ -86,40 +86,45 @@ Monitoring should initially target non-200 HTTP responses. While 4xx responses
are part of normal operation, 5xx should be monitored and alerted.

In addition to monitoring HTTP responses, the router emits statsd events over
UDP. These are all [namespaced under `browserid.router.`](https://github.com/mozilla/browserid/blob/dev/bin/router#L74), and the most interesting stats emitted are API invocation counts, under `browserid.router.wsapi.*`.
UDP. These are all [name-spaced under `browserid.router.`](https://github.com/mozilla/browserid/blob/dev/bin/router#L74), and the most interesting stats emitted are API invocation counts, under `browserid.router.wsapi.*`.

### Logging

What you'll see in the router's logfile is mostly reports of forwarded
requests. That's all it does.
What you'll see in the router's log-file is mostly reports of forwarded
requests. That's all it does. Any `error` level messages are of great
interest.

### Expected Performance Impact

The the router removes HTTP forwarding from the most heavily loaded
The router removes HTTP forwarding from the most heavily loaded
process in the system. Now that **browserid** is no longer responsible
request forwarding, it's expected that we'll see higher possible ADU
request forwarding, it's expected that we'll see slightly higher possible ADU
in load testing, with greater total processor usage on our machines.

## Upcoming Work

### The `static` process
### The **static** process

An immediate desire is to introduce another service for static file
serving. This service would be responsible for serving all of our
static content. This includes both:

* **views**: html pages with no dynamic content that do not vary between
deployments, but change every 2 weeks and hence have a maxAge of 0 to force re-validation
* **static resources**: non-html resources that are served from perma-urls with a hash embedded in the url, having far-future expiration dates.
* **views**: HTML pages with no dynamic content that do not vary between
deployments, but change every 2 weeks and hence have a max-Age of 0 to force cache re-validation for instant cache update after deployment.
* **static resources**: non-HTML resources that are served from perma-urls with a hash embedded in the URL, having far-future expiration dates.

### A CDN

All static content should be moved off to a mozilla controlled HTTPS
All static content should be moved off to a Mozilla controlled HTTPS
CDN for lower latency and faster load times. the **static** process
discussed above would then be nothing more than a CDN origin server.
We will soon deploy a change that will move all static content off to
a different hostname to prepare for the introduction of a CDN.

### Anti abuse heuristics

Now that **router** is in the works, we can move forward and begin to
implement anti-abuse heuristics for different types of live network
attacks (initial focus will be on (D)Dos.
attacks (initial focus will be on [(D)DoS][].

[(D)DoS]: http://en.wikipedia.org/wiki/DoS

0 comments on commit ac58c72

Please sign in to comment.