Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

Merge branch 'master' of github.com:mozilla/anodejsholidayseason

  • Loading branch information...
commit ca589a77b28f4ca766950321c03384b4a67391b0 2 parents 9f4ccbe + 8650be9
@ozten ozten authored
View
103 fantastic-front-end-performance-2.md
@@ -0,0 +1,103 @@
+# Fantastic front-end performance, part 2: caching dynamic content with etagify
+
+You might know that [Connect](https://github.com/senchalabs/connect) puts [ETags](http://en.wikipedia.org/wiki/HTTP_ETag) on static content, but not dynamic content. Unfortunately, if you dynamically generate i18n versions of static pages, those pages don't get caching headers at all--unless you add a build step to pregenerate all pages in all languages. What a lame chore.
+
+This article introduces [etagify](https://github.com/lloyd/connect-etagify), a Connect middleware that generates ETags on the fly by md5-ing outgoing response bodies, storing the hashes in memory. Etagify lets you skip the build step, improves performance more than you might think (we measured a 9% load time improvement in our tests), and it's super easy to use:
+
+### 1. register etagify at startup
+ myapp = require('express').createServer();
+ myapp.use(require('etagify')()); // <--- like this.
+
+### 2. call etagify on routes you want to cache
+ app.get('/about', function(req, res) {
+ res.etagify(); // <--- like that.
+ var body = ejs.render(template, options);
+ res.send(body);
+ });
+
+Read on to learn more about etagify: how it works, when to use it, when not to use it, and how to measure your results.
+
+(Need a refresher on ETags and HTTP caching? We've put together a [cheat sheet](https://gist.github.com/6a68/4971859) to get you back up to speed.)
+
+## How etagify works
+
+By focusing on a single, concrete use case, etagify gets the job done in just a hundred lines of code (including documentation). Let's take a look at the fifteen lines that cover the basics, leaving out Vary header handling edge cases.
+
+There are two parts to consider: hashing outgoing responses & caching the hashes; checking the cache against incoming conditional GETs.
+
+First, here's where we add to the cache. Comments inline.
+
+ // simplified etagify.js internals
+
+ // start with an empty cache
+ // example entry:
+ // '/about': { md5: 'fa88257b77...' }
+ var etags = {};
+
+ var _end = res.end;
+ res.end = function(body) {
+ var hash = crypto.createHash('md5');
+
+ // if the response has a body, hash it
+ if (body) { hash.update(body); }
+
+ // then add the item to the cache
+ etags[req.path] = { md5: hash.digest('hex') };
+
+ // back to our regularly-scheduled programming
+ _end.apply(res, arguments);
+ }
+
+Next, here's how we check against the cache. Again, comments inline.
+
+ // the etagify middleware
+ return function(req, res, next) {
+ var cached = etags[req.path]['md5'];
+
+ // always add the ETag if we have it
+ if (cached) { res.setHeader('ETag', '"' + cached + '"' }
+
+ // if the browser sent a conditional GET,
+ if (connect.utils.conditionalGET(req)) {
+
+ // check if the If-None-Match and ETags are equal
+ if (!connect.utils.modified(req, res)) {
+
+ // cache hit! browser's version matches cached version.
+ // strip out that ETag & bail with a 304 Not Modified.
+ res.removeHeader('ETag');
+ return connect.utils.notModified(res);
+ }
+ }
+ }
+
+### When (and when not) to use etagify
+
+Etagify's approach is super simple, and it's a great solution for dynamically-generated pages that don't change while the server is running, like i18n static pages. However, etagify has some gotchas when dealing with other common use cases:
+
+* if pages change after being first cached, users will always see the stale, cached version
+* if pages are personalized for each user, two things could happen:
+ * if a Vary:cookie header is used to cache users' individual pages separately, then etagify's cache will grow without bound
+ * if no Vary:cookie header is present, then the first version to enter the cache will be shown to all users
+
+## Measuring performance improvements
+
+We didn't foresee huge performance wins with etagify, because conditional GETs still require an HTTP roundtrip, and avoiding page redownloading only saves the user a few KB (see screenshot). However, etagify is a really simple optimization, so even a small gain would justify including it in our stack.
+
+![firebug screen cap showing 2kb savings](http://i.imgur.com/MVSQYKo.jpg)
+
+We tested etagify's effects on performance by spinning up a dev instance of [Persona](https://github.com/mozilla/browserid) on an [awsbox](https://github.com/mozilla/awsbox), opening up firebug, and taking 50 load time measurements of our 'about' page--with and without etagify enabled. (Page load times are a good-enough metric for our use case; you might care more about time till above-the-fold content renders, or the first content hits the page, or the first ad is displayed.)
+
+After gathering raw data, we did some quick statistics to see how much etagify improved performance. We found the mean and standard deviation for both data sets, assuming the measured values were spread out like a [bell curve](http://en.wikipedia.org/wiki/Normal_distribution) around the averages.
+
+Surprisingly, we found that **etagify reduced load time by 9%**, from 1.65 (SD = 0.19) to 1.50 (SD = 0.13) seconds. That's a serious gain for almost no work.
+
+Next, we used the [t-test](http://en.wikipedia.org/wiki/Student%27s_t-test) to check the odds that the improvement could be observed without adding etagify at all. Our [p-value](http://en.wikipedia.org/wiki/P-value) was less than 0.01, meaning less than 1% chance that randomness could have caused the apparent improvement. We can conclude that the measured improvement is statistically significant.
+
+Here's a chart of the averaged before and after data:
+
+![normal distributions with and without etagify](http://i.imgur.com/Tc45vHg.png)
+
+## Bringing it all back home
+
+We think [etagify](https://github.com/lloyd/connect-etagify) is a tiny bundle of win. Even if it's not the right tool for your current project, hopefully our approach of (1) writing focused tools to solve just the problem at hand, and (2) measuring just rigorously enough to be sure you're getting somewhere, gives you inspiration or food for thought.
View
187 fantastic-front-end-performance-3--connect-fonts.md
@@ -0,0 +1,187 @@
+# Fantastic front end performance, part 3 - Big performance wins by serving fonts smarter - A Node.js holiday season, part 8
+
+We reduced Persona's font footprint 85%, from 300 KB to 45 KB, using font subsetting. This post outlines exactly how we implemented these performance improvements, and gives you tools to do the same.
+
+## Introducing connect-fonts
+
+```connect-fonts``` is a Connect font-management middleware that improves ```@font-face``` performance by serving locale-specific subsetted font files, significantly reducing downloaded font size. It also generates locale/browser-specific ```@font-face``` CSS, and manages the CORS header required by Firefox and IE 9+. Subsetted fonts are served from a *font pack*--a directory tree of font subsets, plus a simple JSON config file. Some common open-source fonts are available in pregenerated font packs [on npm](https://npmjs.org/browse/keyword/connect-fonts), and creating your own font packs is straightforward.
+
+(Feeling lost? We've [put together](https://gist.github.com/6a68/5187976) a few references to good ```@font-face``` resources on the web.)
+
+### Static vs dynamic font loading
+
+When you are just serving one big font to all your users, there's not much involved in getting web fonts set up:
+ * generate ```@font-face``` CSS and insert into your existing CSS
+ * generate the full family of web fonts from your TTF or OTF file, then put them someplace the web server can access
+ * add CORS headers to your web server if fonts are served from a separate domain, as Firefox and IE9+ enforce the same origin policy with fonts
+
+These steps are pretty easy; the awesome [FontSquirrel generator](http://www.fontsquirrel.com/tools/webfont-generator) can generate all the missing font files and the ```@font-face``` CSS declaration for you. You've still got to sit down with Nginx or Apache docs to figure out how to add the CORS header, but that's not too tough.
+
+If you want to take advantage of font subsetting to hugely improve performance, things become more complex. You'll have font files for each supported locale, and will need to dynamically modify the ```@font-face``` CSS declaration to point at the right URL. CORS management is still needed. This is the problem ```connect-fonts``` solves.
+
+### Font subsetting: overview
+
+By default, font files contain lots of characters: the Latin character set familiar to English speakers; accents and accented characters added to the Latin charset for languages like French and German; additional alphabets like Cyrillic or Greek. Some fonts also contain lots of funny symbols, particularly if they support Unicode ([☃](http://en.wikipedia.org/wiki/Snowman#Unicode) anyone?). Some fonts additionally support East Asian languages. Font files contain all of this so that they can ably serve as many audiences as possible. All this flexibility leads to large file sizes; Microsoft Arial Unicode, which has characters for every language and symbol in Unicode 2.1, weighs in at an unbelievable 22 megabytes.
+
+In contrast, a typical web page only needs a font to do one specific job: display the page's content, usually in just one language, and usually without exotic symbols. By reducing the served font file to just the subset we need, we can shave off a ton of page weight.
+
+### Performance gains from font subsetting
+
+Let's compare the size of the localized font files vs the full file size for some common fonts and a few locales. Even if you just serve an English-language site, you can shave off a ton of bytes by serving an English subset.
+
+Smaller fonts mean faster load time and a shorter wait for styled text to appear on screen. This is particularly important if you want to use ```@font-face``` on mobile; if your users happen to be on a 2G network, saving 50KB can speed up load time by 2-3 seconds. Another consideration: mobile caches are small, and subsetted fonts have a far better chance of staying in cache.
+
+#### Open Sans regular, size of full font (default) and several subsets (KB):
+
+![Chart comparing file sizes of Open Sans subsets. Full font, 104 KB. Cyrillic, 59 KB. Latin, 29 KB. German, 22 KB. English, 20 KB. French, 24 KB.](https://gist.github.com/6a68/5122048/raw/23d41ed71f079adb21656518f85e262a0fccaade/unzipped.png)
+
+#### Same fonts, gzipped (KB):
+
+![Chart comparing file sizes of Open Sans subsets when gzipped. Full font, 63 KB. Cyrillic, 36 KB. Latin, 19 KB. German, 14 KB. English, 13 KB. French, 15 KB.](https://gist.github.com/6a68/5122048/raw/3f5d6cce8fe1db2721583350dbd01224dd1feb30/gzipped.png)
+
+Even after gzipping, you can reduce font size 80% by using the English subset of Open Sans (13 KB), instead of the full font (63 KB). Consider that this is the reduction for just one font file--most sites use several. The potential is huge!
+
+**Using ```connect-fonts```, Mozilla Persona's font footprint went from 300 KB to 45 KB, an 85% reduction.** This equates to several seconds of download time on a typical 3G connection, and up to 10 seconds on a typical 2G connection.
+
+### Going further with optimizations
+
+If you're looking to tweak every last byte and HTTP request, ```connect-fonts``` can be configured to return generated CSS as a string instead of a separate file. Going even further, ```connect-fonts```, by default, serves up the smallest possible @font-face declaration, omitting declarations for filetypes not accepted by a given browser.
+
+## Example: adding connect-fonts to an app
+
+Suppose you've got a super simple express app that serves up the current time:
+
+ // app.js
+ const
+ ejs = require('ejs'),
+ express = require('express'),
+ fs = require('fs');
+
+ var app = express.createServer(),
+ tpl = fs.readFileSync(__dirname, '/tpl.ejs', 'utf8');
+
+ app.get('/time', function(req, res) {
+ var output = ejs.render(tpl, {
+ currentTime: new Date()
+ });
+ res.send(output);
+ });
+
+ app.listen(8765, '127.0.0.1');
+
+with a super simple template:
+
+ // tpl.ejs
+ <!doctype html>
+ <p>the time is <%= currentTime %>.
+
+Let's walk through the process of adding ```connect-fonts``` to serve the Open Sans font, one of [several](https://npmjs.org/browse/keyword/connect-fonts) ready-made font packs.
+
+### App changes
+
+1. Install via npm:
+
+ $ npm install connect-fonts
+ $ npm install connect-fonts-opensans
+
+2. Require the middleware:
+
+ // app.js - updated to use connect-fonts
+ const
+ ejs = require('ejs'),
+ express = require('express'),
+ fs = require('fs'),
+ // add requires:
+ connect_fonts = require('connect-fonts'),
+ opensans = require('connect-fonts-opensans');
+
+ var app = express.createServer(),
+ tpl = fs.readFileSync(__dirname, '/tpl.ejs', 'utf8');
+
+3. Initialize the middleware:
+
+ // app.js continued
+ // add this app.use call:
+ app.use(connect_fonts.setup({
+ fonts: [opensans],
+ allow_origin: 'http://localhost:8765'
+ })
+The arguments to ```connect_fonts.setup()``` include:
+ * ```fonts```: an array of fonts to enable,
+ * ```allow_origin```: the origin for which we serve fonts; ```connect-fonts``` uses this info to set the Access-Control-Allow-Origin header for browsers that need it (Firefox 3.5+, IE 9+)
+ * ```ua``` (optional): a parameter listing the user-agents to which we'll serve fonts. By default, ```connect-fonts``` uses UA sniffing to only serve browsers font formats they can parse, reducing CSS size. ```ua: 'all'``` overrides this to serve all fonts to all browsers.
+
+4. Inside your route, pass the user's locale to the template:
+
+ // app.js continued
+ app.get('/time', function(req, res) {
+ var output = ejs.render(tpl, {
+ // pass the user's locale to the template
+ userLocale: detectLocale(req),
+ currentTime: new Date()
+ });
+ res.send(output);
+ });
+5. Detect the user's preferred language. Mozilla Persona uses [i18n-abide](https://github.com/mozilla/i18n-abide), and [locale](https://github.com/jed/locale) is another swell option; both are available via npm. For the sake of keeping this example short, we'll just grab the first two chars from the [Accept-Language header](https://developer.mozilla.org/en-US/docs/HTTP/Content_negotiation#The_Accept-Language.3A_header):
+
+ // oversimplified locale detection
+ function detectLocale(req) {
+ return req.headers['accept-language'].slice(0,2);
+ }
+
+ app.listen(8765, '127.0.0.1');
+ // end of app.js
+
+### Template changes
+
+Now we need to update the template. ```connect-fonts``` assumes routes are of the form
+
+ /:locale/:font-list/fonts.css
+for example,
+
+ /fr/opensans-regular,opensans-italics/fonts.css
+In our case, we'll need to:
+
+5. add a stylesheet ```<link>``` to the template matching the route expected by ```connect-fonts```:
+
+ // tpl.ejs - updated to use connect-fonts
+ <!doctype html>
+ <link href="/<%= userLocale %>/opensans-regular/fonts.css" rel="stylesheet">
+
+6. Update the page style to use the new font, and we're done!
+
+ // tpl.ejs continued
+ <style>
+ body { font-family: "Open Sans", "sans-serif"; }
+ </style>
+ <p>the time is <%= currentTime %>.
+
+The CSS generated by ```connect-fonts``` is based on the user's locale and browser. Here's an example for the 'en' localized subset of Open Sans:
+
+ // this is the output with the middleware's ua param set to 'all'.
+ @font-face {
+ font-family: 'Open Sans';
+ font-style: normal;
+ font-weight: 400;
+ src: url('/fonts/en/opensans-regular.eot');
+ src: local('Open Sans'),
+ local('OpenSans'),
+ url('/fonts/en/opensans-regular.eot#') format('embedded-opentype'),
+ url('/fonts/en/opensans-regular.woff') format('woff'),
+ url('/fonts/en/opensans-regular.ttf') format('truetype'),
+ url('/fonts/en/opensans-regular.svg#Open Sans') format('svg');
+ }
+
+If you don't want to incur the cost of an extra HTTP request, you can use the ```connect_fonts.generate_css()``` method to return this CSS as a string, then insert it into your existing CSS files as part of a build/minification process.
+
+That does it! Our little app is serving up stylish timestamps. This example code is available on [github](https://github.com/6a68/connect-fonts-example) and [npm](https://npmjs.org/package/connect-fonts-example) if you want to play with it.
+
+We've covered a pre-made font pack, but it's easy to create your own font packs for paid fonts. There are instructions on the ```connect-fonts``` [readme](https://github.com/shane-tomlinson/connect-fonts).
+
+## Wrapping up
+
+Font subsetting can bring huge performance gains to sites that use web fonts; ```connect-fonts``` handles a lot of the complexity if you self-host fonts in an internationalized Connect app. If your site isn't yet internationalized, you can still use ```connect-fonts``` to serve up your native subset, and it'll still generate ```@font-face``` CSS and any necessary CORS headers for you, plus you'll have a smooth upgrade path to internationalize later.
+
+### Future directions
+
+Today, ```connect-fonts``` handles subsetting based on locale. What if it also stripped out font hinting for platforms that don't need it (everything other than Windows)? What if it also optionally gzipped fonts and added far-future caching headers? There's cool work yet to be done. If you'd like to contribute ideas or code, we'd love the help! Grab the [source](https://github.com/shane-tomlinson/connect-fonts) and dive in.
View
172 taming_configurations_with_node-convict.md
@@ -0,0 +1,172 @@
+Taming Configurations with node-convict
+=====
+
+In this installment of "A Node.JS Holiday Season" series we'll take a look at [`node-convict`](https://github.com/lloyd/node-convict), a tool that helps manage the configuration of node.js applications. It provides transparent defaults and built-in typing to make errors easier to find and debug.
+
+# The Problem
+
+There are two main concerns regarding application configuration:
+
+* Most applications will have at least a few different deployment environments, each with their own configuration needs.
+* Including credentials and sensitive information in source can be problematic.
+
+These concerns could be addressed by initializing certain settings based on the environment, and using environmental variables for more sensitive settings. A common pattern used by node.js developers is to create a module that exports the configuration, e.g.:
+
+ var conf = {
+ // the application environment
+ // "production", "development", or "test
+ env: process.env.NODE_ENV || "development",
+
+ // the IP address to bind
+ ip: process.env.IP_ADDRESS || "127.0.0.1",
+
+ // the port to bind
+ port: process.env.PORT || 0
+
+ // database settings
+ database: {
+ host: process.env.DB_HOST || "localhost:8091",
+ }
+ };
+
+ module.exports = conf;
+
+This works well enough, but there are additional concerns:
+
+* **What if a setting was configured incorrectly?** We can save headaches by detecting and reporting misconfigurations as early as possible.
+* **How easily is it understood** by Ops/QA/and other collaborators that may need to adjust settings or diagnose issues? A more declarative format that also embeds documentation can make lives easier.
+
+# Enter convict
+
+Convict addresses these additional concerns by introducing a **configuration schema** where you can set **type information, default values, environmental variables, and documentation** for each setting.
+
+With convict, the example from above becomes:
+
+ var convict = require('convict');
+
+ var conf = convict({
+ env: {
+ doc: "The applicaton environment.",
+ format: ["production", "development", "test"],
+ default: "development",
+ env: "NODE_ENV"
+ },
+ ip: {
+ doc: "The IP address to bind.",
+ format: "ipaddress",
+ default: "127.0.0.1",
+ env: "IP_ADDRESS"
+ },
+ port: {
+ doc: "The port to bind.",
+ format: "port",
+ default: 0,
+ env: "PORT"
+ },
+ database: {
+ host: {
+ default: "localhost:8091",
+ env: "DB_HOST"
+ }
+ }
+ });
+
+ conf.validate();
+
+ module.exports = conf;
+
+The information is more or less the same, but encoded in the schema. Since all of the information is encoded in the schema, we can export it and display it in an easier to read format, and we can use it for validation. This declarative approach is what helps convict be more robust and collaborator friendly.
+
+# What's in a Schema
+ You'll notice four possible properties for each setting – each aiding in our goal of a more robust and easily digestible configuration.
+
+* **Type information**: the `format` property specifies either a built-in convict format (`ipaddress`, `port`, `int`, etc.), or it can be a function to check a custom format. During validation, if a format check fails it will be added to the error report.
+* **Default values**: Every setting *must* have a default value.
+* **Environmental variables**: If the variable specified by `env` has a value, it will overwrite the setting's default value.
+* **Documentation**: The `doc` property is pretty self-explanatory. The nice part about having it in the schema rather than as a comment is that we can call `conf.toSchemaString()` and have it displayed in the output.
+
+# Layering Additional Configurations
+The set of defaults becomes a base configuration on top of which you can overlay additional configurations, using `conf.load` and `conf.loadFile`. For example, you could conditionally overlay a JavaScript object with settings for a specific environment:
+
+
+ var conf = convict({
+ // snip ... assume the same schema as above
+ });
+
+ if (conf.get('env') === 'production') {
+ // use the production port and database host
+ conf.load({
+ port: 8080
+ database: {
+ host: "ec2-117-21-174-242.compute-1.amazonaws.com:8091"
+ }
+ });
+ }
+
+ conf.validate();
+
+ module.exports = conf;
+
+
+Alternatively, if you create a separate configuration file for each environment, you could simply overlay them with `loadFile`:
+
+ conf.loadFile('./config/' + conf.get('env') + '.json');
+
+`loadFile` can also load multiple files at once, by passing in an array:
+
+ // CONFIG_FILES=/path/to/production.json,/path/to/secrets.json,/path/to/sitespecific.json
+ conf.loadFile(process.env.CONFIG_FILES.split(','));
+
+Layering configurations with `load` and `loadFile` is useful if you have common settings for specific environments that don't *need* to be set in environmental variables. Having separate, declarative JSON configurations can provide greater visibility of which settings should change between environments. And the files are loaded with [cjson](https://github.com/kof/node-cjson), so they may contain as many comments as you want for additional clarification.
+
+Also note that **environmental variables always take precedent**, even after settings have been loaded using `load` or `loadFile`. To see what the current settings look like, you can serialize the whole thing using `conf.toString()`.
+
+# V for Validation
+
+Once the settings have been composed, you can perform validation to check that each setting's value conforms to the correct format, as defined in the schema. Convict provides a few built-in formats such as `"url"`, `"ports"`, and `"ipaddress"`, among others, and you may use one of JavaScript's global constructors (e.g. `Number`) to designate a type. If you leave the `format` property out entirely, convict will check that the setting has the same type as the default value (according to [`Object.prototype.toString.call`](http://perfectionkills.com/instanceof-considered-harmful-or-how-to-write-a-robust-isarray/)). For instance, the following three schemas are equivalent:
+
+ var conf1 = convict({
+ name: {
+ format: String
+ default: 'Brendan'
+ }
+ });
+
+ // with no format specified, convict will assume it's the type
+ // of the default value
+ var conf2 = convict({
+ name: {
+ default: 'Brendan'
+ }
+ });
+
+ // a more succinct version
+ var conf3 = convict({
+ name: 'Brendan'
+ });
+
+Additionally, there is an enum-style format as seen in the examples, in which you can specify an explicit set of acceptable values, e.g. `["production", "development", "test"]`. Any value not in the list will fail validation.
+
+In place of a built-in type, you can also provide your own format checking function. For example, the format function in this schema checks that the setting is a 64 character hex string:
+
+ var check = require('validator').check;
+
+ var conf = convict({
+ key: {
+ doc: "API key",
+ format: function (val) {
+ check(val, 'should be a 64 character hex key').regex(/^[a-fA-F0-9]{64}$/);
+ },
+ default: '3cec609c9bc601c047af917a544645c50caf8cd606806b4e0a23312441014deb'
+ }
+ });
+
+Calling `conf.validate()` will throw an error with details on each setting that failed to validate, if there were any. This is important for avoiding redeployment after each individual configuration error. Using the configuration from the custom key format example, here's what an error would look like:
+
+ conf.set('key', 'foo');
+ conf.validate();
+ // Error: key: should be a 64 character hex key: value was "foo"
+
+
+# Conclusion
+Convict expands on the standard pattern of configuring node.js applications in a way that is more robust and accessible to collaborators, who may have less interest in digging through imperative code in order to inspect or modify settings. With the configuration schema, we can give project collaborators more **context** on each setting as well as provide **validation and early failures** when configuration goes wrong.
Please sign in to comment.
Something went wrong with that request. Please try again.