Permalink
Browse files

initial commit

  • Loading branch information...
0 parents commit 1139d9f8bc95e8b6e9eaf1e55fbe573e7aea3bf1 John Lindal committed Mar 22, 2012
Showing with 1,829 additions and 0 deletions.
  1. +34 −0 LICENSE
  2. +147 −0 README.md
  3. +171 −0 combo-dev.js
  4. +1,412 −0 combo.js
  5. +65 −0 manager.js
@@ -0,0 +1,34 @@
+Software Copyright License Agreement (BSD License)
+
+Copyright (c) 2009, Yahoo! Inc.
+All rights reserved.
+
+Redistribution and use of this software in source and binary forms, with or
+without modification, are permitted provided that the following conditions
+are met:
+
+* Redistributions of source code must retain the above
+ copyright notice, this list of conditions and the
+ following disclaimer.
+
+* Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the
+ following disclaimer in the documentation and/or other
+ materials provided with the distribution.
+
+* Neither the name of Yahoo! Inc. nor the names of its
+ contributors may be used to endorse or promote products
+ derived from this software without specific prior
+ written permission of Yahoo! Inc.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
@@ -0,0 +1,147 @@
+YUI 3 combo handler built on NodeJS that supports versioning, either for
+individual modules or for groups of modules. (The name "stockpile" was
+chosen because it's a synonym of gallery, but without the glamorous
+connotation.)
+
+Versioning individual modules requires this directory structure:
+
+ {namespace}/
+ sp-{namespace}-{module-name}/
+ {version}/
+ sp-{namespace}-{module-name}-min.js
+
+Versioning a module bundle requires this directory structure:
+
+ {bundle-name}/
+ {version}/
+ {module-name}/
+ {module-name}-min.js
+
+The raw and debug versions of the JavaScript are optional. The assets
+directory is also optional.
+
+Namespaces avoid naming collisions between modules from different teams.
+The top-level namespace directory helps avoid the file system's limit on
+the number of subdirectories: each namespace can have the maximum number
+of modules, instead of the limit being global.
+
+Version numbers for individual modules must be specified by configuring
+each of the patterns in the YUI Loader group with this configFn:
+
+ function insertVersion(m)
+ {
+ var s = /^sp-([^-]+)-([^\/]+)/.exec(m.path);
+ if (s && s.length)
+ {
+ m.path = m.path.replace(s[0],
+ s[1] + '/' + s[2] + '/' + moduleVersion[s[0]]);
+ }
+ }
+
+where moduleVersion is a map of module names to version numbers. Prefixing
+all modules with "sp-" makes it easy to define a single YUI Loader group
+with this configFn.
+
+The version number for a module bundle must be specified by configuring the
+root of the YUI Loader group:
+
+ myGroup:
+ {
+ root: 'uifwk/14.2.0.0.253/'
+ }
+
+The advantage of using a bundle is that when modules are requested,
+dependencies within the bundle are automatically included. This reduces
+the number of requests made by YUI Loader.
+
+The disadvantage of using a bundle is that all the modules must be
+published simultaneously, inside the {bundle-name}/{version}/ directory.
+
+Installation
+------------
+
+Install nodejs and then install these packages:
+
+ cd YUI-3-Stockpile
+ npm install yui3 express ejs optimist gzip long-stack-traces
+
+Usage
+-----
+
+Start the combo handler:
+
+ cd YUI-3-Stockpile
+ node combo.js [--config path_to_config_file]
+ [--path path_to_repository] [--port port]
+ [--cache [size_MB] [--cache-log path_to_dump_logs]
+ [--cache-log-interval log_dump_interval_hours]]
+ [--debug]
+
+The default config files are:
+
+ combo server: /usr/share/yui3-stockpile/combo.json
+ admin UI: /usr/share/yui3-stockpile/manager.json
+
+Command line arguments override the values in the config files.
+
+The default paths are:
+
+ combo server repository: /var/yui3-stockpile
+ combo server cache log: /var/log/yui3-stockpile
+
+Code Organization
+-----------------
+
+ combo.js Combo handler
+ combo-dev.js Combo handler for development mode
+ manager.js Admin UI
+ client Client modules
+ server Server modules
+ views Client web pages
+
+Caching
+-------
+
+We only cache the results for minified requests. Raw and debug versions
+are typically requested only in debug mode, which is rare. Unfortunately,
+not all requests will accept compressed responses, so the cache must store
+both raw and gzipped versions.
+
+The cache key is the sorted list of requested files. The cache itself is a
+map of keys to { response data, reference to MRU item }. The MRU items
+(each of which stores a key) are stored in a doubly linked list to allow
+easy re-ordering and dropping of oldest items.
+
+The cache is memory-limited, so when an item is added, the total size is
+updated, and then items are removed from the end of the MRU linked list
+until the size drops back within the prescribed limit.
+
+For tuning, the cache logs the following data: total # of hits and a map
+of cache keys to { # of puts, # of gets, response size }. This can be
+plotted on a graph of log(response size) vs % of total hits. Keys with
+high percentages indicate thrashing. The response size helps determine how
+much the cache should be expanded.
+
+Dependencies
+------------
+
+When a bundle is uploaded, the "requires" configuration for each module is
+updated to include transitive dependencies within the bundle. This ensures
+that YUI Loader will only have to make two requests to get all the required
+modules within the bundle.
+
+Development Mode
+----------------
+
+To enable rapid iteration during development, configure an instance of
+combo-dev.js to load your raw source files for the modules you are
+currently working on. All other modules will be requested from the
+fallback combo handler, usually an instance of combo.js.
+
+combo-dev.js requires a config file, because you typically run a separate
+instance for each project that you are working on:
+
+ cd YUI-3-Stockpile
+ node combo-dev.js --config path_to_config_file [--port port] [--debug]
+
+Command line arguments override the values in the config files.
@@ -0,0 +1,171 @@
+#!/usr/bin/env node
+
+/* Copyright (c) 2012 Yahoo! Inc. All rights reserved.
+ *
+ * The copyrights embodied in the content of this file are licensed by
+ * Yahoo! Inc. under the BSD (revised) open source license.
+ */
+
+var YUI = require('yui3').YUI;
+YUI({
+ gallery: 'gallery-2012.01.11-21-03'
+}).use('json', 'gallery-funcprog', function(Y)
+{
+
+var fs = require('fs'),
+ path = require('path'),
+ url = require('url'),
+ http = require('http'),
+ express = require('express');
+
+// options
+
+var argv = require('optimist')
+ .usage('usage: $0')
+ .option('config',
+ {
+ demand: true,
+ describe: 'Path to configuration file'
+ })
+ .option('port',
+ {
+ describe: 'Port to listen on'
+ })
+ .option('debug',
+ {
+ boolean: true,
+ describe: 'Turn on debugging (causes leaks)'
+ })
+ .argv;
+
+var config = Y.JSON.parse(fs.readFileSync(argv.config));
+config.port = argv.port || config.port;
+
+var debug = argv.debug || config.debug;
+if (debug)
+{
+ require('long-stack-traces');
+}
+
+var app = express.createServer();
+
+function moduleName(s)
+{
+ var m = /([^\/]+?)(-(min|debug))?\.js$/.exec(s);
+ return (m && m.length && m[1]);
+}
+
+app.get('/combo', function(req, res)
+{
+ var query = url.parse(req.url).query;
+ if (!query)
+ {
+ res.end();
+ return;
+ }
+
+ res.setHeader('Content-Type', /\.css/.test(query) ? 'text/css' : 'text/javascript');
+ res.setHeader('Cache-Control', 'no-cache');
+ res.setHeader('Expires', 'Wed, 31 Dec 1969 16:00:00 GMT');
+
+ var module = Y.partition(query.split('&'), function(m)
+ {
+ var name = moduleName(m);
+ return (name && config.modules[ name ]);
+ });
+
+ var module_list = module.rejects.join('&');
+
+ var file_list = Y.reduce(module.matches, [], function(list, m)
+ {
+ Y.each(config.modules[ moduleName(m) ], function(f)
+ {
+ list.push(path.resolve(config.root || '', f));
+ });
+ return list;
+ });
+
+ var file_index = 0,
+ files_done = true,
+ relay_done = true;
+
+ if (module_list)
+ {
+ relay_done = false;
+ var relay_url = config.combo + module_list,
+ relay_data = [];
+
+ Y.log('relay: ' + relay_url, 'debug', 'combo-dev');
+
+ http.get(url.parse(relay_url), function (r)
+ {
+ r.on('data', function(data)
+ {
+ relay_data.push(data);
+ });
+
+ r.on('end', function()
+ {
+ res.write(relay_data.join(''), 'utf-8');
+
+ relay_done = true;
+ checkFinished();
+ });
+ })
+ .on('error', function(err)
+ {
+ Y.log(err.message + ' from ' + relay_url, 'warn', 'combo-dev');
+ relay_done = true;
+ checkFinished();
+ });
+ }
+
+ if (file_list.length > 0)
+ {
+ Y.log('files: ' + file_list, 'debug', 'combo-dev');
+
+ files_done = false;
+ sendFile(file_list[0]);
+ }
+
+ checkFinished();
+
+ function sendFile(f)
+ {
+ fs.readFile(f, 'utf-8', function(err, data)
+ {
+ if (err)
+ {
+ Y.log(err.message, 'warn', 'combo-dev');
+ }
+ else
+ {
+ res.write(data, 'utf-8');
+ }
+
+ file_index++;
+ if (file_index >= file_list.length)
+ {
+ files_done = true;
+ checkFinished();
+ }
+ else
+ {
+ sendFile(file_list[file_index]);
+ }
+ });
+ }
+
+ function checkFinished()
+ {
+ if (files_done && relay_done)
+ {
+ res.end();
+ }
+ }
+});
+
+Y.log('listening on port ' + config.port, 'debug', 'combo-dev');
+app.listen(config.port);
+
+});
Oops, something went wrong. Retry.

0 comments on commit 1139d9f

Please sign in to comment.