Storage-viz visualizers

README.md

@@ -1,2 +1,99 @@
-mongo-storage-info
-==================
+Storage-viz, visualize Mongo's storage and indexes
+==================================================
+
+© 2012 10gen, the MongoDB Company
+
+Authors: Andrea Lattuada
+
+**The commands used by this tool are currently EXPERIMENTAL and UNSUPPORTED.**
+
+Storage-viz is a suite of web-based visualizers that leverage new database commands:
+they make it easier to understand and analyze MongoDB storage and btree layout.
+
+OVERVIEW
+--------
+
+The `storageDetails` command will aggregate statistics related to the
+storage layout (when invoked with `analyze: "diskStorage"`) or the percentage
+of pages currently in RAM (when invoked with `analyze: "pagesInRAM"`) for the
+specified collection, extent or part of extent.
+
+The `indexStats` command provides detailed and aggregate information and
+statistics for the underlying btree of a particular index.
+Stats are aggregated for the entire tree, per-depth and, if requested through
+the `expandNodes` option, per-subtree.
+
+Both commands take a global READ_LOCK and will page in all the extents or btree
+buckets encountered: this will have adverse effects on server performance.
+The commands should never be run on a primary and will cause a secondary to
+fall behind on replication. `diskStorage` when run with
+`analyze: "pagesInRAM"` is the exception as it typically returns rapidly and
+may only page in extent headers.
+
+USAGE
+-----
+
+To use the commands and visualizers you need a recent MongoDB Nightly build.
+
+You can download a MongoDB Nightly binary from http://www.mongodb.org/downloads
+(Nightly builds are generated from the unstable branch and should not be
+used in production).
+
+You can enable the experimental commands with
+
+        --enableExperimentalStorageDetailsCmd
+or
+
+        --enableExperimentalIndexStatsCmd
+
+**NOTE: running mongod with these options is unsafe and not advisable for
+production servers.**
+
+If you'd like to run the commands directly within the shell, helpers are
+available.
+
+Json output:
+
+    db.collection.diskStorageStats({...})
+    db.collection.pagesInRAM({...})
+    db.collection.indexStats({index: "index name", ...})
+
+Their counterparts providing human-readable output follow.
+
+    db.collection.getDiskStorageStats({...})
+    db.collection.getPagesInRAM({...})
+    db.collection.getIndexStats({...})
+
+VISUALIZERS
+-----------
+
+To use the visualizers the server needs to be started with the `--rest --jsonp` command
+line flags.
+
+**NOTE: running mongod with these options is unsafe and not advisable for
+production servers.**
+**`--rest` will allow everyone on the same network as the server to query or
+modify data. Please refer to http://en.wikipedia.org/wiki/JSONP#Security_concerns
+for security concerns related to `--jsonp`.**
+
+The *visualizers* provide a nicer graphical representation but are very early stage
+and have only been tested in Chrome.
+The source code for them is available in this repository.
+
+http://10gen-labs.github.com/storage-viz/diskStorage.html displays storage layout
+and usage.
+
+http://10gen-labs.github.com/storage-viz/indexStats.html shows statistics related
+to the indexing btrees.
+
+http://10gen-labs.github.com/storage-viz/pagesInRAM.html reports which parts of a
+collection is currently in ram.
+
+ADDITIONAL
+----------
+
+If you'd like to report a bug or request a new feature,
+please file an issue on our github repository
+(you must be logged into github to do this):
+
+https://github.com/10gen-labs/storage-viz/issues/new

base.css

@@ -0,0 +1,111 @@
+/*
+Copyright (C) 2012 10gen Inc.
+
+This program is free software: you can redistribute it and/or  modify
+it under the terms of the GNU Affero General Public License, version 3,
+as published by the Free Software Foundation.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+/* --- GRID --- */
+body {
+    font-family: sans-serif;
+    font-size: 10pt;
+    color: #555;
+    /*padding-left: 20px;*/
+    /*padding-right: 20px;*/
+}
+
+table {
+    font-size: 10pt;
+}
+
+* {
+    margin: 0px;
+    padding: 0px;
+}
+
+#container {
+    margin: 0 auto;
+    min-width: 940px;
+}
+
+.grid-td > .grid-tr,
+#container > .grid-tr {
+    width: 100%;
+    display: table;
+    border-collapse: collapse;
+}
+
+.grid-tr {
+    display: table-row;
+}
+
+.grid-td {
+    display: table-cell;
+    border-right: 1px solid #bbb;
+    border-bottom: 1px solid #bbb;
+    min-height: 10px;
+    padding: 6px 10px 6px 10px;
+    vertical-align: top;
+}
+
+.grid-half {
+    width: 50%;
+}
+
+.grid-td:last-child {
+    border-right: none;
+}
+
+/* --- FORMS --- */
+button {
+    padding: 0.3em;
+}
+
+label {
+    margin-right: 0.5em;
+}
+
+input {
+    margin-right: 1em;
+    border: 1px solid;
+    padding: 1px;
+}
+
+/* --- STYLES --- */
+#header {
+    background-color: rgb(64, 40, 23);
+    color: #ccc;
+    border-bottom: none;
+}
+
+#basicInfoRow {
+    font-weight: bold;
+}
+
+.left-table-header {
+    width: 20px;
+    position: relative;
+}
+
+.left-table-header > span {
+    writing-mode: tb-rl;
+    -webkit-transform: rotate(-90deg);
+    -moz-transform: rotate(-90deg);
+    -o-transform: rotate(-90deg);
+    margin-top: 40px;
+    white-space: nowrap;
+    display: block;
+    width: 20px;
+    height: 20px;
+    top: 30px;
+}
+

base.js

@@ -0,0 +1,114 @@
+
+/*
+Copyright (C) 2012 10gen Inc.
+
+This program is free software: you can redistribute it and/or  modify
+it under the terms of the GNU Affero General Public License, version 3,
+as published by the Free Software Foundation.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+(function(){
+
+d3.selection.prototype.value = function(val) {
+    if (arguments.length < 1) return this.property('value');
+    else this.property('value', val);
+}
+
+var base = this.base = {};
+
+// Simple JavaScript Templating
+// John Resig - http://ejohn.org/ - MIT Licensed
+var cache = {};
+
+base.tmpl = function Tmpl(str, data){
+    // Figure out if we're getting a template, or if we need to
+    // load the template - and be sure to cache the result.
+    var fn = !/\W/.test(str) ?
+        cache[str] = cache[str] :
+
+        // Generate a reusable function that will serve as a template
+        // generator (and which will be cached).
+        new Function("obj",
+            "var p=[],print=function(){p.push.apply(p,arguments);};" +
+
+            // Introduce the data as local variables using with(){}
+            "with(obj){p.push('" +
+
+            // Convert the template into pure JavaScript
+            str
+              .replace(/[\r\t\n]/g, " ")
+              .split("<%").join("\t")
+              .replace(/((^|%>)[^\t]*)'/g, "$1\r")
+              .replace(/\t=(.*?)%>/g, "',$1,'")
+              .split("\t").join("');")
+              .split("%>").join("p.push('")
+              .split("\r").join("\\'")
+        + "');}return p.join('');");
+
+    // Provide some basic currying to the user
+    return data ? fn( data ) : fn;
+};
+
+base.jsonp = function(url, callbackName) {
+    var script = document.createElement('script');
+    script.src = url + "&limit=1&jsonp=" + callbackName;
+    document.getElementsByTagName('body')[0].appendChild(script);
+};
+
+base.generateFormFields = function(selection, fields, onClick) {
+    fields.map(function(x) {
+        selection.append('label').attr('for', x.name).text(x.desc);
+        var input = selection.append('input').attr('name', x.name).attr('type', x.type);
+        if (x.default_) input.value(x.default_);
+    });
+    selection.append('button').text('submit').on('click', onClick);
+};
+
+base.collectFormValues = function(selection, fields) {
+    var params = {};
+    fields.map(function(x) {
+        params[x.name] = selection.select('input[name=' + x.name + ']').value();
+    });
+    return params;
+};
+
+base.property = function(obj, name, default_) {
+    obj['_' + name] = default_;
+    obj[name] = function(_) {
+        if (!arguments.length) return obj['_' + name];
+        else obj['_' + name] = _;
+        return this;
+    };
+};
+
+base.fmt = {
+    percent: function(val) { return val.toFixed(3) }, //d3.format('.3p'),
+    ratioToPercent: function(val) { return (val * 100).toFixed(1) + '%' },
+    percentAndErr: function(val, err) {
+        return this.percent(val) + ' (&plusmn;' + this.percent(err) + ')';
+    },
+    suffix: function(bytes) {
+        if( bytes < 1024 )
+            return Math.floor( bytes ) + "b";
+        if( bytes < 1024 * 1024 )
+            return Math.floor( bytes / 1024 ) + "kb";
+        if( bytes < 1024 * 1024 * 1024 )
+            return Math.floor( ( Math.floor( bytes / 1024 ) / 1024 ) * 100 ) / 100 + "Mb";
+       return Math.floor( ( Math.floor( bytes / ( 1024 * 1024 ) ) / 1024 ) * 100 ) / 100 + "Gb";
+    },
+    suffixAndBytes: function(val) { return base.fmt.suffix(val) + ' (' + val + ' bytes)' }
+}
+
+base.fmt.stat = {
+    percentAndErr: function(stat) { return base.fmt.percentAndErr(stat.mean, stat.stddev); }
+}
+
+})();