Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
branch: master
Fetching contributors…

Cannot retrieve contributors at this time

1046 lines (561 sloc) 30.542 kb
<!DOCTYPE html>
<!--[if IEMobile 7 ]><html class="no-js iem7"><![endif]-->
<!--[if lt IE 9]><html class="no-js lte-ie8"><![endif]-->
<!--[if (gt IE 8)|(gt IEMobile 7)|!(IEMobile)|!(IE)]><!--><html class="no-js" lang="en"><!--<![endif]-->
<head>
<meta charset="utf-8">
<title>ql.io blog</title>
<meta name="author" content="ql.io">
<meta name="description" content="Should backend servers send XML or Protobuf responses to ql.io? That is the question this post addresses. I setup a ql.io application hitting a mock &hellip;">
<!-- http://t.co/dKP3o1e -->
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="canonical" href="http://ql-io.github.com/index.html">
<link href="/favicon.ico" rel="icon">
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
<script src="/javascripts/modernizr-2.0.js"></script>
<script src="/javascripts/ender.js"></script>
<script src="/javascripts/octopress.js" type="text/javascript"></script>
<link href="/atom.xml" rel="alternate" title="ql.io blog" type="application/atom+xml">
<!--Fonts from Google"s Web font directory at http://google.com/webfonts -->
<link href="http://fonts.googleapis.com/css?family=PT+Serif:regular,italic,bold,bolditalic" rel="stylesheet" type="text/css">
<link href="http://fonts.googleapis.com/css?family=PT+Sans:regular,italic,bold,bolditalic" rel="stylesheet" type="text/css">
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-21835406-5']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body >
<p class="top-links">
<span style="float:left"><a href="/">ql.io blog</a></span>
<a href="http://ql.io/">project</a> |
<a href="https://github.com/ql-io/ql.io/">github</a> |
<a href="/blog/archives">archives</a> |
<a href="/atom.xml">subscribe</a>
</p>
<nav role="navigation"><!--<ul class="subscription" data-subscription="rss">-->
<!--<li><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>-->
<!---->
<!--</ul>-->
<!---->
<!--<form action="http://google.com/search" method="get">-->
<!--<fieldset role="search">-->
<!--<input type="hidden" name="q" value="site:ql-io.github.com" />-->
<!--<input class="search" type="text" name="q" results="0" placeholder="Search"/>-->
<!--</fieldset>-->
<!--</form>-->
<!---->
<!--<ul class="main-navigation">
<li><a href="/">Blog</a></li>
<li><a href="https://github.com/ql-io/ql.io/">Github</a></li>
<li><a href="/blog/archives">Archives</a></li>
</ul>
-->
</nav>
<div id="main">
<div id="content">
<div class="blog-index">
<article>
<header>
<h1 class="entry-title"><a href="/2012/07/20/xml-protobuf-performance.html">ql.io XML/Protobuf Performance</a></h1>
<p class="meta">
<time datetime="2012-07-20T00:00:00-07:00" pubdate data-updated="true">Jul 20<span>th</span>, 2012</time>
| <a href="/2012/07/20/xml-protobuf-performance.html#disqus_thread">Comments</a>
</p>
<p>
<span class="byline author vcard">Posted by <a href="http://github.com/idralyuk">Igor Dralyuk</a></span>
</p>
</header>
<div class="entry-content"><p>Should backend servers send XML or Protobuf responses to ql.io? That is the question this post addresses.</p>
<p>I setup a ql.io application hitting a mock server and ran JMeter to generate load and collect results.</p>
<p>Source code for the app is on <a href="https://github.com/idralyuk/ql.io-protobuf-test">Github</a>.
Mock server is <a href="https://github.com/idralyuk/ql.io-protobuf-test/tree/master/mock_server">here</a> and JMeter script is <a href="https://github.com/idralyuk/ql.io-protobuf-test/tree/master/jmeter">here</a>.</p>
<p>ql.io tables for eBay&#8217;s Marketplaces APIs are <a href="https://github.com/ql-io/ql.io-ebay-mp-apis">here</a>.</p>
<p>The table used for this test is <a href="https://github.com/ql-io/ql.io-ebay-mp-apis/blob/master/tables/finding/findItemsByKeywords.ql">findItemsByKeywords.ql</a>.</p>
<h3>Test Setup</h3>
<p><strong>Server</strong>: Dev workstation ca. 2011 (Dell Precision T5500 with Intel Xeon E5630 2.53GHz 24Gb RAM), Linux 3.0.0-20-generic #34-Ubuntu SMP Tue May 1 17:24:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux</p>
<p><strong>Client</strong>: Dev workstation ca. 2007 (Dell Precision 690 Intel Xeon DP 5060 3.2GHz, 8GB RAM), SunOS 5.11 joyent_20120517T192048Z i86pc i386 i86pc</p>
<p>The server ran node v0.6.18a and ql.io app 0.7.4. On the same box another node process was serving canned xml and protobuf responses from eBay&#8217;s <a href="http://developer.ebay.com/devzone/finding/callref/findItemsByKeywords.html">FindingService</a> on port 6000 (it was on the same box in order to take the network out of the equation).</p>
<p>The test app was running on port 3000, with two route/table/patch combinations (one for XML, one for Protobuf) pointing to the above response server on localhost:6000.</p>
<p>The client was JMeter 2.7, running in server mode (jmeter-server), with HEAP=&#8221;-Xms1024m -Xmx1024m&#8221;.</p>
<p>See Appendix for implementation details.</p>
<h3>Test 1: 200 users, running for 1 hour, hitting the XML path</h3>
<p><strong>Rate: 203.43 trans/sec. Average response time: 72 ms.</strong></p>
<pre><code>Number of Samples: 734,339
Average Response Time: 72 ms
Minimim Response Time: 25 ms
Maximum Response Time: 1,002 ms
Standard Deviation: 49 ms
Error Percentage: 0.00 %
Transaction rate: 203.43 trans/sec
Throughput: 12,248 KB/sec
</code></pre>
<h3>Test 2: 200 users, running for 1 hour, hitting the Protobuf path</h3>
<p><strong>Rate: 213.91 trans/sec. Average response time: 24 ms.</strong></p>
<pre><code>Number of Samples: 772,200
Average Response Time: 24 ms
Minimim Response Time: 8 ms
Maximum Response Time: 198 ms
Standard Deviation: 26 ms
Error Percentage: 0.00 %
Transaction rate: 213.91 trans/sec
Throughput: 12,388 KB/sec
</code></pre>
<h3>Discussion</h3>
<p>The results show that Protobuf is <strong><em>three</em></strong> times faster than XML!</p>
<p>Is it surprising? Yes. Even with the optimize_for = SPEED option turned on in the .proto file, this is a bit extreme.</p>
<p>The exercise of comparing Protobuf to JSON is left to the reader. It is reasonable to assume that they should be on par, as any conversion step is going to be slower than the native format.</p>
<p>This test doesn&#8217;t take into account network latency. Payload sizes play an important role when network is involved; for the resultset being used in the test (50 items) the sizes were as follows:</p>
<pre><code>mock.xml - 83 KB
mock.protobuf - 27 KB
</code></pre>
<h3>Conclusion</h3>
<p>According to the results of the test, it certainly makes sense to use Protobuf instead of XML.</p>
<p>Further testing (involving the network), needs to be done in order to determine whether protobuf is a better solution than json, but given that there is a significant reduction in payload size, Protobuf is likely to come out a winner in that test as well.</p>
<h2>Appendix</h2>
<h3>URLs hit by JMeter (client)</h3>
<pre><code>http://10.xx.xx.xx:3000/ebay/finding/keywords/xml/ipad
http://10.xx.xx.xx:3000/ebay/finding/keywords/protobuf/ipad
</code></pre>
<h3>Routes (server)</h3>
<pre><code>return select searchResult.item, errorMessage
from ebay.finding.findItemsByKeywordsXML where keywords = '{keywords}'
via route '/ebay/finding/keywords/xml/{keywords}' using method get;
return select searchResult.item, errorMessage
from ebay.finding.findItemsByKeywordsProtobuf where keywords = '{keywords}'
via route '/ebay/finding/keywords/protobuf/{keywords}' using method get;
</code></pre>
<h3>Tables (server)</h3>
<pre><code>create table ebay.finding.findItemsByKeywordsXML
on select post to 'http://localhost:6000/mock.xml'
using headers 'X-EBAY-SOA-SECURITY-APPNAME'='{config.tables.ebay.finding.appname}',
'X-EBAY-SOA-OPERATION-NAME'='findItemsByKeywords'
using defaults format = "JSON", limit = 5, offset = 0
using patch 'findItemsByKeywordsXML.js'
using bodyTemplate "findItemsByKeywords.ejs" type 'application/xml'
resultset 'soapenv:Envelope.soapenv:Body.findItemsByKeywordsResponse'
create table ebay.finding.findItemsByKeywordsProtobuf
on select post to 'http://localhost:6000/mock.protobuf'
using headers 'X-EBAY-SOA-SECURITY-APPNAME'='{config.tables.ebay.finding.appname}',
'X-EBAY-SOA-OPERATION-NAME'='findItemsByKeywords'
using defaults format = "JSON", limit = 5, offset = 0
using patch 'findItemsByKeywordsProtobuf.js'
using bodyTemplate "findItemsByKeywords.ejs" type 'application/xml'
resultset 'findItemsByKeywordsResponse'
</code></pre>
<h3>Patches (server)</h3>
<p>The following patch was used in the XML path: <a href="https://github.com/idralyuk/ql.io-protobuf-test/blob/master/tables/finding/findItemsByKeywordsXML.js">findItemsByKeywordsXML.js</a>.</p>
<p>This additional code was inserted into the above patch to decode Protobuf responses:</p>
<pre><code>var fs = require('fs'),
_ = require('underscore'),
Schema = require('protobuf').Schema,
fis_schema = new Schema(fs.readFileSync(__dirname + '/util/FindItemsByKeywords.desc')),
FindItemsByKeywordsResponse = fis_schema['com.ebay.marketplace.search.v1.services.finditemservice.FindItemsByKeywordsResponse'];
exports['parse response'] = function(args) {
var length = 0, idx = 0;
_.each(args.body, function(b) {
length += b.length;
});
var buf = new Buffer(length);
_.each(args.body, function(b) {
idx = idx + b.copy(buf, idx);
});
var fir = { 'findItemsByKeywordsResponse' : FindItemsByKeywordsResponse.parse(buf) };
return {
type: 'application/json',
content: JSON.stringify(fir)
};
}
</code></pre>
<p><strong>Note: two optimizations can be made to the above code: a) allow the patch to return the json structure instead of a string that will need to be parsed again and b) receive buffer length as an argument in order to avoid looping through the data buffers twice.</strong></p>
<h3>Mock Server</h3>
<pre><code>var _ = require('underscore'),
fs = require('fs'),
url = require('url'),
util = require('util'),
http = require('http');
var port = 6000;
function endsWith(str, suffix) {
return str.indexOf(suffix, str.length - suffix.length) !== -1;
}
var server = http.createServer(function(req, res) {
var file = __dirname + '/data/' + req.url
var cType;
if (endsWith(req.url, '.xml')) {
cType = 'text/xml;charset=UTF-8';
} else if (endsWith(req.url, '.json')) {
cType = 'application/json;charset=UTF-8';
} else if (endsWith(req.url, '.protobuf')) {
cType = 'application/octet-stream;charset=UTF-8';
}
var stat = fs.statSync(file);
res.writeHead(200, {
'Content-Type' : cType,
'Content-Length' : stat.size
});
var readStream = fs.createReadStream(file);
util.pump(readStream, res, function(e) {
if (e) {
console.log(e.stack || e);
}
res.end();
});
});
server.listen(port, function() {
console.log('\nmock server listening on ' + port);
});
</code></pre>
<p>Please send comments/suggestions to <a href="http://groups.google.com/group/qlio">ql.io Google Group</a>.</p>
</div>
</article>
<article>
<header>
<h1 class="entry-title"><a href="/2012/06/29/ql.io-0.7.html">ql.io 0.7</a></h1>
<p class="meta">
<time datetime="2012-06-29T00:00:00-07:00" pubdate data-updated="true">Jun 29<span>th</span>, 2012</time>
| <a href="/2012/06/29/ql.io-0.7.html#disqus_thread">Comments</a>
</p>
<p>
<span class="byline author vcard">Posted by <a href="http://github.com/prabhakhar">Prabhakhar Kaliamurthy</a></span>
</p>
</header>
<div class="entry-content"><p>Today&#8217;s release of ql.io 0.7 includes the following changes:</p>
<h3>Features</h3>
<ul>
<li>Fallback syntax to the language - see https://github.com/ql-io/ql.io/wiki/%5BProposal%5D-Optional-Inputs-and-Errors</li>
<li>Compiler rewritten to output the DAG with dependencies</li>
<li>Explicit depedencies between modules</li>
<li>Support for pre-requisite params - see https://github.com/ql-io/ql.io/wiki/%5BProposal%5D-Optional-Inputs-and-Errors</li>
<li>Retry once for idempotent requests on timeouts</li>
<li>Update the context with the udf filtered data</li>
<li>Support C style block comments</li>
<li>No Compression if the CPU load is > 50%</li>
</ul>
<h3>Bug Fixes</h3>
<ul>
<li>Http client Agent maxSockets increased to 1000 to avoid request backlog on any given socket - https://github.com/ql-io/ql.io/issues/512</li>
<li>Fix https://github.com/ql-io/ql.io/issues/478</li>
<li>Fix https://github.com/ql-io/ql.io/issues/13 Disable autorun in the console.</li>
<li>Add file path/name to comiple errors</li>
</ul>
</div>
</article>
<article>
<header>
<h1 class="entry-title"><a href="/2012/06/08/plan.html">Evented Orchestration</a></h1>
<p class="meta">
<time datetime="2012-06-08T00:00:00-07:00" pubdate data-updated="true">Jun 8<span>th</span>, 2012</time>
| <a href="/2012/06/08/plan.html#disqus_thread">Comments</a>
</p>
<p>
<span class="byline author vcard">Posted by <a href="http://github.com/s3u">Subbu Allamaraju</a></span>
</p>
</header>
<div class="entry-content"><p>One of the core strengths of ql.io is evented orchestration of reads and writes to HTTP APIs using a
declarative language. In recent weeks, the core processing algorithm used to process q.l.io scripts
went through an overhaul to easily infer what goes on when you submit a script for execution. The
outcome of this exercise is a rewrite of the compiler which now takes a given script and outputs an
execution plan. This helped us achieve two things - further simplification of the orchestration
algorithm (which is now just about 80 lines long), and visualization to identify potential latecy
bottlenecks.</p>
<p>Read on to find out how to generate and visualize execution plans.</p>
<p>For instance, consider the script</p>
<pre><code>prodid = select ProductID[0].Value from eBay.FindProducts where
QueryKeywords = 'macbook pro';
details = select * from eBay.ProductDetails where
ProductID in ('{prodid}') and ProductType = 'Reference';
reviews = select * from eBay.ProductReviews where
ProductID in ('{prodid}') and ProductType = 'Reference';
return select d.ProductID[0].Value as id, d.Title as title,
d.ReviewCount as reviewCount, r.ReviewDetails.AverageRating as rating
from details as d, reviews as r
where d.ProductID[0].Value = r.ProductID.Value
via route '/myapi' using method get;
</code></pre>
<p>A visualization of the execution plan of this script is below.</p>
<div style="max-width: 100%;overflow:auto">
<a href="/images/2012-06-08-plan-0.svg"><img src="/images/2012-06-08-plan-0.svg" style="max-width: 1000%" alt="A visualization of a script with one fork and one join"></a>
</div>
<p>By looking at this execution plan we can infer the following:</p>
<ul>
<li>The select statemet on line 8 depends on the statements on lines 3 and 5.</li>
<li>The overall latency of this script depends on the slowest of the statements on lines 3 and 5.</li>
</ul>
<p>Here is the execution plan of another script. This script takes two inputs - a user&#8217;s identity and a
set of IDs of some items, and gets some details from two different APIs (the bottom two nodes). The
responses from those APIs trigger some in-process data extractions and transformations which join
on the node below the node at the top.</p>
<div style="max-width: 100%;overflow:auto">
<a href="/images/2012-06-08-plan-1.svg"><img src="/images/2012-06-08-plan-1.svg" style="max-width: 1000%" alt="Another visualization"></a>
</div>
<p>Again, the overall latency depends on the bottom two nodes.</p>
<p>Here is the execution plan of another script which shows one node ([5]) blocking on another ([1]).</p>
<div style="max-width: 100%;overflow:auto">
<a href="/images/2012-06-08-plan-3.svg"><img src="/images/2012-06-08-plan-3.svg" style="max-width: 1000%" alt="Another visualization"></a>
</div>
<h2>Generating Excecution Plan</h2>
<p>Generating the execution plan is easy. Here is a node.js script.</p>
<script src="https://gist.github.com/2898580.js"> </script>
<p>You can use the compiler in the browser too. Here is script that works in any modern browser.</p>
<script src="https://gist.github.com/2898590.js"> </script>
<h2>Visualization</h2>
<p>I wrote a small tool to compile a script and generate a .dot file, and feed the output to
<a href="http://www.graphviz.org/">Graphviz</a>.</p>
<p><a href="http://bl.ocks.org/d/2898080/">Dot file generator</a> - to generate .dot files for ql.io scripts.</p>
</div>
</article>
<article>
<header>
<h1 class="entry-title"><a href="/2012/05/21/ql.io-0.6.html">ql.io 0.6</a></h1>
<p class="meta">
<time datetime="2012-05-21T00:00:00-07:00" pubdate data-updated="true">May 21<span>st</span>, 2012</time>
| <a href="/2012/05/21/ql.io-0.6.html#disqus_thread">Comments</a>
</p>
<p>
<span class="byline author vcard">Posted by <a href="http://github.com/s3u">Subbu Allamaraju</a></span>
</p>
</header>
<div class="entry-content"><p>Today&#8217;s release of ql.io 0.6 includes the following changes:</p>
<ul>
<li>Support array style reference in columns clause, such as <code>select 'b-1', 'b-3'['c-1'] from a</code>.</li>
<li>Disable ability to enable/disable ecv checks by default. You can turn it on by adding arg
<code>--ecvControl true</code> to the start script.</li>
<li>Add optional parameters in route. Including &#8220;with optional params&#8221; in route would make params
without <code>^</code> prefix optional. When this clause is present, only required tokens are used for
matching a request to a route.</li>
<li>Be able to start the server on multiple ports</li>
<li>Added support for multiple attachments. See docs on insert http://ql.io/docs/insert</li>
<li>End pending connections on close after responses are written.</li>
<li>Support cache events (hit, miss, new, error, info, heartbeat)</li>
<li>Switch to new cluster2</li>
<li>Added new syntaxes &#8220;with part&#8221; and opaque insert param.</li>
<li>Fix expression parsing in string template so that a token like <code>"{obj.prop[?(@.price &gt; 2)]}"</code> is
valid</li>
<li>Add support for escaped quotes in string values</li>
<li>Update PEG.js to 0.7.</li>
<li>Remove duplicates from in clause.</li>
<li>Use <code>hasOwnProperty</code> in place of prop lookup while joining</li>
<li>Deal with non UTF-8 encodings from upstream resources</li>
<li>When joining, use &#8216;==&#8217; to maintain backwards compat</li>
<li>Refactor logging to error, access, proxy and default logs. The proxy log file contains outgoing
req/resp, access log contains incoming requests, error log contains all errors and warnings,
and the rest go to ql.io.log. All these files are rotated.</li>
<li>Include a payload with begin events</li>
<li>Support local offset and limit</li>
<li>Fix the case of alias names with joins and UDFs.</li>
<li>Add UDFs in where clause to post process rows. You can either tweak or remove a row. See
https://gist.github.com/2334012 for semantics of UDFs. UDF support for the where clause is coming
in version 0.7.</li>
</ul>
</div>
</article>
<article>
<header>
<h1 class="entry-title"><a href="/2012/04/28/cluster2.html">Cluster2</a></h1>
<p class="meta">
<time datetime="2012-04-28T00:00:00-07:00" pubdate data-updated="true">Apr 28<span>th</span>, 2012</time>
| <a href="/2012/04/28/cluster2.html#disqus_thread">Comments</a>
</p>
<p>
<span class="byline author vcard">Posted by <a href="http://github.com/s3u">Subbu Allamaraju</a></span>
</p>
</header>
<div class="entry-content"><p>cluster2 is a node.js (>= 0.6.x) compatible multi-process management module. This module grew out of
our needs in operationalizing node.js for <a href="https://github.com/ql-io/ql.io">ql.io</a> at eBay. Built on
node&#8217;s <code>cluster</code>, cluster2 adds several safeguards and utility functions to help support real-world
production scenarios:</p>
<ul>
<li>Scriptable start, shutdown and stop flows</li>
<li>Worker monitoring for process deaths</li>
<li>Worker recycling</li>
<li>Graceful shutdown</li>
<li>Idle timeouts</li>
<li>Validation hooks (for other tools to monitor cluster2 apps)</li>
<li>Events for logging cluster activities</li>
</ul>
<p>See <a href="http://ql-io.github.com/cluster2/">cluster2</a> for more info.</p>
</div>
</article>
<article>
<header>
<h1 class="entry-title"><a href="/2012/03/30/ql.io-0.5.html">ql.io 0.5</a></h1>
<p class="meta">
<time datetime="2012-03-30T00:00:00-07:00" pubdate data-updated="true">Mar 30<span>th</span>, 2012</time>
| <a href="/2012/03/30/ql.io-0.5.html#disqus_thread">Comments</a>
</p>
<p>
<span class="byline author vcard">Posted by <a href="http://github.com/s3u">Subbu Allamaraju</a></span>
</p>
</header>
<div class="entry-content"><p>Today&#8217;s release of ql.io 0.5 includes the following changes:</p>
</div>
<footer>
<a rel="full-article" href="/2012/03/30/ql.io-0.5.html">Read on &rarr;</a>
</footer>
</article>
<article>
<header>
<h1 class="entry-title"><a href="/2012/03/21/ql.io-on-cloudfoundry.html">ql.io on Cloud Foundry</a></h1>
<p class="meta">
<time datetime="2012-03-21T00:00:00-07:00" pubdate data-updated="true">Mar 21<span>st</span>, 2012</time>
| <a href="/2012/03/21/ql.io-on-cloudfoundry.html#disqus_thread">Comments</a>
</p>
<p>
<span class="byline author vcard">Posted by <a href="http://github.com/s3u">Subbu Allamaraju</a></span>
</p>
</header>
<div class="entry-content"><p>Here are the steps to deploy ql.io on Cloud Foundry with node 0.6.x. This setup automatically spawns
a cluster of node processes using node&#8217;s native cluster. A sample app to demonstrate these steps is
on <a href="http://ql-io.github.com/2012/03/21/ql.io-on-cloudfoundry.html">github</a> - this includes an
<a href="https://github.com/ql-io/ql.io-cloudfoundry/blob/master/app.js">app.js</a> and a
<a href="https://github.com/ql-io/ql.io-cloudfoundry/blob/master/package.json">package.json</a> with all the
dependencies.</p>
</div>
<footer>
<a rel="full-article" href="/2012/03/21/ql.io-on-cloudfoundry.html">Read on &rarr;</a>
</footer>
</article>
<article>
<header>
<h1 class="entry-title"><a href="/2012/03/12/en-route.html">En Route</a></h1>
<p class="meta">
<time datetime="2012-03-12T00:00:00-07:00" pubdate data-updated="true">Mar 12<span>th</span>, 2012</time>
| <a href="/2012/03/12/en-route.html#disqus_thread">Comments</a>
</p>
<p>
<span class="byline author vcard">Posted by <a href="http://github.com/shimonchayim">Cylus Penkar</a></span>
</p>
</header>
<div class="entry-content"><p>A route in ql.io is a new consumer-optimized HTTP interface. Routes superimpose a simple and familiar HTTP interface on ql.io scripts without needing to specify an elaborate script in the request. In other words, routes make ql.io a platform to &#8220;build your own APIs&#8221;.</p>
<p>Having built this capability, in this post I want to highlight some potential ways to take advantage of routes.</p>
</div>
<footer>
<a rel="full-article" href="/2012/03/12/en-route.html">Read on &rarr;</a>
</footer>
</article>
<article>
<header>
<h1 class="entry-title"><a href="/2012/02/22/making-peace.html">Making Peace With HTTP APIs</a></h1>
<p class="meta">
<time datetime="2012-02-22T00:00:00-08:00" pubdate data-updated="true">Feb 22<span>nd</span>, 2012</time>
| <a href="/2012/02/22/making-peace.html#disqus_thread">Comments</a>
</p>
<p>
<span class="byline author vcard">Posted by <a href="http://github.com/s3u">Subbu Allamaraju</a></span>
</p>
</header>
<div class="entry-content"><p>Once in a while you come across an HTTP API that uses HTTP in complicated and incorrect ways. There
are many examples of this on the Web today including those from
<a href="http://developer.ebay.com/devzone/xml/docs/reference/ebay/GetMyeBayBuying.html">eBay</a>,
<a href="http://docs.amazonwebservices.com/amazonswf/latest/developerguide/UsingJSON-swf.html">Amazon</a>,
<a href="http://code.google.com/apis/friendconnect/docs/opensocial_rest_rpc.html">Google</a>,
<a href="http://www.bing.com/toolbox/bingdeveloper/">Microsoft</a> and many many others. These can be hard to
use as they require you to follow proprietary styles for constructing requests and parsing
responses. Some of those also don&#8217;t work well with common HTTP infrastructure like caches.</p>
<p>In this post, I would like to show how you can, in four simple steps, use ql.io to hide the
complexity of such APIs.</p>
</div>
<footer>
<a rel="full-article" href="/2012/02/22/making-peace.html">Read on &rarr;</a>
</footer>
</article>
<article>
<header>
<h1 class="entry-title"><a href="/2012/02/13/v0.4.html">ql.io 0.4.0</a></h1>
<p class="meta">
<time datetime="2012-02-13T00:00:00-08:00" pubdate data-updated="true">Feb 13<span>th</span>, 2012</time>
| <a href="/2012/02/13/v0.4.html#disqus_thread">Comments</a>
</p>
<p>
<span class="byline author vcard">Posted by <a href="http://github.com/s3u">Subbu Allamaraju</a></span>
</p>
</header>
<div class="entry-content"><p>Verson 0.4 of ql.io is out to npm today. Here is a quick summary of changes.</p>
<ul>
<li>Use native cluster module to start the app</li>
<li>Upgrade all dependencies to the latest</li>
<li>Limit response size to 10000000 bytes from upstream sources. You can change this with
<code>maxResponseLength</code> in the config.</li>
<li>Limit outgoing requests per statement to 50. You can change this with <code>maxRequests</code> in the config.</li>
<li>Chain events for logging done with log-emitter.</li>
<li>Add a new JSON based interface to browse tables and routes. Try <code>/routes</code> to start browsing.</li>
<li>Integrate <a href="https://github.com/s3u/har-view">har-view</a></li>
</ul>
</div>
</article>
<div class="pagination">
<a class="prev" href="/blog/page/2/">&larr; Older</a>
<a href="/blog/archives">Blog Archives</a>
</div>
</div>
<aside class="sidebar">
<section>
<h1>Recent Posts</h1>
<ul id="recent_posts">
<li class="post">
<a href="/2012/07/20/xml-protobuf-performance.html">ql.io XML/Protobuf Performance</a>
</li>
<li class="post">
<a href="/2012/06/29/ql.io-0.7.html">ql.io 0.7</a>
</li>
<li class="post">
<a href="/2012/06/08/plan.html">Evented Orchestration</a>
</li>
<li class="post">
<a href="/2012/05/21/ql.io-0.6.html">ql.io 0.6</a>
</li>
<li class="post">
<a href="/2012/04/28/cluster2.html">cluster2</a>
</li>
<li class="post">
<a href="/2012/03/30/ql.io-0.5.html">ql.io 0.5</a>
</li>
<li class="post">
<a href="/2012/03/21/ql.io-on-cloudfoundry.html">ql.io on Cloud Foundry</a>
</li>
<li class="post">
<a href="/2012/03/12/en-route.html">En Route</a>
</li>
<li class="post">
<a href="/2012/02/22/making-peace.html">Making Peace with HTTP APIs</a>
</li>
<li class="post">
<a href="/2012/02/13/v0.4.html">ql.io 0.4.0</a>
</li>
</ul>
</section>
</aside>
</div>
</div>
<footer role="contentinfo"><p>
Copyright &copy; 2012 - ql.io -
<span class="credit">Powered by <a href="http://octopress.org">Octopress</a></span>
</p>
</footer>
<script type="text/javascript">
var disqus_shortname = 'qlioblog';
var disqus_script = 'count.js';
(function () {
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
dsq.src = 'http://' + disqus_shortname + '.disqus.com/' + disqus_script;
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
}());
</script>
<script type="text/javascript">
(function(){
var twitterWidgets = document.createElement('script');
twitterWidgets.type = 'text/javascript';
twitterWidgets.async = true;
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
})();
</script>
</body>
</html>
Jump to Line
Something went wrong with that request. Please try again.