Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
branch: master
Fetching contributors…

Cannot retrieve contributors at this time

726 lines (638 sloc) 19.707 kb
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
lang="en" xml:lang="en">
<head>
<title>RHIPE - R and Hadoop Integrated Processing Environment </title>
<meta http-equiv="Content-Type" content="text/html;charset=iso-8859-1"/>
<meta name="generator" content="Org-mode"/>
<meta name="generated" content="2011-04-16 18:45:21 PDT"/>
<meta name="author" content="Saptarshi Guha"/>
<meta name="description" content=""/>
<meta name="keywords" content=""/>
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
html { font-family: Times, serif; font-size: 12pt; }
.title { text-align: center; }
.todo { color: red; }
.done { color: green; }
.tag { background-color: #add8e6; font-weight:normal }
.target { }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right {margin-left:auto; margin-right:0px; text-align:right;}
.left {margin-left:0px; margin-right:auto; text-align:left;}
.center {margin-left:auto; margin-right:auto; text-align:center;}
p.verse { margin-left: 3% }
pre {
border: 1pt solid #AEBDCC;
background-color: #F3F5F7;
padding: 5pt;
font-family: courier, monospace;
font-size: 90%;
overflow:auto;
}
table { border-collapse: collapse; }
td, th { vertical-align: top; }
th.right { text-align:center; }
th.left { text-align:center; }
th.center { text-align:center; }
td.right { text-align:right; }
td.left { text-align:left; }
td.center { text-align:center; }
dt { font-weight: bold; }
div.figure { padding: 0.5em; }
div.figure p { text-align: center; }
textarea { overflow-x: auto; }
.linenr { font-size:smaller }
.code-highlighted {background-color:#ffff00;}
.org-info-js_info-navigation { border-style:none; }
#org-info-js_console-label { font-size:10px; font-weight:bold;
white-space:nowrap; }
.org-info-js_search-highlight {background-color:#ffff00; color:#000000;
font-weight:bold; }
/*]]>*/-->
</style>
<link rel="stylesheet" type="text/css" href="a.css" />
<script type="text/javascript">
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<h1 class="title">RHIPE - R and Hadoop Integrated Processing Environment </h1>
<p>RHIPE(phonetic spelling: hree-pay' <sup><a class="footref" name="fnr.1" href="#fn.1">1</a></sup>) is a java package that integrates the R environment with <a href="http://hadoop.apache.org/core/">Hadoop</a>, the open source implementation
of Google's mapreduce. Using RHIPE it is possible to code map-reduce algorithms in R e.g
</p>
<pre class="src src-R-example">library(Rhipe)
rhinit()
m &lt;- expression({
y &lt;- strsplit(unlist(map.values),<span style="color: #8b2252;">" "</span>)
lapply(y,function(r) rhcollect(r,T))
})
r &lt;- expression(
pre={
count=0
},
reduce={
count &lt;- sum(as.numeric(unlist(reduce.values)),count)
},post={
rhcollect(reduce.key,count)
})
z=rhmr(map=m,reduce=r,comb=T,inout=c(<span style="color: #8b2252;">"text"</span>,<span style="color: #8b2252;">"sequence"</span>),ifolder=<span style="color: #8b2252;">"/tmp/50mil"</span>,ofolder='/tmp/tof')
rhex(z)
</pre>
<div id="table-of-contents">
<h2>Table of Contents</h2>
<div id="text-table-of-contents">
<ul>
<li><a href="#sec-1">1 Mailing List </a></li>
<li><a href="#sec-2">2 Data types supported </a></li>
<li><a href="#sec-3">3 More Information </a></li>
<li><a href="#sec-4">4 Download </a></li>
<li><a href="#sec-5">5 Documentation </a></li>
<li><a href="#sec-6">6 Contact </a></li>
<li><a href="#sec-7">7 News </a></li>
</ul>
</div>
</div>
<div id="outline-container-1" class="outline-2">
<h2 id="sec-1"><span class="section-number-2">1</span> Mailing List </h2>
<div class="outline-text-2" id="text-1">
<p>Mailing list is hosted on Google Groups. The url is
<a href="http://groups.google.com/group/rhipe">http://groups.google.com/group/rhipe</a> . Your first post will be moderated.
</p>
</div>
</div>
<div id="outline-container-2" class="outline-2">
<h2 id="sec-2"><span class="section-number-2">2</span> Data types supported </h2>
<div class="outline-text-2" id="text-2">
<p>Since I use Google's Protocol Buffers, I have not serialized all data types. The
one that can be serialized are: Vectors(character, numeric, integer, complex and
raw, including NA), vectors and attributes. This includes data frames, factors,
time series. This excludes closures, language objects etc, use <code>serialize</code> for this.
</p></div>
</div>
<div id="outline-container-3" class="outline-2">
<h2 id="sec-3"><span class="section-number-2">3</span> More Information </h2>
<div class="outline-text-2" id="text-3">
<p>For more information about what RHIPE is and not, read the FAQ.
Please note, this does not work on Mac OS X Snow Leopard.
</p>
</div>
</div>
<div id="outline-container-4" class="outline-2">
<h2 id="sec-4"><span class="section-number-2">4</span> Download </h2>
<div class="outline-text-2" id="text-4">
<p><b>Source</b>
</p>
<p>
The source code is present on Git, go here <a href="http://github.com/saptarshiguha/RHIPE/">http://github.com/saptarshiguha/RHIPE/</a>
</p>
<p>
To check out the current version, install <code>git</code>
</p>
<pre class="src src-sh">git clone git://github.com/saptarshiguha/RHIPE.git
</pre>
<p>
<b>Versions</b>
</p>
<p>
Read the documentation for installation. Current is the latest version.
</p>
<table border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<caption></caption>
<colgroup><col class="right" /><col class="left" /><col class="left" />
</colgroup>
<thead>
<tr><th scope="col" class="right">Version</th><th scope="col" class="left">Download</th><th scope="col" class="left"></th></tr>
</thead>
<tbody>
<tr><td class="right">0.65.4</td><td class="left"><a href="./dn/Rhipe_0.65.4.tar.gz">Rhipe_0.65.4.tar.gz</a></td><td class="left">Sat Apr 16 18:39:20 PDT 2011</td></tr>
<tr><td class="right">0.65.3</td><td class="left"><a href="./dn/Rhipe_0.65.3.tar.gz">Rhipe_0.65.3.tar.gz</a></td><td class="left">Sat Mar 12 15:13:51 PST 2011</td></tr>
<tr><td class="right">0.65.2</td><td class="left"><a href="./dn/Rhipe_0.65.2.tar.gz">Rhipe_0.65.2.tar.gz</a></td><td class="left">Thu Dec 09 11:29:06 PST 2010</td></tr>
<tr><td class="right">0.65.1</td><td class="left"><a href="./dn/Rhipe_0.65.1.tar.gz">Rhipe<sub>0</sub>.65.1.tar.gz</a></td><td class="left">Tue Dec 07 14:40:25 PST 2010</td></tr>
<tr><td class="right">0.64</td><td class="left"><a href="./dn/Rhipe_0.64.tar.gz">Rhipe_0.64.tar.gz</a></td><td class="left">Thu Oct 21 22:26:20 PDT 2010</td></tr>
<tr><td class="right">0.63</td><td class="left"><a href="./dn/Rhipe_0.63.tar.gz">Rhipe_0.63.tar.gz</a></td><td class="left">Wed Sep 16 03:12:20 EDT 2010</td></tr>
<tr><td class="right">0.61</td><td class="left"><a href="./dn/Rhipe_0.61.tar.gz">Rhipe_0.61.tar.gz</a></td><td class="left">Wed Sep 08 03:12:20 EDT 2010</td></tr>
<tr><td class="right">0.60</td><td class="left"><a href="./dn/Rhipe_0.60.tar.gz">Rhipe_0.60.tar.gz</a></td><td class="left">Fri May 28 17:13:16 EDT 2010</td></tr>
<tr><td class="right">0.59</td><td class="left"><a href="./dn/Rhipe_0.59.tar.gz">Rhipe_0.59.tar.gz</a></td><td class="left">Tue May 04 18:15:26 EDT 2010</td></tr>
<tr><td class="right">0.58</td><td class="left"><a href="./dn/Rhipe_0.58.tar.gz">Rhipe_0.58.tar.gz</a></td><td class="left">Fri Apr 09 01:41:48 EDT 2010</td></tr>
<tr><td class="right">0.57</td><td class="left"><a href="./dn/Rhipe_0.57.tar.gz">Rhipe_0.57.tar.gz</a></td><td class="left">Sat Mar 20 11:20:21 EDT 2010</td></tr>
<tr><td class="right">0.56</td><td class="left"><a href="./dn/Rhipe_0.56.tar.gz">Rhipe_0.56.tar.gz</a></td><td class="left">Sat Feb 27 17:04:44 EST 2010</td></tr>
<tr><td class="right">0.55</td><td class="left"><a href="./dn/Rhipe_0.55.tar.gz">Rhipe_0.55.tar.gz</a></td><td class="left">Sun Feb 21 00:49:04 EST 2010</td></tr>
<tr><td class="right">0.54</td><td class="left"><a href="./dn/Rhipe_0.54.tar.gz">Rhipe_0.54.tar.gz</a></td><td class="left"></td></tr>
<tr><td class="right">0.53</td><td class="left"><a href="./dn/Rhipe_0.53.tar.gz">Rhipe_0.53.tar.gz</a></td><td class="left"></td></tr>
<tr><td class="right">0.52</td><td class="left"><a href="./dn/Rhipe_0.52.tar.gz">Rhipe_0.52.tar.gz</a></td><td class="left"></td></tr>
<tr><td class="right">0.51</td><td class="left"><a href="./dn/Rhipe_0.51.tar.gz">Rhipe_0.51.tar.gz</a></td><td class="left"></td></tr>
<tr><td class="right">0.5</td><td class="left"><a href="./dn/Rhipe_0.5.tar.gz">Rhipe_0.5.tar.gz</a></td><td class="left"></td></tr>
<tr><td class="right">0.44</td><td class="left"><a href="./dn/rhipe.0.44.tgz">rhipe.0.44.tgz</a></td><td class="left"></td></tr>
</tbody>
</table>
</div>
</div>
<div id="outline-container-5" class="outline-2">
<h2 id="sec-5"><span class="section-number-2">5</span> Documentation </h2>
<div class="outline-text-2" id="text-5">
<p>The documentation can be found <a href="./doc/html/index.html">here</a>. PDF version can be found <a href="./doc/RHIPE.pdf">here</a>
</p></div>
</div>
<div id="outline-container-6" class="outline-2">
<h2 id="sec-6"><span class="section-number-2">6</span> Contact </h2>
<div class="outline-text-2" id="text-6">
<p>sguha -AT- purdue -DOT- edu
</p>
</div>
</div>
<div id="outline-container-7" class="outline-2">
<h2 id="sec-7"><span class="section-number-2">7</span> News </h2>
<div class="outline-text-2" id="text-7">
<ul>
<li id="sec-7_1">Sat Apr 16 18:39:39 PDT 2011 <br/>
<ul>
<li>
Quick fix before next major version
</li>
<li>
Added some code so that the user can provide additional JAR files
</li>
<li>
Still Hadoop 0.20 compatible, not yet ready for Hadoop 0.21 (will
work but lots of deprecations)
</li>
<li>
<code>rhput</code> and <code>rhget</code> expand the paths in the source and destination respectively
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_2">Sat Mar 12 15:15:14 PST 2011 <br/>
<ul>
<li>
<code>mapred.input.dir</code> is not passed onto the R code since it be huge.
</li>
<li>
Next version will be Hadoop 0.21 compatible and <b>Hadoop 0.20 incompatible</b>
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_3">Thu Dec 09 11:26:18 PST 2010 <br/>
<ul>
<li>
v 0.65.2
</li>
<li>
Fixed a snprintf bug which caused buffer over runs.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_4">Tue Dec 07 14:39:13 PST 2010 <br/>
<ul>
<li>
Version 0.65.1
</li>
<li>
Added an interactive monitoring feature via <code>rhstatus</code>.
</li>
<li>
Must call <code>rhinit</code> before doing anything.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_5">Thu Oct 21 22:24:18 PDT 2010 <br/>
<ul>
<li>
Version 0.64, minor is 4.
</li>
<li>
Added streaming functions to stream data from a sequence file
(<code>rhstreamsequence</code>).
</li>
<li>
Documentation updated, however example on using <code>biglm</code> is incomplete.
</li>
<li>
The java server is now default.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_6">Wed Sep 08 03:06:34 EDT 2010 <br/>
<ul>
<li>
Version 0.61, minor=3
</li>
<li>
Some modifications made to sorting of keys. Works now.
</li>
<li>
Manual completely re-written
</li>
<li>
The experimental java as a server episode has been re-written.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_7">Wed Aug 04 13:19:09 EDT 2010 <br/>
<ul>
<li>
Version 0.61, minor=2
</li>
<li>
Ordering of numeric and alphabetical keys (not default)
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_8">Wed Jun 30 13:28:21 EDT 2010 <br/>
<ul>
<li>
Version 0.61
</li>
<li>
Added a partitioner that partitions on the i'th element of a scalar vector
(strings, numerics and integers)
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_9">Fri May 28 17:11:16 EDT 2010 <br/>
<ul>
<li>
Now version 0.60
</li>
<li>
Added asynch options to <code>rhex</code>, so jobs can run in the background freeing the
R console. The return value can be used to monitor job progress. See
<a href="http://www.stat.purdue.edu/~sguha/rhipe/doc/html/rhmisc.html">Miscellaneous Commands</a> for more information.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_10">Thu May 06 21:29:36 EDT 2010 <br/>
<ul>
<li>
Added <code>rhcp</code> and <code>rhmv</code> to copy and moves files when both source and destination are on the HDFS (thanks to Jeff Li)
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_11">Tue May 04 18:15:35 EDT 2010 <br/>
<ul>
<li>
Some bugs in the comparator - fixed.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_12">Thu Apr 23 12:48:45 EDT 2010 - <br/>
<ul>
<li>
fixed comparators, <code>rhgetkey</code> working again. (0.59-2)
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_13">Thu Apr 22 12:23:37 EDT 2010 <br/>
<ul>
<li>
Fixed a bug in <code>rhlapply</code>, would not read in data. Thanks to <i>eddyu</i>
</li>
<li>
<code>rhoptions()$version</code> now has displays major, minor , date and notes. I added
this since i make changes to RHIPE but keep the version the same.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_14">Mon Apr 19 02:00:22 EDT 2010 <br/>
<ul>
<li>
Less memory allocation in the key/value(s).
</li>
<li>
<code>rhread</code> now does not do a mapreduce job to convert sequence files to binary. Also has a <code>head</code> like function.
<ul>
<li>
if multicore is installed, then running rhread(..,mc=TRUE) will deserialize in parallel, which might or might be slower &hellip;
</li>
</ul>
</li>
<li>
<code>rhez</code> takes an option <code>mapred</code> which is of the same form as <code>mapred</code> in <code>rhmr</code>. This will override the <code>mapred</code> value in <code>rhmr</code>.
</li>
<li>
<code>rhgetkey</code> takes a parameter <code>skip</code> to read in large databases, also no need for trailing "*".
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_15">Thur Apr 15 <br/>
<ul>
<li>
moved to protobuf-2.3
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_16">Fri Apr 09 01:42:05 EDT 2010 <br/>
<ul>
<li>
<code>rhls</code> can now recurse
</li>
<li>
<code>rhread</code> now need only take a folder (no need for <code>rhmap.sqs</code> to read map files).
Use the <code>type</code> argument to specify sequence(or text) files or map files.
</li>
<li>
Similarly <code>rhmr</code> does not need <code>rhmap.sqs</code>
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_17">Wed Apr 07 16:56:07 EDT 2010 <br/>
<ul>
<li>
<code>rhread</code> takes a <code>max</code> argument that reads in only max number of key-value pairs
</li>
<li>
<code>rhex</code> passes all extra arguments to the system command.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_18">Sat Mar 20 20:51:58 EDT 2010 <br/>
<ul>
<li>
Combiner bug fixed, it's still needs to be tested. Mail if numbers do not match.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_19">Sat Mar 20 11:20:44 EDT 2010 <br/>
<ul>
<li>
Fixed combiner, still alpha, but it halves the wordcount speed. The combiner
logic is run in the R interpreter C code. However it is still alpha, so if you
get erroneous results kindly report them back.
</li>
<li>
Also fixed a buffer overflow in <code>main.c</code>. Thanks to Will Nolan.
</li>
<li>
Values and Keys can be now be upto 256MB.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_20">Fri Feb 19 20:43:25 EST 2010 <br/>
<ul>
<li>
EC2 now works!
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_21">Thu Jan 14 20:19:24 EST 2010 <br/>
<ul>
<li>
Counters are returned to the R session (for <code>rhmr</code> only). That is the return
value of <code>rhmr</code> is a list, the first element indicates success/failure and the
second are all the counters visible in the job UI.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_22">Wed Jan 13 02:52:27 EST 2010 <br/>
<ul>
<li>
Fixed a bug where errors in R code were not appearing. Somewhat fixed. Version
stays the same.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_23">Thu Dec 24 11:58:04 EST 2009 <br/>
Released version 0.54
<ul>
<li>
Introduce a Hadoop Map File Outputformat and functions for reading a key from
map files(see help on rhmr and misc functions)
</li>
<li>
Fixed a bug for the case when no reducer is specified but RHIPE java code
threw an exception.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_24">Sun Dec 13 22:11:53 EST 2009 <br/>
<ul>
<li>
Release **Version 0.53**
</li>
<li>
Bug fixes:
<ul>
<li>
Inserted R_CStackLimits, since I'm using Protobuf a threaded library, it was
upsetting R.
</li>
<li>
Removed Rf_duplicate
</li>
</ul>
</li>
<li>
Data types have been enhanced, now allows scalar vectors with attributes. Experimental.
</li>
<li>
A result of which can now write data.frames and read them back in.
</li>
<li>
Impose 64MB key,value serialization limit(workaround to come in
future). Objects bigger than this will be written successfully,but will fail
to read and will cause the job to fail.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_25">Thu Dec 10 13:28:19 EST 2009 <br/>
<ul>
<li>
<code>rhcounter</code> ,available in mapreduce code, is more versatile. Previously, ','
in the counter names would upset Hadoop. Not anymore, see documentation for <code>rhmr</code>
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_26">Wed Dec 2 12:44:23 EST 2009 <br/>
<ul>
<li>
Failed when running RHIPE from different UID's. Now writes to
<code>/tmp/logger-UID</code>. Version number is still the same
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_27">Mon Oct 12 11:18:31 EDT 2009 <br/>
<ul>
<li>
Removed the dependency on rJava. Getting it to work with Hadoop classpaths
caused to much grief. The actualy RHIPE program remains unchanged but the
client handler (R package) is a bit slower(?)
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_28">Sun Sep 27 22:01:33 EDT 2009 <br/>
<ul>
<li>
Names are <b>only</b> read for VECSXP (list objects), because of a strange bug.
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_29">Tue Sep 8 15:35:24 EDT 2009 <br/>
<ul>
<li>
Moved to Hadoop 0.20
</li>
<li>
Uses protobuf for serialization, fewer R types allowed
</li>
<li>
Does not depend on Rserve, single R package to install
</li>
</ul>
</li>
</ul>
<ul>
<li id="sec-7_30">Fri Aug 7 2009, Version 0.45 <br/>
<ul>
<li>
Web site revamped. Beginning with the current version, the entire
manual is in PDF or can be accessed at the <a href="./doc/index.html/">documentation</a> link.
</li>
<li>
Source code is available on Git, go to the download page for instructions.
</li>
<li>
Stopped seeding via secure random generator, so the user will have
to seed it to avoid correlated streams. On RHEL linux
when running <code>rhlapply</code> on 145K+ tasks, <code>/dev/random</code> would block.
</li>
</ul>
</li>
</ul>
</div>
</div>
<div id="footnotes">
<h2 class="footnotes">Footnotes: </h2>
<div id="text-footnotes">
<p class="footnote"><sup><a class="footnum" name="fn.1" href="#fnr.1">1</a></sup> This is greek for <i>a moment in time</i>. See here for pronunciation:
<a href="http://www.searchgodsword.org/lex/grk/view.cgi?number=4493">Greek Lexicon</a>
</p>
</div>
</div>
<div id="postamble">
<p class="author"> Author: Saptarshi Guha
</p>
<p class="date"> Date: 2011-04-16 18:45:21 PDT</p>
<p class="creator">HTML generated by org-mode 7.3 in emacs 23</p>
</div>
</div>
</body>
</html>
Jump to Line
Something went wrong with that request. Please try again.