Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 90 lines (50 sloc) 3.972 kB
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
1 ---
2 layout: default
3 title: Install Notes
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
4 collapse: false
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
5 ---
6 h1(gemheader). {{ site.gemname }} %(small):: install%
7
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
8 ** "Get the code":#getcode
9 ** "Setup":#setup
10 ** "Installing and Running Wukong with Hadoop":#gethadoop
11 ** "Installing and Running Wukong with Datamapper, ActiveRecord, the command-line and more":#others
12
13
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
14 <notextile><div class="toggle"></notextile>
15
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
16 h2(#getcode). Get the code
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
17
18 Wukong is still under active development. The newest version is available via "Git":http://git-scm.com on "github:":http://github.com/mrflip/{{ site.gemname }}
19
20 pre. $ git clone git://github.com/mrflip/{{ site.gemname }}
21
22 A gem is available from "github:":http://gems.github.com
23
24 pre. $ sudo gem install mrflip-{{ site.gemname }} --source=http://gems.github.com
25
26 or from "gemcutter":http://gemcutter.org
27
28 pre. $ sudo gem install {{ site.gemname }} --source=http://gemcutter.org
29
30 You can instead download this project in either "zip":http://github.com/mrflip/{{ site.gemname }}/zipball/master or "tar":http://github.com/mrflip/{{ site.gemname }}/tarball/master formats.
31
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
32 h3. Get the Dependencies
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
33
34 * Hadoop, pig
35 * extlib, YAML, JSON
36 * Optional gems: trollop, addressable/uri, htmlentities
37
38 <notextile></div><div class="toggle"></notextile>
39
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
40 h2(#setup). Setup
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
41
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
42 1. Allow Wukong to discover where his elephant friend lives by setting a $HADOOP_HOME environment variable: @export HADOOP_HOME="/usr/local/share/hadoop"@
43 2. Add wukong's @bin/@ directory to your $PATH if you'd like to use the "wutils":wutils.html
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
44
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
45 <i>(see also: "Ruby Hadoop Quickstart":http://blog.pdatasolutions.com/post/191978092/ruby-on-hadoop-quickstart)</i>
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
46
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
47 <notextile></div><div class="toggle"></notextile>
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
48
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
49 h2(#gethadoop). Installing and Running Wukong with Hadoop
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
50
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
51 Wukong was primarily developed for Hadoop, and we think it's the best way to use Hadoop (it's certainly the most fun!).
52
53 h3. Run Wukong on the Amazon AWS EC2 Cloud
54
55 h3. Hadoop Infrastructure
56
57 Even if you have a bunch of machines with spare cycles, lots of RAM, and a shared filesystem... do yourself a favor and start out using the "Cloudera AMIs on Amazon's EC2 cloud.":http://www.cloudera.com/hadoop-ec2 There are an overwhelming number of fiddly little parameters and you'll be glad for the user experience before you get into server setup. If it's still mid-late 2009 when you read this, ignore prudence and jump straight to using Hadoop 0.20. It will be a) more fun, b) much more robust (trust me, at "v0.20" you want to live on the bleeding edge), and c) you won't have to suffer through migrating your HDFS two weeks after setup.
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
58
59 To set up hadoop, your best bet are the Cloudera AMIs on Amazon's EC2 compute cloud:
60
61 * http://www.cloudera.com/hadoop-ec2
62 * http://www.cloudera.com/hadoop-ec2-ebs-beta
63
64 EC2 means anyone with a $10 bill can rent a 10-machine cluster with 1TB of distributed storage for 8 hours.
65
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
66 h3. Run Wukong using Amazon AWS Elastic MapReduce
67
68 AWS Elastic MapReduce saves the trouble of even setting up a cluster: click, bam, there it is.
69
70 Phil Ripperger has prepared a "Ruby Hadoop Quickstart":http://blog.pdatasolutions.com/post/191978092/ruby-on-hadoop-quickstart explaining how to get started with Wukong, Hadoop and the Amazon Elastic MapReduce cloud -- it's better than anything we could put here. Thanks Phil!
71
72 h3. Set up a Hadoop cluster
73
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
74 If you have a local cluster, or just want to experiment with a single-machine install, check out the Cloudera packages for both Debian/Ubuntu-based and Redhat/RPM-based Linux systems.
75
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
76 h3. More Hadoop Notes
77
78 I've braindumped some random notes on configuring and using hadoop "over here":hadoop-tips.html
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
79
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
80 <notextile></div><div class="toggle"></notextile>
81
82 h2(#others). Wukong isn't just Hadoop: Datamapper, ActiveRecord, command-line usage and more
83
84 Wukong is used by many in an non-Hadoop environment -- anywhere you can stream data records, you can unleash its monkey power.
2b92784 Fleshed out schema export
Philip (flip) Kromer authored
85
fbeaeb5 Backporting changes from gh-pages docs
Philip (flip) Kromer authored
86 Please see the "usage notes":usage.html#playnice for more!
87
88
89 <notextile></div></notextile>
Something went wrong with that request. Please try again.