heavy lifting in clojure
Pull request Compare This branch is 13 commits ahead, 1 commit behind bguthrie:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.



How to start an ec2 cluster using crane.

;;read in creds "creds.clj"
crane.cluser> (def my-creds (creds "/path/to/creds-dir/"))

{:key "AWS-KEY"
 :secretkey "AWS-SECRET-KEY"
 :private-key-path "/path/to/private-key"
 :key-name "key-name"}

;;read in config "aws.clj"
crane.cluster> (def my-conf (conf "/path/to/conf-dir"))

{:image "ami-"
 :instance-type :m1.large
 :group "ec2-group"
 :instances int
 :instances int
 :creds "/path/to/creds.clj-dir"
 :push ["/file/to/push"
 :user-data "/path/to/user-data-file"

;;create new Jec2 class
crane.cluster> (def ec (ec2 (creds "/path/to/creds-dir")))

;;start cluster 
(launch-hadoop-cluster ec my-conf)

To build:

1. Download and install leiningen http://github.com/technomancy/leiningen
2. Setup jclouds maven snapshot at http://jclouds.rimuhosting.com/maven2/snapshots
3. $ lein deps
4. $ lein install
5. $ java -cp `ls lib/*|xargs echo|sed 's/ /:/g'`:classes jline.ConsoleRunner clojure.main

core libs for compute cloud, blobstore, and ssh.

supported compute clouds: ec2, rackspace, rimuhosting, vcloud, terremark, hosting.com
supported blobstores: s3, rackspace, azure, atmos online

build upon the core is remote repl and cluster capabilities.

in a couple cases, we use an erlang convention of the ! syntax representing "reomote" (on another process) execution.
the first argument to ! in erlang is the pid, wheras in our api, it is an open ssh session or socket.
there is no ! operator or fn, we just use the ! suffix convention to communicate remote execution.

remote shell execution.
(sh! session "tar -xzf repl.tar.gz")

remote repl evaluation.
(eval! socket (execute (workflow some-cascading-workflow)))

Cloud Examples:

Here's a quick example of how to view compute and blob resources.

(ns crane.jclouds
  (:use crane.blobstore)
  (:use crane.compute)
  (:use clojure.contrib.pprint)

 (def user  "rackspace_username")
 (def password "rackspace_password")

 (def compute-name "cloudservers")
 (def compute (compute-context compute-name user password))
 (def blobstore-name "cloudfiles")
 (def blobstore (blobstore-context blobstore-name user password ))

 (pprint (images compute))
 (pprint (nodes compute))
 (pprint (sizes compute))

 (pprint (containers blobstore))
 (pprint (blobs blobstore "your_container_name" )) 

the persistent shell session using jsch ChannelShell is shaky at best, although exec works fine.
eval! may be simpler if we could use LineNumberingPushbackReader see comment in remote_repl.clj
stuff to poll and pull info from hadoop tracker url into repl