Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

heavy lifting in clojure

branch: master

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 src
Octocat-spinner-32 test
Octocat-spinner-32 .gitignore
Octocat-spinner-32 README
Octocat-spinner-32 project.clj
README
SAMPLE:

How to start an ec2 cluster using crane.

;;read in creds "creds.clj"
crane.cluser> (def my-creds (creds "/path/to/creds-dir/"))

{:key "AWS-KEY"
 :secretkey "AWS-SECRET-KEY"
 :private-key-path "/path/to/private-key"
 :key-name "key-name"}

;;read in config "aws.clj"
crane.cluster> (def my-conf (conf "/path/to/conf-dir"))

{:image "ami-"
 :instance-type :m1.large
 :group "ec2-group"
 :instances int
 :instances int
 :creds "/path/to/creds.clj-dir"
 :push ["/file/to/push"
        "/another/file/to/push"]
 :user-data "/path/to/user-data-file"

;;create new Jec2 class
crane.cluster> (def ec (ec2 (creds "/path/to/creds-dir")))

;;start cluster 
(launch-hadoop-cluster ec my-conf)


To build:

1. Download and install leiningen http://github.com/technomancy/leiningen
2. Setup jclouds maven snapshot at http://jclouds.rimuhosting.com/maven2/snapshots
3. $ lein deps
4. $ lein install
5. $ java -cp `ls lib/*|xargs echo|sed 's/ /:/g'`:classes jline.ConsoleRunner clojure.main

core libs for compute cloud, blobstore, and ssh.

supported compute clouds: ec2, rackspace, rimuhosting, vcloud, terremark, hosting.com
supported blobstores: s3, rackspace, azure, atmos online

build upon the core is remote repl and cluster capabilities.

in a couple cases, we use an erlang convention of the ! syntax representing "reomote" (on another process) execution.
the first argument to ! in erlang is the pid, wheras in our api, it is an open ssh session or socket.
there is no ! operator or fn, we just use the ! suffix convention to communicate remote execution.

remote shell execution.
(sh! session "tar -xzf repl.tar.gz")

remote repl evaluation.
eval! 
(eval! socket (execute (workflow some-cascading-workflow)))

Cloud Examples:

Here's a quick example of how to view compute and blob resources.

(ns crane.jclouds
  (:use crane.blobstore)
  (:use crane.compute)
  (:use clojure.contrib.pprint)
)

 (def user  "rackspace_username")
 (def password "rackspace_password")

 (def compute-name "cloudservers")
 (def compute (compute-context compute-name user password))
 (def blobstore-name "cloudfiles")
 (def blobstore (blobstore-context blobstore-name user password ))

 (pprint (images compute))
 (pprint (nodes compute))
 (pprint (sizes compute))

 (pprint (containers blobstore))
 (pprint (blobs blobstore "your_container_name" )) 


TODO:
the persistent shell session using jsch ChannelShell is shaky at best, although exec works fine.
eval! may be simpler if we could use LineNumberingPushbackReader see comment in remote_repl.clj
stuff to poll and pull info from hadoop tracker url into repl
Something went wrong with that request. Please try again.