Skip to content
A leiningen plugin to interact with Amazon Web Services
Clojure
Find file
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
src
test
.gitignore
README.md
project.clj

README.md

lein-aws

This is very use case specific plugin to suit my own needs, but I'd like to expand it to be a more general purpose AWS plugin; pull requests welcomed!

Right it it supposed uploading files to s3 via the s3-put subtask, running and terminating jobflow via the run-flow and terminate-job subtask respectively. It is based on the awesome amazonica library.

Job flows are defined and loaded from a file containing a clojure definition of a keyword identified to an amazonina run-job-flow map.

{:my-flow
  {:name "my-job-flow"
  :log-uri "s3n://emr-logs/logs"
  :instances
    {:instance-groups [
       {:instance-type "m1.large"
        :instance-role "MASTER"
        :instance-count 1
        :market "SPOT"
        :bid-price "0.10"}]}
  :steps [
    {:name "my-step"
     :hadoop-jar-step
       {:jar "s3n://beee0534-ad04-4143-9894-8ddb0e4ebd31/hadoop-jobs/bigml"
        :main-class "bigml.core"
        :args ["s3n://beee0534-ad04-4143-9894-8ddb0e4ebd31/data" "output"]}}]}
}

Since job flows are parsed as clojure, you can factor out common steps:

(let [enable-debugging
      {:name "Enable Debugging"
       :hadoop-jar-step
       {:jar "s3://elasticmapreduce/libs/script-runner/script-runner.jar"
        :args ["s3://elasticmapreduce/libs/state-pusher/0.1/fetch"]}}
      master-m1-xlarge
      {:instance-type "m1.xlarge"
       :instance-role "MASTER"
       :instance-count 1}]
  {:stage-job-flow
   {:name "Run job flow in stage"
    :log-uri "s3n://emr-logs/logs"
    :instances
    {:hadoop-version "1.0.3"
     :instance-groups
     [master-m1-xlarge]}
    :steps
    [enable-debugging
     {:name "run-step-in-staging"
      :hadoop-jar-step
      {:jar "s3n://beee0534-ad04-4143-9894-8ddb0e4ebd31/hadoop-jobs/bigml"
       :main-class "bigml.core"
       :args ["s3n://beee0534-ad04-4143-9894-8ddb0e4ebd31/data" "staging"]}}]}
   :prod-job-flow
   {:name "Run job flow in prod"
    :log-uri "s3n://emr-logs/logs"
    :instances
    {:hadoop-version "1.0.3"
     :instance-groups
     [master-m1-xlarge]}
    :steps
    [enable-debugging
     {:name "my-step"
      :hadoop-jar-step
      {:jar "s3n://beee0534-ad04-4143-9894-8ddb0e4ebd31/hadoop-jobs/bigml"
       :main-class "bigml.core"
       :args ["s3n://beee0534-ad04-4143-9894-8ddb0e4ebd31/data" "prod"]}}]}})

See the amazonica docs for more information.

Usage

You need to let the plugin know about your aws credentils:

:aws {:access-key "ACCESSKEY"
      :secret-key "SECRETKEY"
      :email "aws-login-email"}

Use this for user-level plugins:

Put [lein-aws "0.1.4-SNAPSHOT"] into the :plugins vector of your :user profile, or if you are on Leiningen 1.x do lein plugin install lein-aws 0.1.4-SNAPSHOT.

Use this for project-level plugins:

Put [lein-aws "0.1.4-SNAPSHOT"] into the :plugins vector of your project.clj.

$ lein help aws

License

Copyright © 2013 Soren Macbeth

Distributed under the Eclipse Public License, the same as Clojure.

Something went wrong with that request. Please try again.