Skip to content
Browse files

Updates to documentation

  • Loading branch information...
1 parent a56684b commit 0758eb774a0085cd46fbe13eb9544542b013d357 @mlimotte mlimotte committed May 10, 2012
Showing with 38 additions and 30 deletions.
  1. +4 −0 CHANGES.txt
  2. +1 −0 bin/lemur
  3. +4 −0 examples/sample-jobdef.clj
  4. +1 −1 project.clj
  5. +28 −29 resources/help.txt
View
4 CHANGES.txt
@@ -1,5 +1,9 @@
CHANGES
+0.9.8
+
+- documentation updates
+
0.9.7
- Add LICENSE and copyright notice
View
1 bin/lemur
@@ -46,6 +46,7 @@ DEPS_CP=$BASE/lib/*
CLASSPATH=$LEMUR_CP:$DEPS_CP:$LEMUR_EXTRA_CLASSPATH
+# Use this log4j.properties if it exists. Otherwise, use the one in lemur-*.jar
if [ -f "$BASE/resources/log4j.properties" ]; then
LEMUR_LOG_FILE="-Dlog4j.configuration=file:$BASE/resources/log4j.properties"
fi
View
4 examples/sample-jobdef.clj
@@ -236,6 +236,10 @@
; - ./foo.txt uploaded to ${data-uri}/data.txt
; - /tmp/input-dir uploaded to "s3://${bucket}/foo"
+ ; Enable this feature, so that uploads to S3 will display an ascii progress bar during
+ ; the transfer to give you an indication of how long the upload will take.
+ ;:show-progress? true
+
; Additional bootstrap actions can be added by using a key of the form
; :bootstrap-action.N
; Where N is a unique integer which indicates the order in which the scripts should
View
2 project.clj
@@ -1,4 +1,4 @@
-(defproject lemur "0.9.7"
+(defproject lemur "0.9.8-SNAPSHOT"
:description "Lemur is a tool to launch hadoop jobs locally or on EMR
based on a configuration file, referred to as a jobdef."
View
57 resources/help.txt
@@ -30,17 +30,17 @@ JOBDEF OVERVIEW
What happens when you call fire! is dependent on the lemur "command". I.e. on the
command line, you enter something like
- lemur run my-jobdef.clj [args]
- or
- lemur local my-jobdef.clj [args]
+ lemur run my-jobdef.clj [args]
+ or
+ lemur local my-jobdef.clj [args]
The word right after lemur is the "command". The run command will start the EMR cluster
and submit your steps. The local command will run your job on your workstation using
your local Hadoop installation. Try "lemur help" for more commands.
A good starting point for a new job is to copy
- examples/minimal-sample-jobdef.clj
+ examples/minimal-sample-jobdef.clj
Then refer to examples/sample-jobdef.clj for a more exhaustive list
of features and options with more examples and documentation. WARNING: sample-jobdef.clj
@@ -58,18 +58,17 @@ SYNTAX
white-space. E.g. ["foo" "bar", "baz"] is a list of 3 strings. You can also use spaces,
tabs and new-lines between list items.
-BASE
+BASES
- You'll see a (use-base ...) section below. The use-base specifies another file that
+ A jobdef may have a (use-base ...) section. The use-base specifies another file that
should be included here. The idea is that this file would set some default values and
- behavior for you. Most of the important stuff is in lemur.base, so you
- will want to always include that one.
+ behavior for you (i.e. inherit some options and functionality).
OPTIONS
- Many of the configuration blocks below (i.e defcluster, defstep, update-base) accept
+ Many of the configuration blocks (i.e defcluster, defstep, update-base) accept
key/value pairs, where the key is a keyword (i.e. starts with ":"). When fire! is
- executed near the bottom of your job-def, the options from these sources are merged:
+ executed near the bottom of your jobdef, the options from these sources are merged:
base - from one or more files identified in (use-base ...)
cluster - from your defcluster
@@ -79,10 +78,10 @@ OPTIONS
The sources at the bottom take precedence.
- The value for each key is evaluated at run time according to it's type.
+ The value for each key is evaluated at runtime according to it's type.
* String - "foo"
- Can contain variables which will be substituted at run time. The variables
+ Can contain variables which will be substituted at runtime. The variables
are are enclosed in ${}. For example
"${app}-${runtime}"
which would evaluate to something like "sample-2012-01-27-1200"
@@ -242,31 +241,31 @@ TEST
PROFILES
- Profiles are a general override mechanism where a nested map of key/value
- pairs can be specified in your jobdef. If the profile is active, then the
- entries in the nested map override settings of the same name. Profiles can be
- created in defcluster, defstep, or the base.
+ Profiles are a general override mechanism where a nested map of key/value
+ pairs can be specified in your jobdef. If the profile is active, then the
+ entries in the nested map override settings of the same name. Profiles can be
+ created in defcluster, defstep, or the base.
- A profile can also influence which steps are run as part of (fire!). See
- details in sample-jobdef.clj.
+ A profile can also influence which steps are run as part of (fire!). And which
+ validations and hooks are executed. See details in examples/sample-jobdef.clj.
- There is one implicit profile which is activated automatically. When running in
- local mode, the profile :local is on.
+ There is one implicit profile which is activated automatically. When running in
+ local mode, the profile :local is on.
- The leading colon (:) at the front of the profile name, should be considered part
- of the name, and is required.
+ The leading colon (:) at the front of the profile name, should be considered part
+ of the name, and is required.
- Multiple profiles can be active at once. For example, running a test
- in local mode. In this common case, the active profiles are
- [:local, :test]. The right-most profiles takes precedence.
+ Multiple profiles can be active at once. For example, running a test
+ in local mode. In this common case, the active profiles are
+ [:local, :test]. The right-most profiles take precedence.
- Profiles can be enabled on the command line. For example, to enable the :test
- and :foo profiles (in order)
+ Profiles can be enabled on the command line. For example, to enable the :test
+ and :foo profiles (in order)
lemur local :test :foo jobdef.clj ...
- Commonly, you would create profiles for :test values, and maybe a profile for :live
- or :deployed options.
+ Commonly, you would create profiles for :test values, and maybe a profile for :live
+ or :deployed options.
HOOKS

0 comments on commit 0758eb7

Please sign in to comment.
Something went wrong with that request. Please try again.