Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Updated to leiningen-1.6.2 #2

Open
wants to merge 1 commit into from

2 participants

@ndimiduk
Owner

Fully re-written to reuse code from leiningen rather than copy-pasting
from leiningen. Depends on private functionality from leiningen 1.6.2
and so will likely break with 2.x.

This should also resolve issue #1.

Nick Dimiduk Updated to leiningen-1.6.2
Fully re-written to reuse code from leiningen rather than copy-pasting
from leiningen. Depends on private functionality from leiningen 1.6.2
and so will likely break with 2.x.

This should also resolve issue 1.
902aea6
@kurtharriger

Failed with same error. I bumped the minor revision number to ensure I was getting the latest and ran lein install and lein deps but same error. All I did in the previous pull request was call remove to filter out null values before copy-to-jar. Adding :resources-path to my project.clj still provides an adequate workaround however.
...
No namespaces to :aot compile listed in project.clj.
Exception in thread "main" java.lang.IllegalArgumentException: No method in multimethod 'copy-to-jar' for dispatch value: null (NO_SOURCE_FILE:0)
at clojure.lang.Compiler.eval(Compiler.java:5440)
at clojure.lang.Compiler.eval(Compiler.java:5391)
at clojure.core$eval.invoke(core.clj:2382)
at clojure.main$eval_opt.invoke(main.clj:235)
at clojure.main$initialize.invoke(main.clj:254)
at clojure.main$script_opt.invoke(main.clj:270)
at clojure.main$main.doInvoke(main.clj:354)
at clojure.lang.RestFn.invoke(RestFn.java:457)
at clojure.lang.Var.invoke(Var.java:377)
at clojure.lang.AFn.applyToHelper(AFn.java:172)
at clojure.lang.Var.applyTo(Var.java:482)
at clojure.main.main(main.java:37)
Caused by: java.lang.IllegalArgumentException: No method in multimethod 'copy-to-jar' for dispatch value: null
at clojure.lang.MultiFn.getFn(MultiFn.java:121)
at clojure.lang.MultiFn.invoke(MultiFn.java:172)
at leiningen.jar$write_jar.invoke(jar.clj:142)
at leiningen.hadoop$jar.invoke(hadoop.clj:33)
at leiningen.hadoop$hadoop.invoke(hadoop.clj:41)
at clojure.lang.Var.invoke(Var.java:365)
at clojure.lang.AFn.applyToHelper(AFn.java:161)
at clojure.lang.Var.applyTo(Var.java:482)
at clojure.core$apply.invoke(core.clj:542)
at leiningen.core$apply_task.invoke(core.clj:259)
at leiningen.core$_main.doInvoke(core.clj:325)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at clojure.lang.AFn.applyToHelper(AFn.java:161)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invoke(core.clj:542)
at leiningen.core$_main.invoke(core.clj:328)
at user$eval42.invoke(NO_SOURCE_FILE:1)
at clojure.lang.Compiler.eval(Compiler.java:5424)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jan 30, 2012
  1. Updated to leiningen-1.6.2

    Nick Dimiduk authored
    Fully re-written to reuse code from leiningen rather than copy-pasting
    from leiningen. Depends on private functionality from leiningen 1.6.2
    and so will likely break with 2.x.
    
    This should also resolve issue 1.
This page is out of date. Refresh to see the latest.
View
5 .gitignore
@@ -1,6 +1,7 @@
pom-generated.xml
pom.xml
Manifest.txt
-lein-hadoop*.jar
+*.jar
lib
-classes
+classes
+.lein-*
View
34 README.md
@@ -1,16 +1,38 @@
# lein-hadoop
-This is a plugin for leiningen which generates a jar suitable to submission
-as a hadoop job. Sepcifically, this is a runable jar with lib/ containing all
-the project dependencies.
+This is a plugin for [Leiningen][0] which generates a jar suitable for
+submission as a [Hadoop][1] job. Specifically, this is a runable jar
+with <tt>lib/</tt> containing all project dependencies. The contents
+of <tt>lib/dev</tt> are excluded.
+
+The name of the generated jar can be overridden by specifying a custom
+value for <tt>:hadoop-name</tt> or <tt>:jar-name</tt> in project.clj.
+
+You can use this plugin to build any Hadoop application which requires
+a runable jar, including projects using [MapReduce][2],
+[Cascading][3], and [Cascalog][4].
## Installation
-Add to your project.clj <tt>:dev-dependencies [lein-hadoop "1.0.0"]]</tt>
+Add to your project.clj
+
+ :dev-dependencies [[lein-hadoop "1.1.0"]]
## Usage
-A new target is added which generates PROJECT-hadoop.jar:
+ $ lein clean, deps, hadoop
+
+## License
+
+Copyright © 2010-2012 Nick Dimiduk
+
+Distributed under the Eclipse Public License, the same as Leiningen
+uses.
- $ lein hadoop
+Thanks to the Leiningen team for a relatively pleasant build tool!
+[0]: https://github.com/technomancy/leiningen "Hair Extinguisher"
+[1]: http://hadoop.apache.org/ "Apache Hadoop"
+[2]: http://hadoop.apache.org/mapreduce/ "Hadoop MapReduce"
+[3]: http://www.cascading.org/ "Cascading"
+[4]: http://www.cascalog.org/ "Cascalog"
View
7 project.clj
@@ -1,5 +1,4 @@
-(defproject lein-hadoop "1.0.0"
+(defproject lein-hadoop "1.1.0"
:description "A leiningen plugin to build jars for hadoop."
- :dev-dependencies [[org.clojure/clojure "1.1.0"]
- [org.clojure/clojure-contrib "1.1.0"]
- [lein-clojars "0.6.0"]])
+ :dependencies [[org.clojure/clojure "1.2.1"]]
+ :eval-in-leiningen true)
View
91 src/leiningen/hadoop.clj
@@ -1,42 +1,59 @@
(ns leiningen.hadoop
"Create a jar for submission as a hadoop job."
- (:require [leiningen.compile :as compile])
- (:use [leiningen.pom :only [make-pom make-pom-properties]]
- [leiningen.jar :only [write-jar]]
- [clojure.contrib.java-utils :only [file]]))
+ (:use
+ [clojure.java.io :only [file]]
+ [leiningen.deps :only [deps]]
+ [leiningen.javac :only [javac]])
+ (:require
+ [leiningen.compile :as compile]
+ [leiningen.jar :as jar]))
-(defn- jar
- "Create a $PROJECT-hadoop.jar file containing the compiled .class files
-as well as the source .clj files. If project.clj contains a :main symbol, it
-will be used as the main-class for an executable jar."
- ([project jar-name]
- (compile/compile project)
- (let [jar-file (str (:root project) "/" jar-name)
- filespecs [{:type :bytes
- :path (format "meta-inf/maven/%s/%s/pom.xml"
- (:group project)
- (:name project))
- :bytes (make-pom project)}
- {:type :bytes
- :path (format "meta-inf/maven/%s/%s/pom.properties"
- (:group project)
- (:name project))
- :bytes (make-pom-properties project)}
- (when (and (:resources-path project)
- (.exists (file (:resources-path project))))
- {:type :path :path (:resources-path project)})
- {:type :path :path (:compile-path project)}
- {:type :path :path (:source-path project)}
- {:type :path :path (:library-path project)}
- {:type :path :path (str (:root project) "/project.clj")}]]
- ;; TODO: support slim, etc
- (write-jar project jar-file filespecs)
- (println "Created" jar-file)
- jar-file))
- ([project] (jar project (str (:name project) ".jar"))))
+;; depends on private API :'(
+(def filespecs #'jar/filespecs)
-(defn hadoop
- "Create a jar for submission to hadoop."
- ([project]
- (jar project (str (:name project) "-hadoop.jar"))))
+(defn add-lib-filespecs
+ "Add :library-path to the list of filespecs. To avoid inclusion of lib/dev,
+use in conjunction with exclude-lib-dev."
+ [filespecs project]
+ (concat
+ (when (and (:library-path project)
+ (.exists (file (:library-path project))))
+ [{:type :path
+ :path (:library-path project)}])
+ filespecs))
+(defn exclude-lib-dev
+ "Add (:library-path project)/dev.*.jar to :jar-exclusions"
+ [project]
+ (let [lib-base (-> (:library-path project)
+ (.replace (str (:root project) "/") ""))
+ dev-exclusions {:jar-exclusions
+ [(re-pattern (str lib-base "/dev/.*$"))]}]
+ (merge-with concat project dev-exclusions)))
+
+(defn hadoopjar-name
+ "Determine a file name for the hadoop jar. Respects name definitions from
+project in order: :hadoop-name, :jar-name"
+ [project]
+ (or (:hadoop-name project)
+ (-> (jar/get-default-jar-name project)
+ (.replaceAll ".jar$" "-hadoop.jar"))))
+
+(defn ^{:help-arglists '([])} hadoop
+ "Create a jar for submission to Hadoop.
+
+Create a $PROJECT-$VERSION-hadoop.jar file. This file is identical to the one
+generated by jar, except it includes all project dependencies in the jar
+under lib/. The contents of lib/dev are excluded. Filename can be overridden
+via either :hadoop-name or :jar-name in project.clj."
+ [project]
+ (let [deps-fileset (deps project) ;; fetch deps
+ _ (javac project) ;; compile java
+ status (compile/compile project)] ;; compile clj
+ (if (zero? status)
+ (let [jar-path (jar/get-jar-filename project (hadoopjar-name project))
+ specs (add-lib-filespecs (filespecs project deps-fileset) project)]
+ (jar/write-jar (exclude-lib-dev project) jar-path specs)
+ (println "Created" jar-path)
+ jar-path)
+ status)))
View
27 test/test_hadoop.clj
@@ -1,27 +0,0 @@
-(ns test-hadoop
- (:use [leiningen.core :only [read-project defproject]]
- [leiningen.clean :only [clean]]
- [leiningen.deps :only [deps]]
- [leiningen.hadoop :only [hadoop]] :reload-all)
- (:use [clojure.test]
- [clojure.contrib.set]
- [clojure.contrib.jar :only [filenames-in-jar]]
- [clojure.contrib.java-utils :only [file]])
- (:import [java.util.jar JarFile]))
-
-(def test-project (read-project "sample/project.clj"))
-
-(deftest test-hadoop
- (clean test-project)
- (deps test-project)
- (hadoop test-project)
- (let [jars (set (map #(.getName %)
- (.listFiles (file (:library-path test-project)))))
- contains (set (remove nil? (map (fn [x]
- (last (re-find #"^lib/(.*jar)$" x)))
- (filenames-in-jar (new JarFile
- (str (:root test-project)
- "/"
- (:name test-project)
- "-hadoop.jar"))))))]
- (is (= contains jars))))
Something went wrong with that request. Please try again.