Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

lein uberjar does not handle signed dependency jars correctly #31

Closed
ogrisel opened this Issue · 30 comments

9 participants

@ogrisel

If one of the dependencies is signed, creating a standalone jar with "lein uberjar" yields the following exception when executing the main class of the project:

Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes

In my case the inclusion of a "META-INF/DUMMY.SF" file from one of the depedency (janino-2.5.15.jar) is the root of the problem. Patching leiningen as follows in the attahed patch fixes the problem (however this is probably not the right way to fix it).

@ogrisel
diff --git a/src/leiningen/uberjar.clj b/src/leiningen/uberjar.clj
index 67ebcc1..2e9b2cd 100644
--- a/src/leiningen/uberjar.clj
+++ b/src/leiningen/uberjar.clj
@@ -53,7 +53,7 @@ may wish to clean first."
                     (filter #(.endsWith (.getName %) ".jar"))
                     (cons (file (:root project) (str (:name project) ".jar"))))
           [_ components] (reduce (partial include-dep out)
-                                 [#{"META-INF/plexus/components.xml"} nil]
+                                 [#{"META-INF/plexus/components.xml" "META-INF/DUMMY.SF"} nil]
                                  deps)]
       (when-not (empty? components)
         (.putNextEntry out (ZipEntry. "META-INF/plexus/components.xml"))
@michalmarczyk
Collaborator

I'm planning to look into this soonish, though I won't promise to be particularly fast (since I'm not entirely sure what's involved). Should anyone plan on beating me to it, please let me know. :-)

@ogrisel

In the mean time it's good to have following command at hand:
zip *-standalone.jar -d META-INF/DUMMY.SF

@michalmarczyk
Collaborator

Thanks for the tip!

@technomancy
Owner

Is removing META-INF/DUMMY.SF the correct solution for all signed jars, or just for janino?

@ogrisel

No idea. It was the first and only time I encountered the issue and found this file by binary search with trial and error (removing half remaining files at a time).

@technomancy
Owner

According to an older thread on the mailing list sometimes it has another name. If there's a regex that can always catch it then we can make :jar-exclusions default to that regex.

@fdhenard

This issue occurred for me as well with the SQL Server jdbc driver. I used @ogrisel's zip *-standalone.jar -d META-INF/DUMMY.SF and it worked for me.

It would be nice if this issue was re-opened and fixed

@technomancy
Owner

Is it safe to always exclude META-INF/DUMMY.SF from uberjars? I can do this (in a way that's easy to override) if it's appropriate.

@fdhenard

I don't know about that. I actually had to remove zigbert.sf from the standalone.jar, so I believe it's specific to which .jars have the security signature file. So I ran zip *-standalone.jar -d META-INF/zigbert.sf

@technomancy
Owner

It would be nice if this issue was re-opened and fixed

So do you have a solution to propose? Right now it's already easy enough to just add :uberjar-exclusions [#"zigbert.sf"] to project.clj as the need arises. I don't see a general-purpose fix being possible if the name of the file at fault varies.

@fdhenard

I'm very sorry that I missed that in the conversation above. I just tried it and it worked great. Thanks!

@timmc
Collaborator

[gnu.getopt/java-getopt "1.0.13"] has the same problem, but with TESTKEY.SF. I think any SF is going to be a problem.

@technomancy technomancy reopened this
@timmc timmc referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@timmc
Collaborator

Failing testcase:

project.clj

(defproject clj "1.0.0-SNAPSHOT"
  :description "FIXME: write description"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [gnu.getopt/java-getopt "1.0.13"]] ;; jar contains META-INF/TESTKEY.SF
  :main clj.core)

src/clj/core.clj

(ns clj.core
  (:gen-class))

(defn -main [& args]
  (println "hi"))

Explanation

Jars may be signed with multiple signature files, which take the form of #"^META-INF/[a-zA-Z0-9_-]*.SF$" and corresponding .DSA and .RSA key files. The main problem is that the signature file appears to sign the MANIFEST.MF, which Leiningen overwrites:

SHA1-Digest-Manifest-Main-Attributes: j6FYdHNwvIzMP83Z6mTnxQZekCo=
SHA1-Digest-Manifest: WBCxJyIwje6BE6eYCdhrnB0c7Yk=

http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/jarsigner.html

Patch

Forthcoming as pull request.

@timmc timmc referenced this issue from a commit
@timmc timmc #31: Expand jar signature file exclusions to all .sf files in meta-inf.
[Amended commit to re-escape dot in regex.]
d2e540e
@timmc timmc referenced this issue from a commit in timmc/leiningen
@timmc timmc #31: Expand JAR signature file matcher to all .sf files in meta-inf. …
…Also

make case-insensitive and anchor the end.
032e892
@sneilan

I'm having problems with this too.

@timmc
Collaborator

sneilan: I've submitted a patch, but until a fix is released, I'm using this in my project.clj:

:uberjar-exclusions [#"(?i)^META-INF/[^/]*\.SF$"]
@technomancy
Owner

Looks like this is taken care of.

@sneilan

It works!! Thank you!

@sneilan

I found one more issue.

If I include mmemail as a dependency and run the standalone jar:

Exception in thread "main" java.lang.SecurityException: no manifiest section for signature file entry javax/activation/MimeType.class
at sun.security.util.SignatureFileVerifier.verifySection(SignatureFileVerifier.java:392)
at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:249)
at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:193)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:251)
at java.util.jar.JarVerifier.update(JarVerifier.java:205)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:338)
at java.util.jar.JarFile.getInputStream(JarFile.java:403)
at sun.misc.JarIndex.getJarIndex(JarIndex.java:116)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:623)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:614)
at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:598)
at sun.misc.URLClassPath$3.run(URLClassPath.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:337)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:314)
at sun.misc.URLClassPath.getResource(URLClassPath.java:184)
at java.net.URLClassLoader$1.run(URLClassLoader.java:209)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: gamespot.core. Program will exit.

This is because the lines that look like
Name: javax/activation/DataHandler.class
SHA1-Digest: 6X6xyd3/vc4l1Rm9JKwlSR1eFwI=
from /META-INF/MANIFEST.MF in the mmemail jar aren't included in the standalone jar's /META-INF/MANIFEST.MF.

I add those lines in and it works just fine.

This is with leiningen 1.6.2
sneilan@sneilan-ubuntu:~/BucketsOfNantucket/research/gamespot$ lein -v
Leiningen 1.6.2 on Java 1.6.0_23 OpenJDK 64-Bit Server VM

@sneilan

Or, if I get rid of the SHA1-Digest lines from the MANIFEST.MF in the mmemail jar, it also works! (I also got rid of the two SUN....RSA whatever files.)

This makes everything better again :)

@technomancy
Owner

I'm not sure what this means. Is there more that uberjar should be doing?

@ifesdjeen

This issue still exists in Lein2, haven't tried yet with Master.
But whenever I include Jetty, I keep getting these:

Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
        at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:221)
        at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:176)
        at java.util.jar.JarVerifier.processEntry(JarVerifier.java:288)
        at java.util.jar.JarVerifier.update(JarVerifier.java:199)
        at java.util.jar.JarFile.initializeVerifier(JarFile.java:323)
        at java.util.jar.JarFile.getInputStream(JarFile.java:388)
        at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:692)
        at sun.misc.Resource.cachedInputStream(Resource.java:61)
        at sun.misc.Resource.getByteBuffer(Resource.java:144)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:256)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
@michaelklishin
Collaborator

With what lein2 version? Lein2 Preview 4 was released less than 24 hours ago.

@ifesdjeen

Checked on preview4 just now. Issue is reproducible.

@technomancy
Owner

I just made an uberjar containing Jetty with preview4, so if this is still happening I will need some more details to reproduce the problem.

@ifesdjeen
@ifesdjeen

Actually, it was jetty-server.

Here's the repository where the bug reproduces: https://github.com/ifesdjeen/lein-manifest-bug-reproduce/blob/master/project.clj#L4

It may be actually not a Leningen bug, I understand that. If you could give me a hint what that could be, i'll dig & talk to Ring guys, maybe, as they're using it https://github.com/mmcgrana/ring/blob/master/project.clj#L7

On the side note, I tried also the forementioned java-getopt, and issue reproduced:

[gnu.getopt/java-getopt "1.0.13"]

That also causes the issue:

 [org.eclipse.jetty/jetty-server "7.6.1.v20120215"]

Proof I have correct lein :)

lein2 version
Leiningen 2.0.0-preview4 on Java 1.6.0_31 Java HotSpot(TM) 64-Bit Server VM

And the command I run:

lein2 uberjar && java -Xmx1024m -jar target/bugreproduce-1.0.0-SNAPSHOT-standalone.jar

And output:

All namespaces already :aot compiled.
Created /Users/alexp/p/bugreproduce/target/bugreproduce-1.0.0-SNAPSHOT.jar
Including bugreproduce-1.0.0-SNAPSHOT.jar
Including jetty-util-7.6.1.v20120215.jar
Including jetty-io-7.6.1.v20120215.jar
Including jetty-http-7.6.1.v20120215.jar
Including jetty-continuation-7.6.1.v20120215.jar
Including javax.servlet-2.5.0.v201103041518.jar
Including jetty-server-7.6.1.v20120215.jar
Including clojure-1.3.0.jar
Created /Users/alexp/p/bugreproduce/target/bugreproduce-1.0.0-SNAPSHOT-standalone.jar
Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
    at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:221)
    at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:176)
    at java.util.jar.JarVerifier.processEntry(JarVerifier.java:288)
    at java.util.jar.JarVerifier.update(JarVerifier.java:199)
    at java.util.jar.JarFile.initializeVerifier(JarFile.java:323)
    at java.util.jar.JarFile.getInputStream(JarFile.java:388)
    at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:692)
    at sun.misc.Resource.cachedInputStream(Resource.java:61)
    at sun.misc.Resource.getByteBuffer(Resource.java:144)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:256)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Thank you!

@technomancy technomancy reopened this
@technomancy
Owner

Looks like the regex here just got goofed in 2.x.

@ghoseb

This problem occurs with Jetty (via ring) as well, because Jetty recently moved to the Eclipse Foundation and they are now signing the JAR files.

Here is a minimal test-case to reproduce -

;; project.clj
(defproject leintest "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  ;; this fixes the problem
  ;; :uberjar-exclusions [#"(?i)^META-INF/[^/]*\.(SF|RSA)$"]
  :dependencies [[org.clojure/clojure "1.4.0"]
                 ;; bringing in [ring/ring-jetty-adapter "1.1.0"] works as well
                 [org.eclipse.jetty/jetty-server "8.1.4.v20120524"]]
  :main leintest.core)

;; src/leintest/core.clj
(ns leintest.core
  (:gen-class)
  (:import org.eclipse.jetty.server.Server))

(defn -main
  [& args]
  (println "Hello, world!"))

I have issued a pull request that augments the previous regexp to exclude .RSA files as well. We might need to keep changing the regexp in the future as and when we discover more file-types.

@ghost

Here is my solution!
http://middlesphere-1.blogspot.ru/2014/06/how-to-make-jar-with-dependencies-using.html

The main idea is to put all dependecies into ./lib folder

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.