Skip to content
This repository

lein uberjar does not handle signed dependency jars correctly #31

Closed
ogrisel opened this Issue March 26, 2010 · 29 comments

9 participants

Olivier Grisel Phil Hagelberg Michał Marczyk fdhenard Tim McCormack Sean Neilan αλεx π Michael Klishin Baishampayan Ghose
Olivier Grisel

If one of the dependencies is signed, creating a standalone jar with "lein uberjar" yields the following exception when executing the main class of the project:

Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes

In my case the inclusion of a "META-INF/DUMMY.SF" file from one of the depedency (janino-2.5.15.jar) is the root of the problem. Patching leiningen as follows in the attahed patch fixes the problem (however this is probably not the right way to fix it).

Olivier Grisel
diff --git a/src/leiningen/uberjar.clj b/src/leiningen/uberjar.clj
index 67ebcc1..2e9b2cd 100644
--- a/src/leiningen/uberjar.clj
+++ b/src/leiningen/uberjar.clj
@@ -53,7 +53,7 @@ may wish to clean first."
                     (filter #(.endsWith (.getName %) ".jar"))
                     (cons (file (:root project) (str (:name project) ".jar"))))
           [_ components] (reduce (partial include-dep out)
-                                 [#{"META-INF/plexus/components.xml"} nil]
+                                 [#{"META-INF/plexus/components.xml" "META-INF/DUMMY.SF"} nil]
                                  deps)]
       (when-not (empty? components)
         (.putNextEntry out (ZipEntry. "META-INF/plexus/components.xml"))
Michał Marczyk
Collaborator

I'm planning to look into this soonish, though I won't promise to be particularly fast (since I'm not entirely sure what's involved). Should anyone plan on beating me to it, please let me know. :-)

Olivier Grisel

In the mean time it's good to have following command at hand:
zip *-standalone.jar -d META-INF/DUMMY.SF

Michał Marczyk
Collaborator

Thanks for the tip!

Phil Hagelberg
Owner

Is removing META-INF/DUMMY.SF the correct solution for all signed jars, or just for janino?

Olivier Grisel

No idea. It was the first and only time I encountered the issue and found this file by binary search with trial and error (removing half remaining files at a time).

Phil Hagelberg
Owner

According to an older thread on the mailing list sometimes it has another name. If there's a regex that can always catch it then we can make :jar-exclusions default to that regex.

fdhenard

This issue occurred for me as well with the SQL Server jdbc driver. I used @ogrisel's zip *-standalone.jar -d META-INF/DUMMY.SF and it worked for me.

It would be nice if this issue was re-opened and fixed

Phil Hagelberg
Owner

Is it safe to always exclude META-INF/DUMMY.SF from uberjars? I can do this (in a way that's easy to override) if it's appropriate.

fdhenard

I don't know about that. I actually had to remove zigbert.sf from the standalone.jar, so I believe it's specific to which .jars have the security signature file. So I ran zip *-standalone.jar -d META-INF/zigbert.sf

Phil Hagelberg
Owner

It would be nice if this issue was re-opened and fixed

So do you have a solution to propose? Right now it's already easy enough to just add :uberjar-exclusions [#"zigbert.sf"] to project.clj as the need arises. I don't see a general-purpose fix being possible if the name of the file at fault varies.

fdhenard

I'm very sorry that I missed that in the conversation above. I just tried it and it worked great. Thanks!

Tim McCormack

[gnu.getopt/java-getopt "1.0.13"] has the same problem, but with TESTKEY.SF. I think any SF is going to be a problem.

Phil Hagelberg technomancy reopened this January 08, 2012
Tim McCormack timmc referenced this issue from a commit January 08, 2012
Commit has since been removed from the repository and is no longer available.
Tim McCormack

Failing testcase:

project.clj

(defproject clj "1.0.0-SNAPSHOT"
  :description "FIXME: write description"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [gnu.getopt/java-getopt "1.0.13"]] ;; jar contains META-INF/TESTKEY.SF
  :main clj.core)

src/clj/core.clj

(ns clj.core
  (:gen-class))

(defn -main [& args]
  (println "hi"))

Explanation

Jars may be signed with multiple signature files, which take the form of #"^META-INF/[a-zA-Z0-9_-]*.SF$" and corresponding .DSA and .RSA key files. The main problem is that the signature file appears to sign the MANIFEST.MF, which Leiningen overwrites:

SHA1-Digest-Manifest-Main-Attributes: j6FYdHNwvIzMP83Z6mTnxQZekCo=
SHA1-Digest-Manifest: WBCxJyIwje6BE6eYCdhrnB0c7Yk=

http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/jarsigner.html

Patch

Forthcoming as pull request.

Tim McCormack timmc referenced this issue from a commit January 08, 2012
Tim McCormack #31: Expand jar signature file exclusions to all .sf files in meta-inf.
[Amended commit to re-escape dot in regex.]
d2e540e
Tim McCormack timmc referenced this issue from a commit in timmc/leiningen January 08, 2012
Tim McCormack #31: Expand JAR signature file matcher to all .sf files in meta-inf. …
…Also

make case-insensitive and anchor the end.
032e892
Sean Neilan

I'm having problems with this too.

Tim McCormack

sneilan: I've submitted a patch, but until a fix is released, I'm using this in my project.clj:

:uberjar-exclusions [#"(?i)^META-INF/[^/]*\.SF$"]
Phil Hagelberg technomancy closed this January 25, 2012
Phil Hagelberg
Owner

Looks like this is taken care of.

Sean Neilan

It works!! Thank you!

Sean Neilan

I found one more issue.

If I include mmemail as a dependency and run the standalone jar:

Exception in thread "main" java.lang.SecurityException: no manifiest section for signature file entry javax/activation/MimeType.class
at sun.security.util.SignatureFileVerifier.verifySection(SignatureFileVerifier.java:392)
at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:249)
at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:193)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:251)
at java.util.jar.JarVerifier.update(JarVerifier.java:205)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:338)
at java.util.jar.JarFile.getInputStream(JarFile.java:403)
at sun.misc.JarIndex.getJarIndex(JarIndex.java:116)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:623)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:614)
at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:598)
at sun.misc.URLClassPath$3.run(URLClassPath.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:337)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:314)
at sun.misc.URLClassPath.getResource(URLClassPath.java:184)
at java.net.URLClassLoader$1.run(URLClassLoader.java:209)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: gamespot.core. Program will exit.

This is because the lines that look like
Name: javax/activation/DataHandler.class
SHA1-Digest: 6X6xyd3/vc4l1Rm9JKwlSR1eFwI=
from /META-INF/MANIFEST.MF in the mmemail jar aren't included in the standalone jar's /META-INF/MANIFEST.MF.

I add those lines in and it works just fine.

This is with leiningen 1.6.2
sneilan@sneilan-ubuntu:~/BucketsOfNantucket/research/gamespot$ lein -v
Leiningen 1.6.2 on Java 1.6.0_23 OpenJDK 64-Bit Server VM

Sean Neilan

Or, if I get rid of the SHA1-Digest lines from the MANIFEST.MF in the mmemail jar, it also works! (I also got rid of the two SUN....RSA whatever files.)

This makes everything better again :)

Phil Hagelberg
Owner

I'm not sure what this means. Is there more that uberjar should be doing?

αλεx π

This issue still exists in Lein2, haven't tried yet with Master.
But whenever I include Jetty, I keep getting these:

Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
        at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:221)
        at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:176)
        at java.util.jar.JarVerifier.processEntry(JarVerifier.java:288)
        at java.util.jar.JarVerifier.update(JarVerifier.java:199)
        at java.util.jar.JarFile.initializeVerifier(JarFile.java:323)
        at java.util.jar.JarFile.getInputStream(JarFile.java:388)
        at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:692)
        at sun.misc.Resource.cachedInputStream(Resource.java:61)
        at sun.misc.Resource.getByteBuffer(Resource.java:144)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:256)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Michael Klishin
Collaborator

With what lein2 version? Lein2 Preview 4 was released less than 24 hours ago.

αλεx π

Checked on preview4 just now. Issue is reproducible.

Phil Hagelberg
Owner

I just made an uberjar containing Jetty with preview4, so if this is still happening I will need some more details to reproduce the problem.

αλεx π
αλεx π

Actually, it was jetty-server.

Here's the repository where the bug reproduces: https://github.com/ifesdjeen/lein-manifest-bug-reproduce/blob/master/project.clj#L4

It may be actually not a Leningen bug, I understand that. If you could give me a hint what that could be, i'll dig & talk to Ring guys, maybe, as they're using it https://github.com/mmcgrana/ring/blob/master/project.clj#L7

On the side note, I tried also the forementioned java-getopt, and issue reproduced:

[gnu.getopt/java-getopt "1.0.13"]

That also causes the issue:

 [org.eclipse.jetty/jetty-server "7.6.1.v20120215"]

Proof I have correct lein :)

lein2 version
Leiningen 2.0.0-preview4 on Java 1.6.0_31 Java HotSpot(TM) 64-Bit Server VM

And the command I run:

lein2 uberjar && java -Xmx1024m -jar target/bugreproduce-1.0.0-SNAPSHOT-standalone.jar

And output:

All namespaces already :aot compiled.
Created /Users/alexp/p/bugreproduce/target/bugreproduce-1.0.0-SNAPSHOT.jar
Including bugreproduce-1.0.0-SNAPSHOT.jar
Including jetty-util-7.6.1.v20120215.jar
Including jetty-io-7.6.1.v20120215.jar
Including jetty-http-7.6.1.v20120215.jar
Including jetty-continuation-7.6.1.v20120215.jar
Including javax.servlet-2.5.0.v201103041518.jar
Including jetty-server-7.6.1.v20120215.jar
Including clojure-1.3.0.jar
Created /Users/alexp/p/bugreproduce/target/bugreproduce-1.0.0-SNAPSHOT-standalone.jar
Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
    at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:221)
    at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:176)
    at java.util.jar.JarVerifier.processEntry(JarVerifier.java:288)
    at java.util.jar.JarVerifier.update(JarVerifier.java:199)
    at java.util.jar.JarFile.initializeVerifier(JarFile.java:323)
    at java.util.jar.JarFile.getInputStream(JarFile.java:388)
    at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:692)
    at sun.misc.Resource.cachedInputStream(Resource.java:61)
    at sun.misc.Resource.getByteBuffer(Resource.java:144)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:256)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Thank you!

Phil Hagelberg technomancy reopened this May 18, 2012
Phil Hagelberg
Owner

Looks like the regex here just got goofed in 2.x.

Phil Hagelberg technomancy closed this in 75c2508 May 21, 2012
Baishampayan Ghose
ghoseb commented May 30, 2012

This problem occurs with Jetty (via ring) as well, because Jetty recently moved to the Eclipse Foundation and they are now signing the JAR files.

Here is a minimal test-case to reproduce -

;; project.clj
(defproject leintest "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  ;; this fixes the problem
  ;; :uberjar-exclusions [#"(?i)^META-INF/[^/]*\.(SF|RSA)$"]
  :dependencies [[org.clojure/clojure "1.4.0"]
                 ;; bringing in [ring/ring-jetty-adapter "1.1.0"] works as well
                 [org.eclipse.jetty/jetty-server "8.1.4.v20120524"]]
  :main leintest.core)

;; src/leintest/core.clj
(ns leintest.core
  (:gen-class)
  (:import org.eclipse.jetty.server.Server))

(defn -main
  [& args]
  (println "Hello, world!"))

I have issued a pull request that augments the previous regexp to exclude .RSA files as well. We might need to keep changing the regexp in the future as and when we discover more file-types.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.