Package relocation doesn't relocate references in project source. Results in NoClassDefFound. #55

Closed
kzarzycki opened this Issue Jun 27, 2014 · 9 comments

Comments

Projects
None yet
3 participants
@kzarzycki

Hi,
I'm having issue, with running your shadowJar plugin, more precisely when using "relocate" feature of it.
I'm trying to relocate all google libraries that I use( especially guava classes), to embed them in my jar and be independent of guava that is used in the platform I want to launch my jar in (it's Apache Storm, my jar is a Storm topology).

To achieve this I specify shadowJar task as following:

shadowJar {
     zip64 = true
    exclude 'META-INF/*.RSA', 'META-INF/*.SF', 'META-INF/*.DSA'
    relocate 'com.google', 'mycompany.com.google'
}

I hope I'm right, that that should be it, i.e. all my classes in my jar, that used com.google.* classes, are now "rewritten" to use mycompany.com.google.* classes.
But I have a problem to run the shadow jar:

$ java -cp myjar-1.0-all.jar mycompany.MyMainClass
...
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/inject/PrivateModule
...

When I unpacked the jar, I see that indeed all com.google classes landed in mycompany/com/google directory. There are empty directories in com/google, but I believe this is not the issue.
Do you know what is the issue? I'll be really grateful for your help as it is a major blocker to my project.

Environment:

  • Shadow version : 0.9.0-M5

  • JVM: 1.8.0_05 (Oracle Corporation 25.5-b02)

  • Gradle: 1.12-20140531062749+0000
    to have Shadow working with JVM 1.8, I had to asm to 5.x, I did this by overriding asm dependencies in buildscript:

    classpath 'org.ow2.asm:asm:5.0.3'
    classpath 'org.ow2.asm:asm-commons:5.0.3'

@kzarzycki kzarzycki changed the title from getting NoClassDefFoundError after relocating dependency to getting NoClassDefFoundError after relocating package Jun 27, 2014

@Minecrell

This comment has been minimized.

Show comment
Hide comment
@Minecrell

Minecrell Jun 27, 2014

Contributor

I can confirm this issue with Java 7 as well. The dependency (Guava in this case) is relocated correctly and all references in the dependency will be relocated to their new path.
But the plugin does not relocate the references from the source of the project, they're still pointing to the old package path.
By the way, relocating packages from the source of the project does also not work, so if I have a package notrelocated in my project with a single class in it and want to relocate it to relocated it is still in notrelocated in the JAR.
https://github.com/Minecrell/GradleTest

I have not tested it with Java 8, but maybe you can update ASM to version 5 in the plugin if the older versions do not support Java 8?

Contributor

Minecrell commented Jun 27, 2014

I can confirm this issue with Java 7 as well. The dependency (Guava in this case) is relocated correctly and all references in the dependency will be relocated to their new path.
But the plugin does not relocate the references from the source of the project, they're still pointing to the old package path.
By the way, relocating packages from the source of the project does also not work, so if I have a package notrelocated in my project with a single class in it and want to relocate it to relocated it is still in notrelocated in the JAR.
https://github.com/Minecrell/GradleTest

I have not tested it with Java 8, but maybe you can update ASM to version 5 in the plugin if the older versions do not support Java 8?

@kzarzycki

This comment has been minimized.

Show comment
Hide comment
@kzarzycki

kzarzycki Jun 27, 2014

Exactly, I was just writing my comment about this:) As I believe shadowJar relocates the references inside the dependencies jars. But it does not relocate the references inside the class files of the project it is defined in (as you named it "source of the project").

So now I need to understand how to relocate references in the source of the project. One workaround would be to generate jar out of it first and then make shadowJar out of all jars including the one from sources.
Does anyone suggest a gradle snippet for doing it inside one project?

Thanks

Exactly, I was just writing my comment about this:) As I believe shadowJar relocates the references inside the dependencies jars. But it does not relocate the references inside the class files of the project it is defined in (as you named it "source of the project").

So now I need to understand how to relocate references in the source of the project. One workaround would be to generate jar out of it first and then make shadowJar out of all jars including the one from sources.
Does anyone suggest a gradle snippet for doing it inside one project?

Thanks

@Minecrell

This comment has been minimized.

Show comment
Hide comment
@Minecrell

Minecrell Jun 27, 2014

Contributor

If you need it now I think you could use the older version of the shadow plugin (0.8). It works a little bit different and the new version is easier and faster to use but the relocation is working properly with it (Java 7, you need to test if you can get it working with Java 8).
Here is the README, if you wonder how to use it: https://github.com/johnrengelman/shadow/tree/0.8

Contributor

Minecrell commented Jun 27, 2014

If you need it now I think you could use the older version of the shadow plugin (0.8). It works a little bit different and the new version is easier and faster to use but the relocation is working properly with it (Java 7, you need to test if you can get it working with Java 8).
Here is the README, if you wonder how to use it: https://github.com/johnrengelman/shadow/tree/0.8

@kzarzycki

This comment has been minimized.

Show comment
Hide comment
@kzarzycki

kzarzycki Jun 27, 2014

Thanks Minecrell! OK, I'll try version 0.8 then, we will see whether I can get all the features I need to use from it as well (java 8, zip64, relocate). I will inform about the output

Thanks Minecrell! OK, I'll try version 0.8 then, we will see whether I can get all the features I need to use from it as well (java 8, zip64, relocate). I will inform about the output

@johnrengelman johnrengelman changed the title from getting NoClassDefFoundError after relocating package to Package relocation doesn't relocate references in project source. Results in NoClassDefFound. Jun 27, 2014

@johnrengelman johnrengelman added the bug label Jun 27, 2014

@johnrengelman johnrengelman added this to the 1.0.0 milestone Jun 27, 2014

@johnrengelman

This comment has been minimized.

Show comment
Hide comment
@johnrengelman

johnrengelman Jun 27, 2014

Owner

Thanks for the report. It's definitely a bug, the relocation code is only kicking in when handling the classes in JAR dependencies.

I'll get this fixed for 1.0.0.

Owner

johnrengelman commented Jun 27, 2014

Thanks for the report. It's definitely a bug, the relocation code is only kicking in when handling the classes in JAR dependencies.

I'll get this fixed for 1.0.0.

@johnrengelman

This comment has been minimized.

Show comment
Hide comment
@johnrengelman

johnrengelman Jun 27, 2014

Owner

@kzarzycki v0.8 won't support zip64, that was one of the driving factors for the re-write in 0.9.0.
I've already got a test in place and a fix for the relocation issue. There is one more bug I'm going to try and fix and then I'll be releasing a v1.0.0 (hopefullly this afternoon).

Owner

johnrengelman commented Jun 27, 2014

@kzarzycki v0.8 won't support zip64, that was one of the driving factors for the re-write in 0.9.0.
I've already got a test in place and a fix for the relocation issue. There is one more bug I'm going to try and fix and then I'll be releasing a v1.0.0 (hopefullly this afternoon).

@kzarzycki

This comment has been minimized.

Show comment
Hide comment
@kzarzycki

kzarzycki Jun 27, 2014

Thanks John, Minecrell!

John, that's very good that you got a fix already, I'll wait for the new version of your awesome plugin:)

Unfortunately, it seems there will be another issue that is now related to JDK8, lambdas & asm 5.x. As I believe it's more an issue of asm than shadow, I have just submitted a bug on OW2 asm issue tracker:
http://forge.ow2.org/tracker/?func=detail&aid=317531&group_id=23&atid=100023

But it also might be an issue of how you use asm in shadow, so I wanted to submit an issue for you too.
Could you please look into that too? I've created a sample project for showing this issue here:
https://github.com/kzarzycki/asm-lambda-deserialization-error-test

It should help you jump right to the problem. I'll be gratefull if you could look into that.

Thanks John, Minecrell!

John, that's very good that you got a fix already, I'll wait for the new version of your awesome plugin:)

Unfortunately, it seems there will be another issue that is now related to JDK8, lambdas & asm 5.x. As I believe it's more an issue of asm than shadow, I have just submitted a bug on OW2 asm issue tracker:
http://forge.ow2.org/tracker/?func=detail&aid=317531&group_id=23&atid=100023

But it also might be an issue of how you use asm in shadow, so I wanted to submit an issue for you too.
Could you please look into that too? I've created a sample project for showing this issue here:
https://github.com/kzarzycki/asm-lambda-deserialization-error-test

It should help you jump right to the problem. I'll be gratefull if you could look into that.

@johnrengelman

This comment has been minimized.

Show comment
Hide comment
@johnrengelman

johnrengelman Jun 27, 2014

Owner

I don't personally use the relocation feature and I'm not using Java 8 on any projects yet, so this will likely be far down my list. Contributions are always welcome. Can you please open a new ticket with the above information?

Owner

johnrengelman commented Jun 27, 2014

I don't personally use the relocation feature and I'm not using Java 8 on any projects yet, so this will likely be far down my list. Contributions are always welcome. Can you please open a new ticket with the above information?

@Minecrell

This comment has been minimized.

Show comment
Hide comment
@Minecrell

Minecrell Jun 27, 2014

Contributor

Working for me in version 1.0.0. Thanks!

Contributor

Minecrell commented Jun 27, 2014

Working for me in version 1.0.0. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment