Skip to content
This repository has been archived by the owner on Nov 10, 2023. It is now read-only.

Fixes #1496 The version of asm that buck is using pre-dates Java 9... #1497

Closed
wants to merge 1 commit into from

Conversation

brettwooldridge
Copy link
Contributor

and chokes on any jar containing Java 9 class files, such as the latest log4j (v2.9.0).

This pull request updates the asm dependency, and ignores "module-info.class" files when calculating the ABI. I'm sure these changes will be revisited when Java 9 officially releases.

@jkeljo
Copy link
Contributor

jkeljo commented Sep 5, 2017

Yeah I would think that module-info.class will be required for correct compilation with javac 9. Buck doesn't yet support javac 9, and as you say all of this will likely change once the official release occurs, but why ignore the files now? Was it causing some kind of issue to leave them in?

@brettwooldridge
Copy link
Contributor Author

@jkeljo Two issues...

First, asm v5.0.3 cannot grok any Java 9 class files, which blows up the ABI calculation code in buck -- in my case when log4j2 v2.9.0 is used.

Second, apparently the module-info.class itself that is contained within the log4j2 maven artifact contains invalid class information even when asm v6-Beta is used. Unclear whether this is an asm issue or log4j2 issue. Excluding module-info.class across the board allows the ABI calculation code to succeed for all of the other Java 9 classes in the log4j2 jar.

@brettwooldridge
Copy link
Contributor Author

@ttsugriy It is absolutely not safe to ignore module-info.class with respect to computing the ABI. 😁 Having said that, because of either a bug in asm v6-Beta, or more likely an invalid module-info.class in the log4j2 artifact, the log4j2 jar is not grok'able by asm (and therefore buck) unless that class is excluded.

Presumably, by the time Java 9 goes GA, either asm or log4j will have gotten their act together, at which point the exclusion can be removed. Also presumably, additional buck changes are likely to be required to support Java 9.

@brettwooldridge
Copy link
Contributor Author

brettwooldridge commented Sep 5, 2017

It should be noted, we couldn't care less about Java 9 support (we use Java 8). We just wanted to use log4j2 v2.9.0, which apparently contains both Java 7/8 as well as Java 9 classes that end up choking asm 5.0.3.

@jkeljo
Copy link
Contributor

jkeljo commented Sep 6, 2017

Glancing at the history for the ASM 6.0 branch it looks to be mostly dealing with modules, so it seems pretty safe to use. If the ABI tests are passing, I'm good with that part.

For log4j, I think you should work around that in your own build. Use a java_binary rule with a blacklist to build a new jar that strips out the module-info.class files rather than put the exclusion in Buck itself. You can then either check that jar in instead of the log4j one, or wrap the java_library in a prebuilt_jar and depend on that rule instead of one for the raw log4j jar.

@ttsugriy
Copy link
Contributor

ttsugriy commented Sep 6, 2017

I second @jkeljo's suggestion to not put module info work-around in buck's codebase and certainly not without a big all caps warning and follow-up task to fix this as soon as it's properly supported :)

@brettwooldridge
Copy link
Contributor Author

@jkeljo @ttsugriy I'm on board with those suggestions. I'll make the changes and update the pull request in the next 24h. Thanks for your feedback.

@facebook-github-bot
Copy link
Contributor

@brettwooldridge has updated the pull request.

@brettwooldridge
Copy link
Contributor Author

@jkeljo @ttsugriy Sorry for the delay. Pull request updated with exclusion of module-info.class removed.

@brettwooldridge
Copy link
Contributor Author

@jkeljo BTW, what is the blacklist in java_library that you speak of, I do not find it in the documentation?

Not to be too critical, but I can now count a handful of times when I've come across undocumented build rule properties mentioned in answers to issues etc. In my opinion, a property should either be supported and documented, or supported and documented as experimental -- or it shouldn't exist.

If it exists in the documentation, even if flagged as experimental, the community has an opportunity to provide feedback on its usefulness or behavior. Otherwise, users end up baking undocumented properties into their BUCK files, laying landmines for future developers maintaining those files should the property disappear or the behavior be altered.

@facebook-github-bot
Copy link
Contributor

@ttsugriy has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@ttsugriy
Copy link
Contributor

Looks like these changes break the build, @brettwooldridge :

BUILD FAILED: //third-party/java/asm:asm#class-abi failed on step calculate_abi_from_classes with an exception:
null
java.lang.RuntimeException
	at org.objectweb.asm.ClassVisitor.visitModule(ClassVisitor.java:148)
	at org.objectweb.asm.ClassReader.readModule(ClassReader.java:760)
	at org.objectweb.asm.ClassReader.accept(ClassReader.java:661)
	at org.objectweb.asm.ClassReader.accept(ClassReader.java:525)
	at com.facebook.buck.jvm.java.abi.DirectoryReader.visitClass(DirectoryReader.java:61)
	at com.facebook.buck.jvm.java.abi.JarReader.visitClass(JarReader.java:53)
	at com.facebook.buck.jvm.java.abi.StubJarClassEntry.of(StubJarClassEntry.java:55)
	at com.facebook.buck.jvm.java.abi.StubJarEntry.of(StubJarEntry.java:30)
	at com.facebook.buck.jvm.java.abi.StubJar.writeTo(StubJar.java:89)
	at com.facebook.buck.jvm.java.abi.StubJar.writeTo(StubJar.java:69)
	at com.facebook.buck.jvm.java.CalculateAbiFromClassesStep.execute(CalculateAbiFromClassesStep.java:47)
	at com.facebook.buck.step.DefaultStepRunner.runStepForBuildTarget(DefaultStepRunner.java:45)
	at com.facebook.buck.rules.CachingBuildRuleBuilder$BuildRuleSteps.executeCommandsNowThatDepsAreBuilt(CachingBuildRuleBuilder.java:1694)
	at com.facebook.buck.rules.CachingBuildRuleBuilder$BuildRuleSteps.run(CachingBuildRuleBuilder.java:1648)
	at com.facebook.buck.util.concurrent.WeightedListeningExecutorService.lambda$2(WeightedListeningExecutorService.java:104)
	at com.facebook.buck.util.concurrent.WeightedListeningExecutorService.lambda$0(WeightedListeningExecutorService.java:78)
	at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:211)
	at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:200)
	at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:130)
	at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:988)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

@brettwooldridge
Copy link
Contributor Author

@ttsugriy This smells like the same error I got when I was not excluding module-info.class, and indicates to me that it may well be a bug in ASM. I will investigate further this week.

@facebook-github-bot
Copy link
Contributor

@brettwooldridge has updated the pull request.

…ava 9, and chokes on any jar containing Java 9 class files, such as the latest log4j (v2.9.0).
@facebook-github-bot
Copy link
Contributor

@brettwooldridge has updated the pull request.

@brettwooldridge
Copy link
Contributor Author

@ttsugriy Awaiting the outcome of the automated builds, but I think I've addressed the issue.

@jkeljo
Copy link
Contributor

jkeljo commented Sep 22, 2017

@brettwooldridge, it's on java_binary, and it is documented. 6th one down on https://buckbuild.com/rule/java_binary.html. There is a similar (also documented) feature on java_library (remove_classes), but it only affects things that would be included in the java_library itself. Since you want to construct a jar that is exactly like an existing jar minus a few files, java_binary is what you need.

As to the documentation thing. I'm not a core Buck team member so don't take the following as authoritative. I agree that we shouldn't be suggesting undocumented features as solutions (at least without filing a task to document them). I think that requiring everything to be supported (with some things marked experimental) would require a change in how Buck is developed, such as using feature branches instead of developing things in master under feature flags or simply undocumented until they're stable, and would probably slow down feature development.

@facebook-github-bot
Copy link
Contributor

@ttsugriy has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@ttsugriy
Copy link
Contributor

Awesome, thank you for working on this, @brettwooldridge! Java 9 was released yesterday, so I was about to start working on this, but I hope that you nailed it :) I'm kicked off internal builds and if they pass, will merge your changes in. Thanks again!

@brettwooldridge
Copy link
Contributor Author

@ttsugriy Cool. My local test run passed without error (er, at least without related errors).

@ttsugriy
Copy link
Contributor

Looks like one test is failing on Linux:

Stub for com/example/buck/A.class is not correct expected:<...tation;() : FIELD, 0[] // invisible

  // ...> but was:<...tation;() : FIELD, 0[;] // invisible

  // ...>
stacktrace: org.junit.ComparisonFailure: Stub for com/example/buck/A.class is not correct expected:<...tation;() : FIELD, 0[] // invisible

  // ...> but was:<...tation;() : FIELD, 0[;] // invisible

  // ...>
	at org.junit.Assert.assertEquals(Assert.java:115)
	at com.facebook.buck.jvm.java.abi.StubJarTest$Tester.checkStubJar(StubJarTest.java:4411)
	at com.facebook.buck.jvm.java.abi.StubJarTest$Tester.createAndCheckStubJar(StubJarTest.java:4399)
	at com.facebook.buck.jvm.java.abi.StubJarTest.preservesTypeAnnotationsInFields(StubJarTest.java:1118)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at com.facebook.buck.testrunner.JUnitRunner.run(JUnitRunner.java:97)
	at com.facebook.buck.testrunner.BaseRunner.runAndExit(BaseRunner.java:272)
	at com.facebook.buck.testrunner.JUnitMain.main(JUnitMain.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.facebook.buck.jvm.java.runner.FileClassPathRunner.main(FileClassPathRunner.java:80)

stdout:

stderr:
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier; support was removed in 8.0

@ttsugriy
Copy link
Contributor

I was able to reproduce it on my mac with:

./bin/buck test //test/com/facebook/buck/jvm/java/abi:abi --filter StubJarTest

@ttsugriy
Copy link
Contributor

Looks like it's a trivial test fix. I'll give it a try and rerun all tests again.

@brettwooldridge
Copy link
Contributor Author

@ttsugriy I reproduced the test failure on my Mac as well. I am assuming it is not transient. However, it may also be the case that the testcase needs to be updated for ASM 6.0/Java 9 behavior. It's 4am here in Tokyo, so I'll take a look over the weekend. If you nail it down or find anything interesting, let me know.

@ttsugriy
Copy link
Contributor

@brettwooldridge, sure, no rush. I fixed the test, but now I'm getting many other errors like:

Failed to calculate ABI for third-party/java/jackson/jackson-core-2.7.8.jar.
java.lang.IllegalArgumentException
java.lang.IllegalArgumentException
	at org.objectweb.asm.ClassVisitor.<init>(ClassVisitor.java:79)
	at org.objectweb.asm.ClassVisitor.<init>(ClassVisitor.java:64)
	at org.objectweb.asm.tree.ClassNode.<init>(ClassNode.java:209)
	at com.facebook.buck.jvm.java.abi.StubJarClassEntry.of(StubJarClassEntry.java:41)
	at com.facebook.buck.jvm.java.abi.StubJarEntry.of(StubJarEntry.java:30)
	at com.facebook.buck.jvm.java.abi.StubJar.writeTo(StubJar.java:89)
	at com.facebook.buck.jvm.java.abi.StubJar.writeTo(StubJar.java:69)
	at com.facebook.buck.jvm.java.CalculateAbiFromClassesStep.execute(CalculateAbiFromClassesStep.java:47)
	at com.facebook.buck.step.DefaultStepRunner.runStepForBuildTarget(DefaultStepRunner.java:45)
	at com.facebook.buck.rules.CachingBuildRuleBuilder$BuildRuleSteps.executeCommandsNowThatDepsAreBuilt(CachingBuildRuleBuilder.java:1708)
	at com.facebook.buck.rules.CachingBuildRuleBuilder$BuildRuleSteps.run(CachingBuildRuleBuilder.java:1661)
	at com.facebook.buck.util.concurrent.WeightedListeningExecutorService.lambda$2(WeightedListeningExecutorService.java:104)
	at com.facebook.buck.util.concurrent.WeightedListeningExecutorService.lambda$0(WeightedListeningExecutorService.java:78)
	at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:211)
	at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:200)
	at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:130)
	at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:988)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Investigating...

@ttsugriy
Copy link
Contributor

hm, looks like this error happens when attempting to build buck with older version of buck - other integration and end to end tests seem to work well. Will try to land this change by the end of day.

@brettwooldridge
Copy link
Contributor Author

brettwooldridge commented Sep 22, 2017

@ttsugriy Yeah, I didn't do that. I always run ant clean ; ant, then bin/buck build buck. (Really going to bed now) 😩 Thanks for chasing that down.

... it does look like an additional semicolon is emitted (by ASM6?) and is causing that particular test to fail a literal text match. I suspect that it is safe to simply update the expected text.

@ttsugriy
Copy link
Contributor

yep, @brettwooldridge, the semicolon looks harmless to me. Looks like the issue I pasted above was just an issue with one of the internal jobs we use for static analysis and these changes (+ test fix) are ready to be committed. I'm working on that... :)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants