Java 8 support of ASM (INVOKESPECIAL/STATIC on interfaces)

[johanneslerch]

According to Eric the new ASM frontend should support Java 8 Bytecode. Is that correct?
If i apply my analysis on the Java Class Library version 8 i get the stack trace below.

Soot arguments: -w -keep-line-number -allow-phantom-refs -p cg all-reachable:true -f none -include-all -cp /Users/joe/JavaRuntimes/jre1.8.0/lib/jce.jar:/Users/joe/JavaRuntimes/jre1.8.0/lib/jsse.jar -process-dir /Users/joe/JavaRuntimes/jre1.8.0/lib/rt.jar

Exception in thread "main" java.lang.IllegalArgumentException: INVOKESPECIAL/STATIC on interfaces require ASM 5
at org.objectweb.asm.MethodVisitor.visitMethodInsn(MethodVisitor.java:477)
at org.objectweb.asm.tree.MethodNode.visitMethodInsn(MethodNode.java:461)
at org.objectweb.asm.ClassReader.readCode(ClassReader.java:1416)
at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1017)
at org.objectweb.asm.ClassReader.accept(ClassReader.java:693)
at org.objectweb.asm.ClassReader.accept(ClassReader.java:506)
at soot.asm.AsmClassSource.resolve(AsmClassSource.java:70)
at soot.SootResolver.bringToHierarchy(SootResolver.java:230)
at soot.SootResolver.bringToSignatures(SootResolver.java:255)
at soot.SootResolver.bringToBodies(SootResolver.java:291)
at soot.SootResolver.processResolveWorklist(SootResolver.java:164)
at soot.SootResolver.resolveClass(SootResolver.java:129)
at soot.Scene.tryLoadClass(Scene.java:639)
at soot.Scene.loadBasicClasses(Scene.java:1210)
at soot.Scene.loadNecessaryClasses(Scene.java:1289)
at soot.Main.run(Main.java:174)
at soot.Main.main(Main.java:147)
at flow.twist.AbstractAnalysis.execute(AbstractAnalysis.java:27)
at flow.twist.mains.GenericCallerSensitiveBiDi.main(GenericCallerSensitiveBiDi.java:28)

The message says ASM 5 should be used. In my local copy of the soot project a asm-debug-all-5.0.3.jar is referenced, so i would assume ASM 5 is already present. Do i have to declare some additional options to enable its use?

[StevenArzt]

We do not have Java 8 support at the moment. ASM should support it, but the Soot classes that translate ASM's output to Jimple statements have not been tested with Java 8 yet. Feel free to try it out and contribute, though :-)

[ericbodden]

Hi Steven. The main purpose of this bug report is to get Java 8 support! So please don't just dismiss it.

The error message seems odd. Let me try to reproduce this...

[ericbodden]

I have now created a new branch features/java8support for this.
https://github.com/Sable/soot/tree/feature/java8support

Converting the visitors to ASM5 was simple, but then the real problem is showing: there seems to be no conversion code yet for invokedynamic, which is instruction type 6:
http://asm.ow2.org/asm50/javadoc/user/constant-values.html#org.objectweb.asm.tree.AbstractInsnNode.INVOKE_DYNAMIC_INSN

Could Samuel still look into this? BTW Does he not have a GitHub account? I can't find him...

He can look up here to see how the instructions should be created:
https://github.com/Sable/soot/blob/develop/src/soot/coffi/CFG.java#L4340

I hope that nothing changed int he format from Java 7 to 8.

[StevenArzt]

Samuel will leave at the end of the month, so I prefer to have him look at the few bug reports we got so far for ASM on Java 7. That's not much work, but he hasn't much time left either.

Would the conversion to ASM be something we can directly merge into the develop branch regardless of Java 8? That might spare us some merging effort later on.

Eric Bodden notifications@github.com wrote:

I have now created a new branch features/java8support for this.
https://github.com/Sable/soot/tree/feature/java8support

Converting the visitors to ASM5 was simple, but then the real problem is showing: there seems to be no conversion code yet for invokedynamic, which is instruction type 6:
http://asm.ow2.org/asm50/javadoc/user/constant-values.html#org.objectweb.asm.tree.AbstractInsnNode.INVOKE_DYNAMIC_INSN

Could Samuel still look into this? BTW Does he not have a GitHub account? I can't find him...

—
Reply to this email directly or view it on GitHub.

[ericbodden]

Hi Steven. I am not sure what you mean by "Would the conversion to ASM be something we can directly merge into the develop branch regardless of Java 8?". The ASM frontend is already part of "develop"...

BTW the bug mentioned above is actually really a Java 7 bug. Also Java 7 bytecode can already contain invokedynamic calls. It's just that Java 7 source was not able to produce such bytecode. With Java 8 that has changed. The method that Soot now chokes on is this one:
http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/nio/file/Files.java#Files.asUncheckedRunnable%28java.io.Closeable%29

Apparently, lambdas can sometimes (?) use invokedynamic.
Actually we should have a student project on resolving invokedynamic calls in such simple situations.

[ericbodden]

This website gives more information:
http://cr.openjdk.java.net/~briangoetz/lambda/lambda-translation.html

[StevenArzt]

I meant the ASM 5 visitors to which you ported the frontend. Sorry for mistyping it and just writing ASM.

Eric Bodden notifications@github.com wrote:

This website gives more information:
http://cr.openjdk.java.net/~briangoetz/lambda/lambda-translation.html

—
Reply to this email directly or view it on GitHub.

[ericbodden]

Ah I now see what you mean. I just cherry-picked it into develop.

[ericbodden]

Over the weekend I had an email exchange with Brian Götz. He confirmed to me that it should be entirely possible to get rid of those invokedynamic-calls introduced by lambda captures. In fact, J2ME does exactly that, replacing them by invokevirtual/static calls. I have opened a new feature request for this:
#226

We should nevertheless extend the ASM frontend such that it does not choke on indy calls in the first place.

[ericbodden]

I added some support to AsmMethodSource but it's still giving me some exception unfortunately:

Exception in thread "main" java.lang.RuntimeException: Failed to convert <java.util.Spliterator$OfInt: void forEachRemaining(java.util.function.Consumer)>
at soot.asm.AsmMethodSource.getBody(AsmMethodSource.java:1739)
at soot.SootMethod.getBodyFromMethodSource(SootMethod.java:91)
at soot.SootMethod.retrieveActiveBody(SootMethod.java:324)
at soot.jimple.toolkits.callgraph.OnFlyCallGraphBuilder.processNewMethod(OnFlyCallGraphBuilder.java:529)
at soot.jimple.toolkits.callgraph.OnFlyCallGraphBuilder.processReachables(OnFlyCallGraphBuilder.java:420)
at soot.jimple.toolkits.callgraph.CallGraphBuilder.build(CallGraphBuilder.java:84)
at soot.jimple.toolkits.callgraph.CHATransformer.internalTransform(CHATransformer.java:43)
at soot.SceneTransformer.transform(SceneTransformer.java:39)
at soot.Transform.apply(Transform.java:90)
at soot.RadioScenePack.internalApply(RadioScenePack.java:57)
at soot.jimple.toolkits.callgraph.CallGraphPack.internalApply(CallGraphPack.java:49)
at soot.Pack.apply(Pack.java:116)
at soot.PackManager.runWholeProgramPacks(PackManager.java:553)
at soot.PackManager.runPacksNormally(PackManager.java:453)
at soot.PackManager.runPacks(PackManager.java:391)
at soot.Main.run(Main.java:203)
at soot.Main.main(Main.java:146)
Caused by: java.lang.AssertionError: 186 already has a unit, $stack5 = dynamicinvoke "accept" <java.util.function.IntConsumer (java.util.function.Consumer)>(l1) <java.lang.invoke.LambdaMetafactory: java.lang.invoke.CallSite metafactory(java.lang.invoke.MethodHandles$Lookup,java.lang.String,java.lang.invoke.MethodType,java.lang.invoke.MethodType,java.lang.invoke.MethodHandle,java.lang.invoke.MethodType)>(class "(I)V", handle: <java.util.function.Consumer: void accept(java.lang.Object)>, class "(I)V")
at soot.asm.AsmMethodSource.setUnit(AsmMethodSource.java:323)
at soot.asm.AsmMethodSource.assignReadOps(AsmMethodSource.java:375)
at soot.asm.AsmMethodSource.convertInvokeDynamicInsn(AsmMethodSource.java:1278)
at soot.asm.AsmMethodSource.convert(AsmMethodSource.java:1574)
at soot.asm.AsmMethodSource.getBody(AsmMethodSource.java:1737)
... 16 more

I don't quite understand how the stack model works there.
@StevenArzt can you figure this out?

My command line is:

-w -keep-line-number -allow-phantom-refs -p cg all-reachable:true -f none -include-all -cp /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home/jre/lib/rt.jar java.nio.file.Files java.lang.invoke.LambdaMetafactory

[ericbodden]

Thanks Steven!

Cheers,
Eric

On 23.09.2014, at 15:54, Steven Arzt notifications@github.com wrote:

Closed #225 via 3349714.

—
Reply to this email directly or view it on GitHub.

[Variadicism-zz]

@ericbodden By the sound of your last comment, it seems you were successful. Is this true? Did you guys manage to get ASM to work with Java 8?

If so, I'm confused.... What does Soot have to do with ASM? Could I use this to get my ASM to work with Java 8 using these changes that you guys made to Soot on GitHub? I have been trying hard to do just that, but I've been stuck on the same problem you mentioned at the start of this post with ASM 5.0.3.

I would really appreciate any advice. If I can't figure out how to make ASM work with Java 8, the A.P.I. I'm making will be have to be downgraded to Java 7, which is going to significantly impact my timetable and make the A.P.I. uglier and slower. (The second best solution I've come up with besides those Java 8 default interface methods for this A.P.I. is using Java reflection, which is uglier and slower.)

[ericbodden]

What problem exactly are you facing? I just had to apply this change to make the new API work:

1e2225b

In Soot we are using ASM to parse the bytecode, creating Jimple code out of it.

Cheers,

Eric

Sent from my mobile

On Sep 25, 2014 8:25 PM, REALDrummer notifications@github.com wrote:

@ericboddenhttps://github.com/ericbodden By the sound of your last comment, it seems you were successful. Is this true? Did you guys manage to get ASM to work with Java 8?

If so, I'm confused.... What does Soot have to do with ASM? Could I use this to get my ASM to work with Java 8 using these changes that you guys made to Soot on GitHub? I have been trying hard to do just that, but I've been stuck on the same problem you mentioned at the start of this post with ASM 5.0.3.

I would really appreciate any advice. If I can't figure out how to make ASM work with Java 8, the A.P.I. I'm making will be have to be downgraded to Java 7, which is going to significantly impact my timetable and make the A.P.I. uglier and slower. (The second best solution I've come up with besides those Java 8 default interface methods for this A.P.I. is using Java reflection, which is uglier and slower.)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/225#issuecomment-56861802.

[Variadicism-zz]

@ericbodden Ah. I guess I was mistaken. I thought for a moment that you had changed ASM 5.0.3 to work with Java 8, but I think that actually 5.0.3 already works with Java 8. I just need to modify the code that's calling it to work with Java 8.

I'm attempting to use Minecraft Coders' Pack to deobfuscate, modify, then reobfuscate Minecraft code. This system uses ASM for the reobfuscation step, but the version of ASM they used is old and does not include Java 8 support. I am attempting to replace their ASM with ASM 5.0.3 in the code, but the code is still calling ASM functions in terms of ASM 4 and causing errors.

I will figure it out. Sorry to bother you. This post has helped me locate the problem; I appreciate it.