New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jasmali feature (lost "-j" switch) #199

Closed
iBotPeaches opened this Issue Mar 18, 2015 · 16 comments

Comments

Projects
None yet
2 participants
@iBotPeaches
Copy link
Owner

iBotPeaches commented Mar 18, 2015

Original issue 88 created by Brut.alll on 2010-08-27T13:57:02.000Z:

Many people ask about "-j" switch from my video on YouTube. I think I have to tell you whole story from the beginning to the end to explain, why it isn't there. I have decided to write this in the form of an issue, not wiki, because I'm considering to add this feature or maybe something similar.

So... months ago I was thinking about possibility to debug smali code step by step, which would greatly improve working with existing apps. I did some tests to prove it's possible to inject fake debugging symbols and successfully run debugging session. Injection was really easy thing to do, but there was much harder thing: if you want to watch variables, then IDE have to know at least declarations of classes, methods, fields, etc., so it will be able to properly read memory contents. Actually I was wrong about that, cause debugger server gives all that data to a client, so client doesn't really need these declarations. But at this point I didn't know about that.

I started to work on "Java declarations decoder" tool, so variables watching will be possible. It would create proper Java declarations of classes, methods and fields, but doesn't touch hardest part: method bodies, the real code. Then I got an idea to make possible to easily mix smali and Java. If we have all declarations in Java, so why not to write some code in it? Even if we can't decompile existing code to Java, we could at least add new one in this language.

Mixing process was:

  • Compile Java code to smali (javac -> dx -> baksmali).
  • Mix above smali code with smali "sources" (3rd party code mostly). Apktool was using annotations to distinguish Java and smali methods.
  • Assemble mixed smali code using smali tool.

Great thing was that we have all these Java declarations, so we could use full power of our IDE. Also this made possible to use 3rd party objects in our code, because javac will compile all method calls, field accesses, etc. to proper Lsome/package/SomeClass;->methodName()Z form. It was unbelievable for me, but I was able to mix bytecodes from two different compilations, with references between them and resulting bytecode was valid as a whole. Awesome! :-) I had built basic declarations decoding mechanisms into apktool and then I had created "-j" switch video to show people this awesomeness on some simple sample app.

Unfortunately when I tried to move from sample app to productive (or at least beta) quality I had failed. I found that declarations decoding will be much, much harder then I thought. I thought decoding will be 1:1, so all I need is to read all names, types, method arguments, etc. and write them in Java. I was wrong. First, Java language is very restrictive. You can't add empty methods just like that, cause this will be a syntax error. If method returns something, you have to really return same type, if there are final fields, then you have to set them in constructors, if parent class has some constructors, then you have to call these constructors, so I would need to generate whole inheritance tree of classes :-/

But even worse is: Java sources and compiled bytecode isn't 1:1 at all, there are many exceptions from this rule - javac generates a lot of new code at compile time. Some of these additional bits, e.g. auto-generated methods of Enums wouldn't be a problem - they would just mess up real code a little. But there are much worse cases. I found two methods with exactly the same names and arguments. In the bytecode it's valid because methods are identified by whole prototype string, so: method name, arguments types and return type - these methods had different return type. But in the Java it's invalid because methods aren't identified by return type, so this case couldn't be converted to Java, even by doing this manually. Of course I could change name of one of these methods, but then I would have to change all references to it as well - which is possible, but not that easy.

Still it should be possible to convert as much as possible automatically and leave other bits written in original smali form, but error detection wouldn't be that easy. My approach was very simple: read bytecode declarations one after another and output them in the Java form. If I would want to not convert problematic declarations, I would need some advanced code analyzing algorithms to detect these cases.

My enthusiasm dropped, but fortunately I found that actually I don't need these declarations to run smali debugging. I left declarations decoding idea - "maybe someday".

Some time passed, I have added new features to apktool and now it would be good to think about Jasmali (Java and smali mixing) again. Actually... I was using Java and smali mixing all that time, but without apktool support. This is possible to do very easily, without any advanced algorithms. I have written shell script which has... ~5 lines of code :-) It uses same technique as listed above, but because I don't have declarations in Java and because mixing of methods is much more complicated, my approach is limited:

  • I mix whole files/classes, not individual methods. I add new Java files, there I put my code and if I want to modify something in 3rd party code, then I do this in smali. Actually even if I would be able to mix smali and Java methods in one file, I would not use that feature. It's messy. It's better to have your code in your own classes and do minimal changes to 3rd party code.
  • I have to write Java declarations of 3rd party code manually. Of course not all of them, just these I reference in my code.

My script is here: http://pastebin.com/ge39wRZS . All you have to do is:

  • Install linux ;-)
  • Decode apk using apktool.
  • Make sure you have javac, dx and apktool in your path.
  • Place above script in main dir.
  • Place or link android.jar file from Android SDK. Newer platform is better.
  • Create brut.iface and brut.src dirs - select them as Java sources dirs in your IDE.
  • Create your Java classes in brut.src dir.
  • If you want to use some 3rd party objects in your code, then create simple dummy/stub class with identical declarations as these in smali code. You don't have to declare all methods and fields, just these you are using.
  • Run above script. You have to do this after any change to brut.src dir.
  • Run apktool b.
  • Have fun.

Of course this is possible to do on Windows as well, but I won't port above script. I'm sure there are many good developers on windows and they can do that ;-) Also I think my script should work on Cygwin.

Now I'm considering, what to do next. Easiest solution would be to implement this above technique in apktool. It would do everything automatically on "apktool b" command - without any scripts. Also it could help us create stub classes. These files have to be valid Java files, so apktool can't create them all at "apktool d" time (for same reasons as above), but it can create these files on demand. "apktool generate-ifaces Lcom/android/example;->someMethod(IID)Z Lfoo/bar/Baz;->i:I;" - something like that.

Also I could try to work more on that declarations decoding algorithms. I could analyze the bytecode a little to minimize risk of generating invalid code. Then: if I will be able to decode most of things, I could go back to full decoding as in "-j" video. Some problematic parts of code would be pure smali, so if you would want to do something with them, you would have to do it manually. If I will be able to properly decode only some of bits, then I could try to mix up full decoding with this limited solution above. Apktool would try to create as much declarations in iface dir as it can ( "apktool d"), but smali dir wouldn't be decoded - only classes declarations, as in "apktool d -d" command. Then user would be able to add his own code directly into these files and they would be linked against declarations in iface dir, so he would have nearly full functionality of my video on YouTube. I think this solution is quite possible to do.

The question is: how many people are there, who really need this feature? It would be a lot of work, you know.

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #1 originally posted by Brut.alll on 2010-08-28T15:54:04.000Z:

Issue 82 has been merged into this issue.

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #2 originally posted by vampire0 on 2011-01-12T18:32:55.000Z:

An additional suggestion to what was shown in that cool video: to be able to mix Java and smali code within one method. One possible solution to achieve this mixing would maybe be to internally split a joint Java/Smali method into several methods, each either pure Smali or Java which call always the next one in the end. This way you could mix Java and Smali arbitratirly within one method which would be even cooler than what you've shown in that video.

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #3 originally posted by Brut.alll on 2011-05-03T12:17:00.000Z:

I wanted to make possible to mix smali and Java in one method, but that would be very, very hard and would probably degrade performance of resulting code.

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #4 originally posted by vampire0 on 2011-05-03T14:04:04.000Z:

Why do you think it would be hard? Don't you think my proposal would work relatively easily and without too much performance loss?

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #5 originally posted by Brut.alll on 2011-05-03T14:10:32.000Z:

If you will split method to several real methods, then you will hit serious performance loss, because each method call uses some CPU cycles. But if you want to split it virtually, so it will be joined back by apktool before assembling, then you would get real problems to pass variables between method chunks - you don't know, which registers javac/dx will choose to store variables.

Or maybe I don't understand you :-)

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #6 originally posted by vampire0 on 2011-05-06T00:11:30.000Z:

Of course you will loose some CPU cycles and resources for the method call. But as long as you don't mix Smali and Java code you will not get multiple methods and thus will not have a performance loss.
If you mix Smali and Java code, then you have decreased performance, but it is possible which I as user would prefer and find super cool.
So as far as I can see, you get an additional feature which usage will cause a slight loss in performance but you don't have to use it.

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #7 originally posted by vladimir@slate-project.org on 2011-07-01T20:08:55.000Z:

As an alternative to writing your own decompiler, could using an existing one (via for example JD's Eclipse integration http://java.decompiler.free.fr/?q=jdeclipse) be an option?

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #8 originally posted by vampire0 on 2011-07-01T20:21:28.000Z:

What has a decompiler to do with this issue or the -j switch?

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #9 originally posted by vladimir@slate-project.org on 2011-07-01T20:25:39.000Z:

To be able to decode the files you need directly to Java so you can change them at will (without the need to see smali).

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #10 originally posted by vampire0 on 2011-07-01T20:32:03.000Z:

I know what a decompiler is for.
But again, what has a decompiler to do with this issue or the -j switch?

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #11 originally posted by Brut.alll on 2011-07-18T18:12:11.000Z:

@vladimir
You can't use existing decompilers, because Android doesn't use JVM bytecode, but Dalvik bytecode. You can try dex2jar + JD, but this isn't too reliable.

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #12 originally posted by kamen.vitanov on 2012-11-27T00:05:00.000Z:

I'd rather have back the old support for '-d' option - where it was generating .java files with all the small code as a comment. It worked great but unfortunately the latest version doesn't support it - issue 309

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #13 originally posted by adrian.r.tiron on 2012-12-24T07:40:59.000Z:

i don't understand.....so in conclusion if i have an apk is it possible to debug it/set some breakpoints and modify it and compile/install it back on a phone? I don't have the java sources.
Thanks very much!

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #14 originally posted by michee08 on 2012-12-24T07:41:28.000Z:

i don't understand.....so in conclusion if i have an apk is it possible to debug it/set some breakpoints and modify it and compile/install it back on a phone? I don't have the java sources.
Thanks very much!

@iBotPeaches

This comment has been minimized.

Copy link
Owner

iBotPeaches commented Mar 18, 2015

Comment #15 originally posted by connor.tumbleson on 2012-12-30T03:16:41.000Z:

<empty>

@telkomsel13

This comment has been minimized.

Copy link

telkomsel13 commented May 15, 2015

#1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment