NPE when analyzing non-resolvable method.

[JesusFreke]

Original issue 29 created by JesusFreke on 2010-05-14T08:26:25.000Z:

What seems to be the problem?
NPE when analyzing non-resolvable method. Should be ValidationException.

What is the exact smali/baksmali command that you ran?
java -jar baksmali-1.2.2.jar -x CheckinProvider.odex -c
core.odex:framework.odex:ext.odex:services.odex:android.policy.odex

What version are you using?
1.2.2

[JesusFreke]

Comment #1 originally posted by JesusFreke on 2010-06-08T19:01:53.000Z:

Yeah, get:
"Replaced unresolvable optimized instruction with a throw"

When deodexing CheckinProvider, even with the full bootclasspath in -c switch.

Any ideas JF?

[JesusFreke]

Comment #2 originally posted by JesusFreke on 2010-06-09T08:51:09.000Z:

I am not sure, but is this correct?

.line 1940
.local v0, bNeedCancel:Z
#v0=(Null);v1=(Uninit);v2=(Uninit);v3=(Uninit);v4=(Reference,Landroid/os/PowerManager;);v5=(Uninit);v6=(Reference,Landroid/os/PowerManager$WakeLock;);v7=(One);v8=(Reference,Ljava/lang/String;);v9=(Reference,Ljava/lang/String;);v10=(Uninit);p0=(Reference,Lcom/android/providers/htcCheckin/htcCheckinService$htcCheckinReceiver;);
const/4 v2, 0x0
#v0=(Null);v1=(Uninit);v2=(Null);v3=(Uninit);v4=(Reference,Landroid/os/PowerManager;);v5=(Uninit);v6=(Reference,Landroid/os/PowerManager$WakeLock;);v7=(One);v8=(Reference,Ljava/lang/String;);v9=(Reference,Ljava/lang/String;);v10=(Uninit);p0=(Reference,Lcom/android/providers/htcCheckin/htcCheckinService$htcCheckinReceiver;);

I do not understand why v2 now equals Null. To me it looks like an Integer constant equal 0 (zero). Or am I wrong?

The "Replaced unresolvable optimized..." error looks like:

#v0=(Null);v1=(Uninit);v2=(Null);v3=(Uninit);v4=(Reference,Landroid/os/PowerManager;);v5=(Uninit);v6=(Reference,Landroid/os/PowerManager$WakeLock;);v7=(Reference,Lcom/android/providers/htcCheckin/htcCheckinService;);v8=(Reference,Ljava/lang/String;);v9=(Reference,Ljava/lang/String;);v10=(Uninit);p0=(Reference,Lcom/android/providers/htcCheckin/htcCheckinService$htcCheckinReceiver;);
#Replaced unresolvable optimized instruction with a throw
throw v2
#invoke-virtual-quick {v2, v8}, vtable@0x21
#v0=(Null);v1=(Uninit);v2=(Null);v3=(Uninit);v4=(Reference,Landroid/os/PowerManager;);v5=(Uninit);v6=(Reference,Landroid/os/PowerManager$WakeLock;);v7=(Reference,Lcom/android/providers/htcCheckin/htcCheckinService;);v8=(Reference,Ljava/lang/String;);v9=(Reference,Ljava/lang/String;);v10=(Uninit);p0=(Reference,Lcom/android/providers/htcCheckin/htcCheckinService$htcCheckinReceiver;);

[JesusFreke]

Comment #3 originally posted by JesusFreke on 2010-06-09T09:34:21.000Z:

@tommy4st
0x0 could be 0, false or null - it depends on context, so everything is ok.

About "Replaced unresolvable optimized...": it's a feature, not a bug. And referencing JF's words:

http://code.google.com/p/smali/issues/detail?id=31#c2
"It's likely that class doesn't exist on the phone. You'll need to see if the class
exists in any of the framework files. (baksmali them, and see if there's a smali file
for the class that it can't find)."

It's weird to me, because AFAIR behnam.fakhimi's problem is related to some method, which is used, so even original app should FC.

Anyway I don't know, what these things have in common with above NPE issue ;-)

[JesusFreke]

Comment #4 originally posted by JesusFreke on 2010-06-09T13:16:12.000Z:

Actually, my comments on issue 31 are unrelated to the "Replaced unresolvable optimized..." thing. This isn't caused by not being able to resolve a method. It occurs when the object register referenced by an odexed instruction is always null. I've been asked about this a number of times, so I think I'll just write up a wiki page about it. I'll post back here once the wiki page is up (probably this evening)

[JesusFreke]

Comment #5 originally posted by JesusFreke on 2010-06-09T13:50:48.000Z:

Hmmmm, the simple fact is that an baksmali'd from odex and smali'd app will exhibit no errors in processing and then crash in operation... so whether it's a bug or not, it makes the process unpredictable!

Examples of apps with problems are the HtcMusic app and the CheckinProvider app... if I can provide any further info that might help please shout.

[JesusFreke]

Comment #6 originally posted by JesusFreke on 2010-06-09T14:02:45.000Z:

I have noticed that the unresolved function can be very random. For example the code can look like this:

invoke-virtual {}, bla

and then a few lines later you have

# replaced unresolved method with throw
invoke-virtual-quick {}, vtable@0x00

a lot of times the unresolved code should in fact be the same call that was a couple of lines earlier.

[JesusFreke]

Comment #7 originally posted by JesusFreke on 2010-06-09T15:34:40.000Z:

@modacopaul
This might be even worse. If you baksmali already resmalied code, there will be no traces of what has happened - several lines of code missing and that's all.

[JesusFreke]

Comment #8 originally posted by JesusFreke on 2010-06-09T15:38:34.000Z:

@modacopaul, you need to search the code for "unresolvable" right after you deodexed. If you deodex, smali and then baksmali there will be no trace of unresolvable (except for the throw) because they are just comments and are stripped out when smaling.

[JesusFreke]

Comment #9 originally posted by JesusFreke on 2010-06-10T01:01:50.000Z:

master.homer: It's not random at all. It occurs when the object register is always null. I've just put up a wiki page that helps to explain this:

http://code.google.com/p/smali/wiki/UnresolvableOdexInstruction

Brut.all: that is correct: the code that is removed is "dead" code that is impossible to reach (and in some cases, impossible to deodex). This is all handled in a manner that preserves the functionality/semantics of the code. Or if it isn't, it's a bug in baksmali. But nothing I've seen yet indicates that there's a bug, other than the issue that I mention at the bottom of the wiki page.

If you think there is in fact an issue in how this is handled, please provide a specific example, preferably with the original code before being odexed (if available), the odexed code, and the code that baksmali generated after deodexing :)

[JesusFreke]

Comment #10 originally posted by JesusFreke on 2010-06-10T01:50:40.000Z:

The original issue that brut.all had submitted has been fixed :)

http://code.google.com/p/smali/source/detail?r=732

[JesusFreke]

Comment #11 originally posted by JesusFreke on 2010-06-10T07:32:33.000Z:

@jf
Ahh, thanks for explanation. I must say I was surprised when we were talking about that problem earlier and you said that you don't see anything wrong ;-) Now I see.

But what for something like:

if-nez v0, :cond_1
const v1, 0x5
goto :goto_1
:cond_1
invoke-static {}, Lsome/Class;->getSomeObject()Lsome/Class;
move-result-object v1
:goto_1
invoke-virtual-quick {v1), vtable@7

Then you can't resolve odex instruction, because v1 is in conflicted state, but if v0 would be always true, then this code would work perfectly fine before deodexing. Of course javac can't output such instructions and baksmali can't do everything, because it statically analyzes the code. I'm just wondering, whether it is possible to easily forbid files deodexing using this technique.

[JesusFreke]

Comment #12 originally posted by JesusFreke on 2010-06-10T07:34:55.000Z:

Agreed, this is a tricky scenario... it at least needs to be 'caught'...

P

[JesusFreke]

Comment #13 originally posted by JesusFreke on 2010-06-10T12:22:33.000Z:

Here is the output of my deodexing of FRF50s app and framework, with my deodexing script looking for the 'unresolvable' methods. These unresolvable methods are quite frequent and cause issues of course when smali'ing back (the relevant apps crash). Any thoughts on the best workaround?

:-(

P

APP:

Contacts.odex
com/android/contacts/ExportVCardActivity$ActualExportThread.smali: #Replaced unresolvable optimized instruction with a throw

DownloadProvider.odex
com/android/providers/downloads/DownloadThread.smali: #Replaced unresolvable optimized invoke--range-quick instruction
com/android/providers/downloads/DownloadThread.smali: #Replaced unresolvable optimized invoke--range-quick instruction
com/android/providers/downloads/DownloadThread.smali: #Replaced unresolvable optimized invoke--range-quick instruction
com/android/providers/downloads/DownloadThread.smali: #Replaced unresolvable optimized invoke--range-quick instruction

Gmail.odex
com/google/android/gm/provider/MailSync.smali: #Replaced unresolvable optimized instruction with a throw

GoogleServicesFramework.odex
com/google/android/gsf/loginservice/MigrateToAccountManagerBroadcastReceiver.smali: #Replaced unresolvable optimized instruction with a throw
com/google/android/gsf/loginservice/MigrateToAccountManagerBroadcastReceiver.smali: #Replaced unresolvable optimized instruction with a throw

MediaProvider.odex
com/android/providers/media/MediaThumbRequest.smali: #Replaced unresolvable optimized invoke--range-quick instruction
com/android/providers/media/MediaThumbRequest.smali: #Replaced unresolvable optimized invoke--range-quick instruction

Mms.odex
com/android/mms/ui/ComposeMessageActivity.smali: #Replaced unresolvable optimized instruction with a throw
adding: classes.dex (deflated 52%)

PassionQuickOffice.odex
de/schlichtherle/io/ArchiveControllers.smali: #Replaced unresolvable optimized instruction with a throw

Settings.odex
com/android/settings/TetherSettings.smali: #Replaced unresolvable optimized instruction with a throw
com/android/settings/TetherSettings.smali: #Replaced unresolvable optimized instruction with a throw

Street.odex
com/google/android/street/HttpCache.smali: #Replaced unresolvable optimized instruction with a throw

FRAMEWORK:

am.odex
android/server/BluetoothService$BondState.smali: #Replaced unresolvable optimized instruction with a throw
android/server/BluetoothService$BondState.smali: #Replaced unresolvable optimized instruction with a throw
com/google/android/mms/pdu/PduPersister.smali: #Replaced unresolvable optimized instruction with a throw

framework.odex
android/content/pm/RegisteredServicesCache.smali: #Replaced unresolvable optimized instruction with a throw
android/database/sqlite/SQLiteDirectCursorDriver.smali: #Replaced unresolvable optimized instruction with a throw
android/media/MiniThumbFile.smali: #Replaced unresolvable optimized instruction with a throw
android/net/http/HttpsConnection.smali: #Replaced unresolvable optimized invoke-*-range-quick instruction
android/server/BluetoothService$BondState.smali: #Replaced unresolvable optimized instruction with a throw
android/server/BluetoothService$BondState.smali: #Replaced unresolvable optimized instruction with a throw
com/google/android/mms/pdu/PduPersister.smali: #Replaced unresolvable optimized instruction with a throw

services.odex
com/android/server/BackupManagerService$PerformRestoreTask.smali: #Replaced unresolvable optimized instruction with a throw
com/android/server/BackupManagerService$PerformRestoreTask.smali: #Replaced unresolvable optimized invoke--range-quick instruction
com/android/server/BackupManagerService$PerformRestoreTask.smali: #Replaced unresolvable optimized invoke--range-quick instruction
com/android/server/BackupManagerService$PerformRestoreTask.smali: #Replaced unresolvable optimized invoke--range-quick instruction
com/android/server/BackupManagerService$PerformRestoreTask.smali: #Replaced unresolvable optimized invoke--range-quick instruction
com/android/server/BackupManagerService.smali: #Replaced unresolvable optimized instruction with a throw
com/android/server/DropBoxManagerService.smali: #Replaced unresolvable optimized invoke-*-range-quick instruction
com/android/server/WallpaperManagerService.smali: #Replaced unresolvable optimized instruction with a throw
com/android/server/WallpaperManagerService.smali: #Replaced unresolvable optimized instruction with a throw
com/android/server/WindowManagerService.smali: #Replaced unresolvable optimized instruction with a throw
com/android/server/WindowManagerService.smali: #Replaced unresolvable optimized instruction with a throw
com/android/server/am/ActivityManagerService.smali: #Replaced unresolvable optimized instruction with a throw

[JesusFreke]

Comment #14 originally posted by JesusFreke on 2010-06-10T12:27:45.000Z:

@paul, the only way I could get it to work was by comparing the unresolvable code from a working smali. Feel free to use the framework and apps from VillainROM 10.1 if you don't have time to do that.

[JesusFreke]

Comment #15 originally posted by JesusFreke on 2010-06-10T12:29:15.000Z:

This is Froyo... so that's not really an option...

P

[JesusFreke]

Comment #16 originally posted by JesusFreke on 2010-06-10T12:30:37.000Z:

Oh, I see. Well, then you'll have to do that I guess.

[JesusFreke]

Comment #17 originally posted by JesusFreke on 2010-06-10T12:33:25.000Z:

There is no Froyo working smali. Doh. We need a fix. :)

P

[JesusFreke]

Comment #18 originally posted by JesusFreke on 2010-06-10T12:55:26.000Z:

Paul, go read http://code.google.com/p/smali/wiki/UnresolvableOdexInstruction. "Replaced unresolvable optimized" isn't something to be concerned about.

[JesusFreke]

Comment #19 originally posted by JesusFreke on 2010-06-10T12:57:19.000Z:

@jf, but they are, if you don't fix them you get FC's (due to the throw). (CheckinProvider for example)

[JesusFreke]

Comment #20 originally posted by JesusFreke on 2010-06-10T12:59:58.000Z:

Yeah, I read the wiki entry, but a whole ton of apps are crashing due to this. On Froyo for example, the MediaProvider ones break Gallery completely and cause constant FCs!

P

[JesusFreke]

Comment #21 originally posted by JesusFreke on 2010-06-10T13:03:47.000Z:

Brut.all, I'm not positive, but I'm pretty sure baksmali would in fact complain about something like that, where the object register was in a conflicted state. Have you seen otherwise?

[JesusFreke]

Comment #22 originally posted by JesusFreke on 2010-06-10T13:10:20.000Z:

modacopaul: OK. so what code is baksmali replacing incorrectly? Just giving me a list of all the places where it replaces an unresolvable odex instruction doesn't tell me anything :).

Some me some specifics. You say it crashes - so give me the details on the crash, and the odexed and deodexed code where it crashed, and the changes you made to fix it.

[JesusFreke]

Comment #23 originally posted by JesusFreke on 2010-06-10T13:22:51.000Z:

OK, as an example, here's the code from MediaProvider...

after baksmali:

.line 163
#Replaced unresolvable optimized invoke-*-range-quick instruction
#with a generic method call that will throw a NullPointerException
invoke-virtual/range {v20 .. v20}, Ljava/lang/Object;->hashCode()I
#invoke-virtual-quick/range {v20 .. v20}, vtable@0xb

#move-wide/from16 v9, v16

.line 164
.end local v16           #magic:J
.restart local v9       #magic:J
#goto :goto_90

.line 161
.end local v9           #magic:J
.restart local v16       #magic:J
:catchall_9f
move-exception v5

Now, bearing in mind we have no other Froyo code other than what i'm trying to deodex here, it's tricky... but I did compare these lines to an Eclair file, and the lines are as follows...

.line 122
invoke-virtual/range {v21 .. v21}, Landroid/os/ParcelFileDescriptor;->close()V

move-wide/from16 v9, v17

.line 123
.end local v17           #magic:J
.restart local v9       #magic:J
goto :goto_2e

.line 120
.end local v9           #magic:J
.restart local v17       #magic:J
:catchall_91
move-exception v5

I'll post the logcat it generates in the crash shortly...

P

[JesusFreke]

Comment #24 originally posted by JesusFreke on 2010-06-10T13:24:10.000Z:

master.homer: The whole point is that the code would have thrown an NPE at that point anyway - or if it wouldn't have, then it's a bug in baksmali. You say that

"if you don't fix them you get FC's (due to the throw)" - and it sounds like you are talking about all instances where baksmali replaces a deodexable instruction with a throw. But this is valid and intended functionality.

I think the point that I'm trying to make is not to talk about these cases in the aggregate - because there are definitely cases where this is the "right thing to do". (really, the only thing to do).

Instead, focus on the specific cases that cause you problems. You say that it crashes - so show me the crash. And the odexed and deodexed code where it crashed, and the changes you made to fix it.

[JesusFreke]

Comment #25 originally posted by JesusFreke on 2010-06-10T13:28:16.000Z:

paul, I need more context. The code that is leading up to the unresolvable odexed instruction is extremely important. I need to be able to determine if the object register would always be null or not. If you want to pastebin the whole method, that would be fine :). Using -r in baksmali would be even better :)

[JesusFreke]

Comment #26 originally posted by JesusFreke on 2010-06-10T13:29:53.000Z:

It must be remembered that the cases where this is causing a crash on device (FC) are not in the minority... i've had the issue on Hero 2.1 CheckinProvider and Music apps and now in the Froyo MediaProvider. It's a pain because these crashes don't immediately manifest themselves in testing, they can be in very specific parts of functionality in applications...

P

[JesusFreke]

Comment #27 originally posted by JesusFreke on 2010-06-10T13:32:07.000Z:

Logcat below.

06-10 13:31:32.968: ERROR/AndroidRuntime(409): FATAL EXCEPTION: main
06-10 13:31:32.968: ERROR/AndroidRuntime(409): java.lang.VerifyError: com.android.providers.media.MediaThumbRequest
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at com.android.providers.media.MediaProvider.(MediaProvider.java:91)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at java.lang.Class.newInstanceImpl(Native Method)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at java.lang.Class.newInstance(Class.java:1429)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at android.app.ActivityThread.installProvider(ActivityThread.java:4494)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at android.app.ActivityThread.installContentProviders(ActivityThread.java:4281)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4237)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at android.app.ActivityThread.access$3000(ActivityThread.java:125)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2071)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at android.os.Handler.dispatchMessage(Handler.java:99)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at android.os.Looper.loop(Looper.java:123)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at android.app.ActivityThread.main(ActivityThread.java:4627)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at java.lang.reflect.Method.invokeNative(Native Method)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at java.lang.reflect.Method.invoke(Method.java:521)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:868)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:626)
06-10 13:31:32.968: ERROR/AndroidRuntime(409): at dalvik.system.NativeStart.main(Native Method)

[JesusFreke]

Comment #28 originally posted by JesusFreke on 2010-06-10T13:32:59.000Z:

paul: wait, does that code cause a crash, or does dalvik reject it on install? I'm guessing the latter, because it's illegal for execution to reach a move-exception instruction normally.

That looks like the same issues that I described at the bottom of the wiki page. Or at least, one that is very similar, and which I am already aware of (and attempting to fix).

I believe the issue with this specific case isn't that the deodexable code shouldn't be replaced, but it should probably be replaced with a throw instead, so that dalvik doesn't think execution might continue on past the invoke (which we know will always cause an NPE, but dalvik doesn't)

[JesusFreke]

Comment #29 originally posted by JesusFreke on 2010-06-10T13:33:33.000Z:

Full smali file - http://content.modaco.net/dropzone/MediaThumbRequest.smali

P

[JesusFreke]

Comment #30 originally posted by JesusFreke on 2010-06-10T13:38:17.000Z:

I checked several "Repplaced..." things in framework.odex and MediaProvider.odex files - seems baksmali works ok. All modifications were done for the code which is unaccessible: register's value is 0x0 and there is some if-nez check right before replaced code.

[JesusFreke]

Comment #31 originally posted by JesusFreke on 2010-06-10T13:44:15.000Z:

Brut.all - exactly. I think that all the issues that every is seeing are in fact due to that issue that I mentioned at the bottom of the wiki page (although the issue is slightly wider in scope than I had described there, but it's still essentially the same issue). Which is extremely tricky to fix. But I think I'm getting a handle on it :).

Paul: for that code, instead of the fix that you used, try replacing:

invoke-virtual/range {v20 .. v20}, Ljava/lang/Object;->hashCode()I

with a:

throw v20

and leaving the rest of the commented out code after it commented out. I believe that is what baksmali should do for this specific case.

The actual fix in the code that you suggested is actually the goto instruction - this makes dalvik happy because it prevents execution from continuing to the move-exception instruction.

Also - details on the crash would still be nice. i.e. anything related in logcat when the crash happens. There will almost certainly be some messages from dalvik, saying that it doesn't like the move-exception instruction.

[JesusFreke]

Comment #32 originally posted by JesusFreke on 2010-06-10T13:45:44.000Z:

Paul, sorry. I hadn't seen the logcat info before I asked for it. Thanks! Although, can you take a look and see if there's any other messages from dalvik before that?

[JesusFreke]

Comment #33 originally posted by JesusFreke on 2010-06-10T13:46:11.000Z:

Ah ha...

W/dalvikvm( 462): VFY: invalid use of move-exception
W/dalvikvm( 462): Verifier rejected class Lcom/android/providers/media/MediaThumbRequest;
W/dalvikvm( 462): threadid=1: thread exiting with uncaught exception (group=0x4001d7f0)

:)

P

[JesusFreke]

Comment #34 originally posted by JesusFreke on 2010-06-10T13:49:33.000Z:

Paul: yep! thought so :). Like I said, I'm already aware of this specific issue, and am working on a fix.

[JesusFreke]

Comment #35 originally posted by JesusFreke on 2010-06-10T13:53:26.000Z:

Yeah, i'd missed that in the logcat, doh! :) Thanks!

P

[JesusFreke]

Comment #36 originally posted by JesusFreke on 2010-06-10T13:53:42.000Z:

Brut.all: as I think about it more, I think that dalvik itself might complain about a case like you described when odexing it, where the object register is conflicted. Have you actually seen a case like this in practice?

[JesusFreke]

Comment #37 originally posted by JesusFreke on 2010-06-10T13:56:29.000Z:

No. I was theorizing.

[JesusFreke]

Comment #38 originally posted by JesusFreke on 2010-06-10T13:59:11.000Z:

But could javac produce such situation? Maybe some kind of optimizer like proguard - it may do some runtime guessing.

[JesusFreke]

Comment #39 originally posted by JesusFreke on 2010-06-10T14:28:41.000Z:

It's not an issue of whether javac or an optimizer could produce it. It would actually be in the domain of dx, since it does all the register allocating, etc. (normal java bytecode doesn't have registers at all, it's stack based). But even if they (or dx) did produce code like that, I'm fairly sure dalvik would refuse to odex it - so a case like that would never need to be deodexed :)

[JesusFreke]

Comment #40 originally posted by JesusFreke on 2010-06-10T14:35:29.000Z:

FYI: I've created another issue specifically for the issue we've been discussing.

http://code.google.com/p/smali/issues/detail?id=33

[JesusFreke]

Comment #41 originally posted by JesusFreke on 2010-06-11T05:39:27.000Z:

I've committed some experimental changes which should better handle cases like these. Please grab a copy of the ExperimentalDeodexChange branch and try it out, and let me know how it works for you :)

svn checkout http://smali.googlecode.com/svn/branches/ExperimentalDeodexChange smali

[JesusFreke]

Comment #42 originally posted by JesusFreke on 2010-06-11T05:41:38.000Z:

<empty>