New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

taimen android 8.1 deodexing issues #3

Closed
Furniel opened this Issue Dec 20, 2017 · 10 comments

Comments

Projects
None yet
4 participants
@Furniel

Furniel commented Dec 20, 2017

Tested taimen-opm1.171019.011-factory-2df1c1cb deodexing and got two errors during framework deodexing:

[DEBUG] [11132] 2017/12/20 20:22:23 (vdexExtractor.c:206 main) Processing 'services.vdex'
[DEBUG] [11132] 2017/12/20 20:22:23 (vdex.c:64 vdex_isVersionValid) Vdex version '010' detected
------ Vdex Header Info ------
magic header & version      : vdex-010
number of dex files         : 1 (1)
dex size (overall)          : 85b388 (8762248)
verifier dependencies size  : 120f8 (73976)
verifier dependencies offset: 85b3a4 (8762276)
quickening info size        : a72 (2674)
quickening info offset      : 86d49c (8836252)
dex files info              :
  [0] location checksum : 7c35823a (2083881530)
---- EOF Vdex Header Info ----
[DEBUG] [11132] 2017/12/20 20:22:23 (vdex.c:110 vdex_GetNextDexFileData) Processing first Dex file at offset:0x1c
[DEBUG] [11132] 2017/12/20 20:22:23 (dex.c:327 dex_isValidDexMagic) Dex version '037' detected
[ERROR] [11132] 2017/12/20 20:22:23 (vdex_backend_v10.c:451 vdex_process_v10) Unexpected checksum (89f9fc35 vs fa7bb981) - failed to unquicken Dex file
[DEBUG] [11132] 2017/12/20 20:22:23 (vdex.c:217 vdex_process) Took 78 ms to process Vdex file
[ERROR] [11132] 2017/12/20 20:22:23 (vdexExtractor.c:262 main) Failed to process Dex files - skipping 'services.vdex'
[INFO] 0 out of 1 Vdex files have been processed
[INFO] 0 Dex files have been extracted in total
[INFO] Extracted Dex files are available in '.'

and

[INFO] Processing 1 file(s) from wifi-service.vdex
[DEBUG] [2284] 2017/12/20 20:28:09 (vdexExtractor.c:206 main) Processing 'wifi-service.vdex'
[DEBUG] [2284] 2017/12/20 20:28:09 (vdex.c:64 vdex_isVersionValid) Vdex version '010' detected
------ Vdex Header Info ------
magic header & version      : vdex-010
number of dex files         : 1 (1)
dex size (overall)          : 1cffac (1900460)
verifier dependencies size  : 34f0 (13552)
verifier dependencies offset: 1cffc8 (1900488)
quickening info size        : 74 (116)
quickening info offset      : 1d34b8 (1914040)
dex files info              :
  [0] location checksum : a8dc636b (2833015659)
---- EOF Vdex Header Info ----
[DEBUG] [2284] 2017/12/20 20:28:09 (vdex.c:110 vdex_GetNextDexFileData) Processing first Dex file at offset:0x1c
[DEBUG] [2284] 2017/12/20 20:28:09 (dex.c:327 dex_isValidDexMagic) Dex version '037' detected
[ERROR] [2284] 2017/12/20 20:28:09 (vdex_backend_v10.c:451 vdex_process_v10) Unexpected checksum (c684b9f3 vs f02f3db5) - failed to unquicken Dex file
[DEBUG] [2284] 2017/12/20 20:28:09 (vdex.c:217 vdex_process) Took 31 ms to process Vdex file
[ERROR] [2284] 2017/12/20 20:28:09 (vdexExtractor.c:262 main) Failed to process Dex files - skipping 'wifi-service.vdex'
[INFO] 0 out of 1 Vdex files have been processed
[INFO] 0 Dex files have been extracted in total
[INFO] Extracted Dex files are available in '.'

files.zip

@anestisb

This comment has been minimized.

Show comment
Hide comment
@anestisb

anestisb Dec 20, 2017

Owner

Thanks for reporting. Will have a look at them tomorrow.

Owner

anestisb commented Dec 20, 2017

Thanks for reporting. Will have a look at them tomorrow.

@anestisb anestisb added the bug label Dec 20, 2017

@anestisb anestisb self-assigned this Dec 20, 2017

@Furniel

This comment has been minimized.

Show comment
Hide comment
@Furniel

Furniel Dec 21, 2017

Tested taimen apps deodexing, everything ok except some google apps(vdexExtractor fails with same type of error):

CalculatorGooglePrebuilt.vdex
Chrome.vdex
GoogleCamera.vdex
Maps.vdex
PrebuiltDeskClockGoogle.vdex
PrebuiltGmail.vdex
YouTube.vdex
services.vdex
wifi-service.vdex
GoogleContacts.vdex
GoogleDialer.vdex
Phonesky.vdex
SettingsGoogle.vdex

log.txt
vdexFiles

Furniel commented Dec 21, 2017

Tested taimen apps deodexing, everything ok except some google apps(vdexExtractor fails with same type of error):

CalculatorGooglePrebuilt.vdex
Chrome.vdex
GoogleCamera.vdex
Maps.vdex
PrebuiltDeskClockGoogle.vdex
PrebuiltGmail.vdex
YouTube.vdex
services.vdex
wifi-service.vdex
GoogleContacts.vdex
GoogleDialer.vdex
Phonesky.vdex
SettingsGoogle.vdex

log.txt
vdexFiles

@anestisb

This comment has been minimized.

Show comment
Hide comment
@anestisb

anestisb Dec 21, 2017

Owner

Thanks. The more affected files the easier to spot the pattern. Will start digging later today.

Owner

anestisb commented Dec 21, 2017

Thanks. The more affected files the easier to spot the pattern. Will start digging later today.

@anestisb

This comment has been minimized.

Show comment
Hide comment
@anestisb

anestisb Dec 25, 2017

Owner

Oh boy that was painful. I've concluded that the problem is in the way the mentioned bytecode files have been compiled. More specifically, for some reason the compiler backend that generated the oat / vdex files, is shuffling the class list.

If you run the following you will see that the class list is not alphabetically ordered by class descriptor as expected.

vdexExtractor -i ~/Desktop/vdex/GoogleDialer.vdex -f --no-unquicken --dis | grep "class #" | cut -d "'" -f2

Furthermore if you compare the previous list against the original dex file it has been generated from, you can further confirm the shuffling problem.

dexdump GoogleDialer/classes.dex | grep "Class descriptor  :" | cut -d "'" -f2

Finally, to even further verify that this is indeed the problem, you can run the libart oatdump utility against the problematic oat files to also notice the out-of-order issue. It should be also noted that the unquickening logic currently in AOSP master is also affected by the problematic cases.

So of course when the vdex extractor tool decompiles the out-of-ordered version, the checksum of the generated file will not match the original (ordered) one. However, all methods are properly decompiled since the QuickeningInfo offsets are correct for the out-of-order version.

The ART runtime has no problem dealing with the out-of-order version since the resources have been pre-verified and inserted into the loader's class list.

I think the issue is also related to the non-deterministic behaviour of the ART compiler that has been recently reported by the @thestinger while working the reproducible builds feature for the CopperheadOS.

So to wrap-up, there is nothing I can do from this tool's perspective to work with these shuffled files. Putting the classes back to the correct order will require a massive change of backward fix-ups in the extracted dex files. Something which I'm not planning to do.

Owner

anestisb commented Dec 25, 2017

Oh boy that was painful. I've concluded that the problem is in the way the mentioned bytecode files have been compiled. More specifically, for some reason the compiler backend that generated the oat / vdex files, is shuffling the class list.

If you run the following you will see that the class list is not alphabetically ordered by class descriptor as expected.

vdexExtractor -i ~/Desktop/vdex/GoogleDialer.vdex -f --no-unquicken --dis | grep "class #" | cut -d "'" -f2

Furthermore if you compare the previous list against the original dex file it has been generated from, you can further confirm the shuffling problem.

dexdump GoogleDialer/classes.dex | grep "Class descriptor  :" | cut -d "'" -f2

Finally, to even further verify that this is indeed the problem, you can run the libart oatdump utility against the problematic oat files to also notice the out-of-order issue. It should be also noted that the unquickening logic currently in AOSP master is also affected by the problematic cases.

So of course when the vdex extractor tool decompiles the out-of-ordered version, the checksum of the generated file will not match the original (ordered) one. However, all methods are properly decompiled since the QuickeningInfo offsets are correct for the out-of-order version.

The ART runtime has no problem dealing with the out-of-order version since the resources have been pre-verified and inserted into the loader's class list.

I think the issue is also related to the non-deterministic behaviour of the ART compiler that has been recently reported by the @thestinger while working the reproducible builds feature for the CopperheadOS.

So to wrap-up, there is nothing I can do from this tool's perspective to work with these shuffled files. Putting the classes back to the correct order will require a massive change of backward fix-ups in the extracted dex files. Something which I'm not planning to do.

@anestisb

This comment has been minimized.

Show comment
Hide comment
@anestisb

anestisb Dec 25, 2017

Owner

Btw I plan to raise this to the Google tracker since I don't believe it's an intended compiler behaviour.

Owner

anestisb commented Dec 25, 2017

Btw I plan to raise this to the Google tracker since I don't believe it's an intended compiler behaviour.

@anestisb anestisb added wontfix and removed bug labels Dec 25, 2017

@Furniel

This comment has been minimized.

Show comment
Hide comment
@Furniel

Furniel Dec 25, 2017

Thanks for your research. I tested android 8.1 roms deodexing for all other devices(angler, bullhead, marlin, ryu, sailfish, walleye) and they are all affected(in all cases, the list of files is the same - interesting why only this files are affected). Maybe this will help with your report to Google.

Furniel commented Dec 25, 2017

Thanks for your research. I tested android 8.1 roms deodexing for all other devices(angler, bullhead, marlin, ryu, sailfish, walleye) and they are all affected(in all cases, the list of files is the same - interesting why only this files are affected). Maybe this will help with your report to Google.

@ingbrzy

This comment has been minimized.

Show comment
Hide comment
@ingbrzy

ingbrzy Feb 18, 2018

Same errors for MIUI on Android 8.1...
services.jar and wifi-service.jar failed to deodex from vdex files.. they failed even with oatdump from API 27... What the hell is Google doing to the dev community...

btw what is services.jar.prof file for?

framework-miui.zip

ingbrzy commented Feb 18, 2018

Same errors for MIUI on Android 8.1...
services.jar and wifi-service.jar failed to deodex from vdex files.. they failed even with oatdump from API 27... What the hell is Google doing to the dev community...

btw what is services.jar.prof file for?

framework-miui.zip

@ale8530

This comment has been minimized.

Show comment
Hide comment
@ale8530

ale8530 Feb 19, 2018

@ingbrzy I also saw your zip, changed Google files again :(.

@anestisb what do you think about it?

ale8530 commented Feb 19, 2018

@ingbrzy I also saw your zip, changed Google files again :(.

@anestisb what do you think about it?

@anestisb

This comment has been minimized.

Show comment
Hide comment
@anestisb

anestisb Feb 19, 2018

Owner

As mentioned in #3 (comment) I can't really do anything. The reordering doesn't appear to be deterministic or based on something we can calculate and revert. But even if it was possible, the backwards repair process if I modify the dexClassDef list requires significant code additions.

I've examined a few samples from the list you've provided and it appears that the files can be properly decompiled (method implementations compared to original DEX), although the class list has a different order thus the checksum doesn't verify the output file.

A possible hack-around would be to iterate with the disassembler the decompiled file and if no errors occur (all offsets & index restrictions are ok) force a CRC repair and mark the file as valid. Or export an additional command-line arg to ignore CRC errors and force a repair. Then its user's responsibility to verify the sanity of the output file.

In general I would expect that if a quickened file is decompiled successfully it should be functionally equivalent to the original DEX file (ART runtime is running the same process). However, the only way to prove this assumption is via the original CRC. If we hack around it, that guarantee is over.

All suggestions are welcome.

PS: I'm temporarily re-opening the issue due to this recent discussion.

Owner

anestisb commented Feb 19, 2018

As mentioned in #3 (comment) I can't really do anything. The reordering doesn't appear to be deterministic or based on something we can calculate and revert. But even if it was possible, the backwards repair process if I modify the dexClassDef list requires significant code additions.

I've examined a few samples from the list you've provided and it appears that the files can be properly decompiled (method implementations compared to original DEX), although the class list has a different order thus the checksum doesn't verify the output file.

A possible hack-around would be to iterate with the disassembler the decompiled file and if no errors occur (all offsets & index restrictions are ok) force a CRC repair and mark the file as valid. Or export an additional command-line arg to ignore CRC errors and force a repair. Then its user's responsibility to verify the sanity of the output file.

In general I would expect that if a quickened file is decompiled successfully it should be functionally equivalent to the original DEX file (ART runtime is running the same process). However, the only way to prove this assumption is via the original CRC. If we hack around it, that guarantee is over.

All suggestions are welcome.

PS: I'm temporarily re-opening the issue due to this recent discussion.

@anestisb anestisb reopened this Feb 19, 2018

anestisb added a commit that referenced this issue Mar 4, 2018

Option to ignore CRC errors in decompiled Dex files
More info at #3

Signed-off-by: Anestis Bechtsoudis <anestis@census-labs.com>

@anestisb anestisb added upstream issue and removed wontfix labels Mar 4, 2018

@anestisb

This comment has been minimized.

Show comment
Hide comment
@anestisb

anestisb Mar 4, 2018

Owner

Decided to implement a CRC error ignore flag so users will not have to comment code to keep out-of-order decompiled files. For most cases it is expected that the decompiled Dex can be normally processed from decompilers and disassemblers (JEB2, dexdump, dex2jar, etc.).

Owner

anestisb commented Mar 4, 2018

Decided to implement a CRC error ignore flag so users will not have to comment code to keep out-of-order decompiled files. For most cases it is expected that the decompiled Dex can be normally processed from decompilers and disassemblers (JEB2, dexdump, dex2jar, etc.).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment