Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
taimen android 8.1 deodexing issues #3
Tested taimen-opm1.171019.011-factory-2df1c1cb deodexing and got two errors during framework deodexing:
Tested taimen apps deodexing, everything ok except some google apps(vdexExtractor fails with same type of error):
Oh boy that was painful. I've concluded that the problem is in the way the mentioned bytecode files have been compiled. More specifically, for some reason the compiler backend that generated the oat / vdex files, is shuffling the class list.
If you run the following you will see that the class list is not alphabetically ordered by class descriptor as expected.
Furthermore if you compare the previous list against the original dex file it has been generated from, you can further confirm the shuffling problem.
Finally, to even further verify that this is indeed the problem, you can run the libart oatdump utility against the problematic oat files to also notice the out-of-order issue. It should be also noted that the unquickening logic currently in AOSP master is also affected by the problematic cases.
So of course when the vdex extractor tool decompiles the out-of-ordered version, the checksum of the generated file will not match the original (ordered) one. However, all methods are properly decompiled since the QuickeningInfo offsets are correct for the out-of-order version.
The ART runtime has no problem dealing with the out-of-order version since the resources have been pre-verified and inserted into the loader's class list.
I think the issue is also related to the non-deterministic behaviour of the ART compiler that has been recently reported by the @thestinger while working the reproducible builds feature for the CopperheadOS.
So to wrap-up, there is nothing I can do from this tool's perspective to work with these shuffled files. Putting the classes back to the correct order will require a massive change of backward fix-ups in the extracted dex files. Something which I'm not planning to do.
Thanks for your research. I tested android 8.1 roms deodexing for all other devices(angler, bullhead, marlin, ryu, sailfish, walleye) and they are all affected(in all cases, the list of files is the same - interesting why only this files are affected). Maybe this will help with your report to Google.
As mentioned in #3 (comment) I can't really do anything. The reordering doesn't appear to be deterministic or based on something we can calculate and revert. But even if it was possible, the backwards repair process if I modify the dexClassDef list requires significant code additions.
I've examined a few samples from the list you've provided and it appears that the files can be properly decompiled (method implementations compared to original DEX), although the class list has a different order thus the checksum doesn't verify the output file.
A possible hack-around would be to iterate with the disassembler the decompiled file and if no errors occur (all offsets & index restrictions are ok) force a CRC repair and mark the file as valid. Or export an additional command-line arg to ignore CRC errors and force a repair. Then its user's responsibility to verify the sanity of the output file.
In general I would expect that if a quickened file is decompiled successfully it should be functionally equivalent to the original DEX file (ART runtime is running the same process). However, the only way to prove this assumption is via the original CRC. If we hack around it, that guarantee is over.
All suggestions are welcome.
PS: I'm temporarily re-opening the issue due to this recent discussion.
referenced this issue
Feb 20, 2018
added a commit
Mar 4, 2018
Decided to implement a CRC error ignore flag so users will not have to comment code to keep out-of-order decompiled files. For most cases it is expected that the decompiled Dex can be normally processed from decompilers and disassemblers (JEB2, dexdump, dex2jar, etc.).