Extremely high memory consumption

[hamiltont]

As seen below, a 500KB apk uses up to 176MB (a 300x increase!!) of memory when decoding just the smali, without resources. This makes large scale analysis of many APKs very hard. I'm wondering if something can be done to reduce this. I found a related issue JesusFreke/smali#181 that says this should not be a problem, but that's not my experience. Posting on apktool project in hopes there can be some command line flag (like --low-mem) that trades speed for memory if a user needs that option.

This happens across all APKs I'm working with.

Note: To retest this on Mac, use /usr/bin/time -l apktool.... and note that the output of RSS is supposedly in bytes for OS X.

Information

1. Apktool Version (apktool -version) - 2.2.1
2. Operating System (Mac, Linux, Windows) - Linux
3. APK From? (Playstore, ROM, Other) - Playstore

Steps to Reproduce

[ec2-user@ip-10-0-233-18 test]$ ls -lh
total 556K
drwxrwxr-x 6 ec2-user ec2-user 4.0K Dec 26 19:11 az
-rw-rw-r-- 1 ec2-user ec2-user 551K Dec 26 19:08 az.apk
[ec2-user@ip-10-0-233-18 test]$ /usr/bin/time -v apktool d -f --no-res az.apk
I: Using Apktool 2.2.1 on az.apk
I: Copying raw resources...
I: Baksmaling classes.dex...
I: Copying assets and libs...
I: Copying unknown files...
I: Copying original files...
	Command being timed: "apktool d -f --no-res az.apk"
	User time (seconds): 2.84
	System time (seconds): 0.20
	Percent of CPU this job got: 13%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:22.37
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 176136
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 47144
	Voluntary context switches: 2273
	Involuntary context switches: 19
	Swaps: 0
	File system inputs: 0
	File system outputs: 15728
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

[hamiltont]

Perhaps someone can run apktool inside a java profiler and see where these allocations are coming from

Also, I can mention that the virtual memory is consistently in the >3GB range. While that does seem a bit high to me, it's not a "problem" in that it's not limiting anything. However, high RSS is an actual physical limit that reduces the ability to run apktool in parallel

[iBotPeaches]

You have to remember that the source APK is not the only package being loaded/utilized. The framework file (as of API 24) is about a 2.7M apk and that is loaded on every decode.

I did find a problem where we are reloading the same framework APK twice, which is highly inefficient.

The next major slowdown point comes from unknown files as it creates a new zip (apk) on the fly as it injects those unknown files. I'll have to look into editing the same file vs a tmp file.

Finally, a lot of string manipulation is hurting the runtime, according to the profiler.

I do have lots and lots of crashing bugs to fix though, so optimization has unfortunately fallen behind. I'll do my best for the easy solutions, but any fix requiring a good deal of rewriting will probably have to wait.