A few optimizations

[bconstanzo]

I've been profiling and testing a few changes that have a major impact on the Windows performance of ALEAPP.

The reported time goes down by about 3x on my benchmarks. The changes don't affect behavior as far as I could test, the results are consistent between runs, it just goes faster.

Mainly this is achived by placing a few caches in place (functools.lru_cache with maxsize set to None), and avoiding duplicating work (fnmatch.fnmatch usage is changed by a "deconstructed" version of it). Tried to keep the code as simple and readable as it could be, while netting some nice speedups.

There's also a small bit that covers specifically snappy decompression, where I managed to improve the algorithm just a bit. Enough to get a 4-5% performance increase, without resorting to overly complicated code.

Methodology:

• Ran ALEAPP against Josh Hickman's Android 12 test image.
• Checked reported run time.
• Profiled everything and analyzed the call graphs and functions (with cProfile, gprof2dot, and the amazing line_profiler).
• Got millions of files and hundreds of gigabytes of storage space used up on my disk.
• Found out a few spots where things could be improved.
• Rinse and repeat.

[jijames]

@bconstanzo do you have the tests you used for this? Just for reference.
@abrignoni looks great.

Real results without usagestats on Linux:
Before
real 0m24.984s
user 0m9.424s
sys 0m0.541s

After
real 0m22.285s
user 0m9.364s
sys 0m0.469s

[bconstanzo]

Under Windows 10 Home 64 bit, I set up a virtual env with Python 3.10.3 and all the dependencies from the requirements.txt file.

Then it was just cloning ALEAPP on sunday night, and ran:

python aleapp.py -t tar -i path_to_image -o path_for_reports

I have a Ryzen 4600H, 24GB RAM and ran against a SATA HDD just for going with the worst case scenario, though I also tested against an nvme drive and it ran just as fast.

For profiling I'd run it as python -m cProfile -o profile_file.pstats aleap.py... or add the @Profile decorator and go with a kernprof -lv aleapp.py ....

The timings and speedups I commented about in the commits are based on what I saw during the profile runs, which under cProfile were running about half as fast as it'd run normally, and the reported time by the tool.

Right now I just benchmarked with a very simple script:

import time
import subprocess

t0 = time.perf_counter()
p = subprocess.run(
  r'python aleapp.py -t tar -i "D:\Test\xLEAPP\test_data\Magnet Acquire\Android 12 - Data.tar" -o "D:\Test\xLEAPP\output\aleapp-magnet"',
  shell=True
)
t1 = time.perf_counter()

print()
print()
print(f"Time taken: {t1 - t0}")

And it gave me 477 seconds for a clone of ALEAPP (which the tool reported as 6 minutes and 34 seconds) and 202 seconds for the patched version (which the tool reported as 2 minutes sharp). That is just shy of 2.4x faster, and there clearly is something off with the reported time.