Use WALA master instead of custom version

[reddr]

LibScout uses JOANA's WALA fork that adopted and improved ScanDroid's dex frontend to allow Android app analyses. At the time of integration, the state was still immature and a lot of integral parts were missing, i.e. multi-dex support. During LibScout's development, some issues have been fixed and workaround patches have been applied to a custom version (lib/wala.jar) that is statically linked during compilation time.

Relying on a custom version is definitively sub-optimal, both in terms of integration and maintainability.
To be able to move from this old and customized WALA to upstream WALA it needs to be checked, whether all features currently used in LibScout (through custom patches etc.) are supported in the WALA master.

I already checked some of LibScout's patches against WALA master. I commented on the following (important) ones that are either unresolved or should already work with the latest WALA version:

• [wala_dex_dexfilemodule_multidex_support.patch]
  Adds multi-dex support to WALA dex frontend that was based on dexlib1
  [Status: resolved] WALA master uses dexlib2 that has builtin support for multi-dex. This would also solve Multi-dex APKs not completely processed #19 .

• [wala_cha_workaround_unresolvable_superclasses.patch]
  WALA's CHA builder discards classes when their superclasses cannot be resolved. This affects library classes that extend other library classes not being in the classpath. However, the same library class might be resolved within an app where all dependencies are statically linked. This is a workaround patch that forces the CHA builder to keep such classes by changing it's superclass to java.lang.Object to allow code reuse detection.
  [Status: resolved] Addressed in issue Allow WALA to support missing superclasses? wala/WALA#322. Instead of rewriting to Object, WALA's patch creates a synthetic PhantomSuperClass.

• [wala_add_check_synthethic_methods.patch]
  Addition of isMethodSynthetic call as a workaround (see also Undeclared Method isMethodSynthetic being called in LibraryProfiler.java 155 and many other places #16). This became necessary since a proper implementation of isSynthetic for IMethod subclasses (DexIMethod, ShrikeBTMethod,..) caused some unintended side-effects. Did not dig deeper, as this affects all IMethod subclasses.
  [Status: not implemented] WALA master still returns false for isSynthetic calls.

• [wala_dex_ssa_fix_for_methods_without_parameters.patch]
  WALA creates an empty IR for static initializer methods without arguments. This patches fixes the SSA Builder.
  [Status: probably not, needs to be checked]

I'm going to test WALA master and create issues for the open problems.
CC @eighthave

[eighthave]

Thanks for doing this! It'll make Debian integration a lot easier. Let us know when you know which WALA lib you need. If the JOANA library is plain Java, it should be relatively easy to package.

[reddr]

Time for an update:

• As stated above, WALA master has already multi-dex support and also allows to keep unresolvable superclasses in the CHA.

• My pull request for adding isSynthetic checks on IMethod/IClass has recently been merged into master Impl of IMethod.is(Wala)Synthetic and IClass.isSynthetic wala/WALA#359

• While testing WALA master, I encountered two bugs and proposed patches (Fixes #360 Exception in AndroidAnalysisScope  wala/WALA#361 and Fixes IllegalStateException wala/WALA#358). They have also been merged into master.

• As for the problem with an empty IR for clinit methods, good news is that this has been resolved already some time ago, so no need for additional action.

This should be everything necessary to port LibScout to WALA master. If everything works with the provided patches, I'll request a new WALA release that can then be added to LibScout's gradle config.

[reddr]

After some initial testing, it seems that WALA's PhantomClass workaround for unresolvable classes (e.g. classes used as superclass that are not statically linked in the app code) throws exceptions in certain situations (see discussion in wala/WALA#335). This needs to be resolved somehow before we can proceed.

[reddr]

We are getting closer, I just proposed a patch for building the ClassHierarchy that does not dismiss classes when superclasses are not statically resolvable (see wala/WALA#377).

Besides that, there is a minor issue with the multi-dex support. Although, it is supported in the current WALA version, there is currently no helper function to conveniently setup an AnalysisScope for multi-dex apks. I pushed a PR as well (wala/WALA#378), but, in theory, this could also be added as workaround in LibScout's code base.

[reddr]

So finally I upgraded the code base to the current WALA/dexlib2 master (LibScout version 2.3.0)!

Still, I have to provide pre-compiled binaries, as not all patches are merged yet.
The current status is as follows:

• Introduce ClassHierarchyFactory.makeWithRoot wala/WALA#377 is successfully merged (don't dismiss classes with unresolvable superclasses)
• Regarding the WALA patch for AnalysisScope multi-dex support (Fix Android scope to work with multi-dex apks wala/WALA#378), it turned out that for a proper solution a tiny change in the dexlib2 code is required. I opened a PR (make opcodes nullable in loadDexContainer JesusFreke/smali#673), but the dexlib development apparently has slowed down significantly, thus it's unsure when this is going to be merged.

Anyways, I patched the respective code bases and provide the pre-compiled binaries to speed up things. As soon as there is progress in the PRs, I will update LibScout accordingly.

[eighthave]

is there a tagged release of WALA that will work with LibScout yet? Once there is, I'll update the Debian package.

[reddr]

Not yet, as soon as the multi-dex support PR is merged, I'll ask for a tagged release.

[reddr]

I updated LibScout to the new smali v2.2.6.

In the meantime, the remaining PR got merged into WALA master. Now waiting for the new release to update LibScout accordingly.

[reddr]

We are finally there. LibScout 2.3.1 fetches the new WALA libs (v1.5.1) from maven central!

[eighthave]

excellent! I'll upload them both to Debian hopefully today. Can you recommend a simple, command line test case to add to the package's autopkgtest?

[reddr]

Hmm, if it is compiled with gradle you can execute the jar without args. Then a usage message is shown with an exit code 1.

For a functional test, it would have to be set up to profile a single lib version and then used in match mode on a small app test includes it and finally check the json output.

The single library profile could also be pre-packaged, such that only the match mode would have to be run.

For some test, LibScout could only profile a single lib version and you test whether a .libv file is output with a specific size.

But all these tests, require some kind of setup.

[eighthave]

Almost done with the libscout package update! FYI, here you can follow the progress of the new packages I made to get libscout in: * https://tracker.debian.org/pkg/libscout * https://tracker.debian.org/pkg/wala * https://tracker.debian.org/pkg/cava * https://tracker.debian.org/pkg/libsmali-java