v1.8.0-beta1

About beta1

Four months have gone by with zero commits, so it is as good a time as any to release beta1. It extends identifier encoding and adds various forms of identifier mapping using map files similar to those produced by ProGuard. The v1.7.0-to-v1.8.0-beta1 transition was coded in a spree of two months; I brought the code to a releasable state, then I was too burned out to release and document the changes.

Intro

After years of avoiding the ever present issue of obfuscation, with v1.8.0 DexPatcher-tool gets a new system of transforms that is completely orthogonal to its legacy workings and allows tackling obfuscated code effectively. Rather than imposing 'the right way' of doing things, it supports several different ideas and workflows. To be most useful, these workflows need to be supported in future DexPatcher-gradle plugins, but creative users of the tool have already started to roll their own flows around the new features.

Because of the expansion in scope, v1.8.0 turned out to be a huge release, by far the biggest one ever since the project started. DexPatcher-tool is a 5 year old project. Still, in the move from v1.7.0 to v1.8.0, its source code generously more than doubled in size, an observation that is valid for code, comments, test harness, etc, and does not even count the substantial amount of existing code that was changed.

A change of this magnitude should not be released without real-world testing, and this release has had almost no testing done on it. An untested codebase this complex is most likely teeming with all sorts of nasty bugs. It is almost irresponsible of me to move it out of alpha this early, but my motives are:

• I want to communicate that the architecture of the v1.8.0 release as well as its command line interface should finally be stable. What follows should be bug fixes. Lots of them.
• Though I would like to, I do not think I will be able to embark on an app patching project to test this release in the field myself.
• By this token, testing will have to happen as users take up patching projects, and I do not want to delay future developments until feedback trickles in.
• I exercised extreme caution when modifying existing code, so likely no regressions where introduced in preexisting functionality.

The release is v1.8 instead of v2 because, from the perspective of the patch writer, the patching language has not changed; only better tooling for dealing with obfuscated code is provided. I am reserving v2 for when DexPatcher tool supports patching the actual bytecode inside of methods, instead of simply moving it around or replacing it.

In the months after I coded this release, some new nice-to-have features cropped up, but i decided against including them in v1.8.0. These are:

• A 'hollow' code transform that would create legal but mostly empty skeleton bytecode to be fed to decompilers to create skeleton classes that DexPatcher patch writers can use as templates.
• Switching to dexlib2 v2.4.0, which includes better support for Android 10.
• A convenience code transform to support defining Android 10's hiddenapi_class_data_item using Java annotations.

Obfuscation support

This release adds support for processing obfuscated code. It allows patching all bytecode in which illegal identifiers (eg: containing illegal characters or matching reserved words) or clashing identifiers (eg: clashing package and class names) are used.

Support is provided by optional code transforms that deal with identifier coding and mapping.

Identifier decoding transforms

Identifier decoding transforms decode instances of identifier codes found in bytecode. The following new command line options control these transforms:

    --decode-source           decode identifiers in source
    --decode-patches          decode identifiers in patches
    --decode-output           decode identifiers in output
    --code-marker <marker>    identifier code marker (default: '_$$_')
    --no-decode-errors        treat decode errors as warnings

Identifier codes can be used to define patches in legal Java that reference illegal identifiers in obfuscated bytecode. The patches can be decoded prior to patching (producing patching diagnostics with obfuscated names) or, alternatively, obfuscated identifiers present in the source bytecode can be encoded before patching and decoded after patching (producing patching diagnostics with encoded names).

A sample patching supposedly obfuscated code using the former strategy is available: see source and patch. Identifier code samples can be found here. The result of running the patched code is here.

Identifier encoding transforms

Identifier encoding transforms encode some names found in bytecode as instances of identifier codes. The following new command line options control these transforms:

    --code-marker <marker>    identifier code marker (default: '_$$_')

    --encode-source                encode identifiers in source
    --encode-map <file>            encode map file (repeatable option)
    --invert-encode-map            use inverse of encode map file
    --escape-non-ascii             escape non-ASCII characters
    --escape-non-latin             escape non-ASCII/Latin-1 characters
    --no-ascii-escapes             do not output ASCII escapes
    --no-code-point-escapes        do not output code point escapes
    --obfuscated-types <ptrn>      pattern for binary type names
                                   (form: '[<pkg>/...][<cls>$...]<cls>')
    --obfuscated-packages <ptrn>   pattern for non-qualified package names
    --obfuscated-classes <ptrn>    pattern for non-qualified class names
                                   (form: '[<cls>$...]<cls>')
    --obfuscated-members <ptrn>    pattern for member names
    --encode-all-classes           encode all class names
    --encode-obfuscated-packages   encode obfuscated package names
    --encode-obfuscated-classes    encode obfuscated class names
    --encode-obfuscated-members    encode obfuscated member names
    --encode-reserved-chars        encode names with reserved characters
    --encode-reserved-words        encode names matching reserved words
    --encode-class-hints           encode type hints in classes
    --encode-member-hints          encode type hints in members
    --encode-member-type           encode member type in members
    --no-identifier-type           do not encode identifier type
    --no-multiple-hints            only allow unique type hints
    --no-nested-classes            disable nested class processing
    --ignored-hint-type <type>     fully qualified name of type
                                   (use '-' to remove defaults)
                                   (repeatable option)
    --ignored-hint-types <ptrn>    pattern for binary type names
                                   (form: '[<pkg>/...][<cls>$...]<cls>')
    --encode-compilable            allow recompile of obfuscated code

Identifier encoding can be used to make obfuscated identifiers legal before patching and restore the obfuscation afterwards, with the patch code and patching diagnostics referring to the legal encoded names exclusively. Encoding can also help with code analysis by assigning descriptive encoded names to obfuscated names based on automatic bytecode analysis and/or identifier map files that the user provides (which would typically be generated using external AI deobfuscation tools). The encoded bytecode is typically fed to interactive viewers and decompilers, presenting the user with a unified encoded view of the codebase being worked on.

Several identifier encoding samples are available: by encode map (source, encode map, output), by pattern matching (source, output parts 1 and 2), by double encoding (source, output), and by character escaping (source, output). In these samples the encoding is not reverted after patching to allow dumping the generated bytecode for documentation purposes, but in real-world usage a decode transform would typically revert all encodings before the output is generated.

Identifier mapping tr...

v1.8.0-alpha4

Obfuscation support

This release adds support for processing obfuscated code. It allows patching all bytecode in which illegal identifiers (eg: containing illegal characters or matching reserved words) or clashing identifiers (eg: clashing package and class names) are used.

Support is provided by optional identifier renaming code transforms that decode instances of identifier codes found in bytecode. The following new command line options control these transforms:

    --decode-source           decode identifiers in source
    --decode-patches          decode identifiers in patches
    --decode-output           decode identifiers in output
    --code-marker <marker>    identifier code marker (default: '_$$_')
    --no-decode-errors        treat decoder errors as warnings

A small sample patching obfuscated code is available: see source and patch. Identifier code samples can be found here. And the result of running the patched code is here.

Anonymous class support

Special thanks to @andrewleech whose work inspired these changes.

This release adds support for processing anonymous classes. It allows easier patching of existing anonymous classes, and also defining new anonymous classes in patches without risk of them name-clashing with the existing ones.

Support is provided by optional type renaming code transforms that deanonymize and reanonymize classes. The following new command line options control these transforms:

    --deanon-source <plan>    deanonymize anonymous classes in source
    --deanon-patches <plan>   deanonymize anonymous classes in patches
    --reanon-source <plan>    reanonymize anonymous classes in source
    --reanon-patches <plan>   reanonymize anonymous classes in patches
    --reanon-output <plan>    reanonymize anonymous classes in output
    --no-anon-errors          treat anonymizer errors as warnings

The <plan> argument is a string that represents an anonymous class renaming template and is described here. Sample code involving anonymous classes is available: see source and patch.

Pre-transform stages

Code transforms run on-demand as pieces of bytecode are internally operated upon. This means that it is not required to keep models of the transformed bytecode in memory, and thus memory usage is low and mostly independent of bytecode size. However, as the transforms execute spread out in time, their log output will also be spread throughout the patching process, making the logs difficult to analyze.

To mitigate this effect, this release adds optional pre-transform stages to the processing pipeline that fully run a transform or set of transforms ahead of time to ensure their side effects occur before or during a particular stage, but never after. These stages are configured with the following new command line option:

    --pre-transform <set>     add pre-transform stages (default: 'out')
                              (<set>: 'none'|'dry'|'out'|'inout'|'all')

The added pre-transform stages depend on the chosen set:

• all: a pre-transform stage is added after each transform is applied. Each transform produces a completely separate log output. This is the slowest mode and is only recommended for debugging transform issues.
• inout: a pre-transform stage is added per input dex file after the set of transforms corresponding to each file is applied. a pre-transform stage is also added after the set of output transforms is applied to the result of the patching stage, but before the output dex file is written, if any. Each pre-transform stage produces a separate log output for each input and output dex (even if no output file is written). Within each stage, the logs of the corresponding transforms will be spread out. This is the fastest mode that will detect and log all input transform errors, including those errors that occur in input sections that will be discarded during the patch process.
• out: a single pre-transform stage is added to the pipeline, right after the set of output transforms is applied to the result of the patching stage, but before the output dex file is written, if any. All transform side effects and their output will be spread throughout the patching process, but they are guaranteed to occur before the output dex file is written. This is the fastest mode that will detect all errors in the tool's processing pipeline before the output file is created, and thus is the fastest mode that guarantees that no output file is generated in case of errors. This is the default mode.
• dry: a single pre-transform stage is added to the pipeline, right after the set of output transforms is applied to the result of the patching stage, but only if a dry run is detected (ie: no output dex file will be written). Dry runs may be explicitly requested, or a normal run may become dry if an error is detected before the output file is created. Note that transform errors may still occur after that point. This is the fastest mode that will detect all errors in the tool's processing pipeline that would occur if an output dex file were generated, even if it is not. No mode is faster for successful, non-dry runs. This mode is recommended for use with the DexPatcher Gradle plugins.
• none: no pre-transform stages are added to the pipeline. Transform errors may be masked during dry runs, as large sections of bytecode may not be fully processed during those runs. Conversely, transform errors may occur after the output dex file is written during non-dry runs. This mode is recommended for embedded uses of the DexPatcher tool where only a success/failure exit status is required.

Using code transforms with the DexPatcher Gradle plugins

The DexPatcher Gradle plugins have not yet been updated to support the DexPatcher tool's new command line options. However, code transforms can still be used by manually adding the required command line options to the extraArgs properties of Gradle tasks of type lanchon.dexpatcher.gradle.tasks.DexpatcherTask.

Other changes

• Retention policy of DexPatcher annotations changed from CLASS to RUNTIME. This change is required for compatibility with the new d8 dexer. Contrary to dx, this dexer removes annotations with CLASS retention, presumably because it is expected that all bytecode processing happens before dexing. Note that DexPatcher always strips its annotations during processing, so this change does not affect the bytecode produced by the tool.
• This release fixes a bug existing since the introduction of cross-class edits in v1.5.0, which involves arrays of the type being rewritten. This bug stems from a sample code bug in the documentation of smali's dexlib2 library. More info here.
• Minor command line changes:
  • The -d option is now accepted as a short for --debug.
  • The tool will no longer print usage information when it cannot parse the command line.
• Client API change:
  • The builder pattern is now used to construct core Context objects.
• Changes to the build system:
  • Reproducible builds.
  • Gradle updated from v2 to v6.
  • Switched to the java-library plugin.
  • Shadow plugin updated from v1 to v5.
  • Deduplication of resources.
  • About info moved to META-INF/about.
  • Rewritten tests:
    • Locally installed tools are no longer required to run tests:
      • Required tools are dynamically downloaded.
      • All tests now run in build servers.
    • Switched dexer from dx to d8.
    • Improved output of test results.
  • New build diagnostics.
  • Build code sharing with other DexPatcher projects.
  • Signed artifacts.
  • Publish artifacts to local and remote repositories.
  • Compatible with Sonatype and JitPack.
  • Added Travis CI setup with full testing.

$ dexpatcher --help
DexPatcher version 1.8.0-alpha4 by Lanchon (https://dexpatcher.github.io/)

usage: dexpatcher [<option> ...] [--output <patched-dex-or-dir>]
                  <source-dex-apk-or-dir> [<patch-dex-apk-or-dir> ...]

main options:
 -?,--help                    print this help message and exit
 -a,--api-level <n>           android api level (default: auto-detect)
    --annotations <package>   package name of DexPatcher annotations
                              (default: 'lanchon.dexpatcher.annotation')
 -d,--debug                   output debugging information
    --dry-run                 do not write output files (much faster)
 -J,--multi-dex-jobs <n>      multi-dex thread count (implies: -m -M)
                              (default: available processors up to 4)
 -m,--multi-dex               enable multi-dex support
 -M,--multi-dex-threaded      multi-threaded multi-dex (implies: -m)
    --max-dex-pool-size <n>   maximum size of dex pools (default: 65536)
    --no-auto-ignore          no trivial default constructor auto-ignore
 -o,--output <dex-or-dir>     name of output file or directory
 -p,--path          ...

v1.7.0

Changes:

• Bumped dexlib2 version to 2.3.4, adding partial support for Android 10 (API level 29).
• Improved detection and diagnostics of suspicious code changes.
• Removed support for the DexTag type (deprecated in DexPatcher v1.1.0) and its associated --compat-dextag command-line option.
• Changed the default value of the targetClass annotation element from Void.class to void.class. (This potentially braking change should not be an issue in practice given that Java compilers typically do not embed the default value of elements.)

$ dexpatcher --help
DexPatcher version 1.7.0 by Lanchon (https://dexpatcher.github.io/)
usage: dexpatcher [<option> ...] [--output <patched-dex-or-dir>]
                  <source-dex-apk-or-dir> [<patch-dex-apk-or-dir> ...]
 -?,--help                    print this help message and exit
 -a,--api-level <n>           android api level (default: auto-detect)
    --annotations <package>   package name of DexPatcher annotations
                              (default: 'lanchon.dexpatcher.annotation')
    --debug                   output debugging information
    --dry-run                 do not write output files (much faster)
 -J,--multi-dex-jobs <n>      multi-dex thread count (implies: -m -M)
                              (default: available processors up to 4)
 -m,--multi-dex               enable multi-dex support
 -M,--multi-dex-threaded      multi-threaded multi-dex (implies: -m)
    --max-dex-pool-size <n>   maximum size of dex pools (default: 65536)
    --no-auto-ignore          no trivial default constructor auto-ignore
 -o,--output <dex-or-dir>     name of output file or directory
 -p,--path                    output relative paths of source code files
 -P,--path-root <root>        output absolute paths of source code files
 -q,--quiet                   do not output warnings
    --stats                   output timing statistics
 -v,--verbose                 output extra information
    --version                 print version information and exit

The official soundtrack for DexPatcher v1.7.0 is James Grant and Jody Wisternoff's Anjunadeep 200.

v1.6.3

Changes:

• Added DexAction.NONE, supporting implicit handling of static constructors when using default actions. More information here. This value is being added to help provide backwards compatibility in the face of semantic changes that the implementation of the future @DexCheck tag will bring to existing tags.
• Bumped dexlib2 version to 2.2.7.

v1.6.2

Changes:

• Bumped dexlib2 version to 2.2.5, adding support for Android 8 and 9 (API levels 26 through 28).

DexPatcher v1.6.2 is hereby provided without guarantees of any kind, as it is being released in between glasses of white wine from the sunny beaches of Barcelona. Cheers!

Note: DexPatcher v1.6.1 was never officially released due to some showstopper bugs in dexlib2 v2.2.4.

Update: The very minor changes from dexlib2 v2.2.5 to v2.2.6 do not affect DexPatcher in any way, so a DexPatcher update is not warranted.

v1.6.0

Changes:

• Unified processing of direct and virtual methods and of static and instance fields. Private methods (which are direct) and virtual methods can now target each other. More information here.
• Auxiliary methods added by actions such as @DexWrap and @DexAppend are now always private, even when the targeted methods are virtual. Also, the naming of such methods has changed.
• A dot can now be prepended to the fully qualified name of a targeted type or package, allowing targeting of the default package and types therein by fully qualified name.
• Every possible legal identifier within dex files should now be correctly handled regardless of how weird they might be (to the extent that dexlib2 supports such weirdness).
• Improved detection of erroneous code changes and diagnostics of suspicious changes.
• Logging of package actions now use friendly type names instead of type descriptors.
• Extensive refactoring and code cleanup.
• Bumped dexlib2 version to 2.2.2.

The official soundtrack for DexPatcher v1.6.0 is Kidnap Kid's Anjunadeep 162.

v1.5.0

Changes:

• Added the ability to rename classes by rewriting all references to their type within their declarations and code to account for the type change. This allowed the implementation of the following features which are generally useful, but especially so when dealing with obfuscated code:
  • Cross-class @DexEdit with class rewrite. More information here.
  • Cross-class @DexReplace with class rewrite. More information here.
• Deprecated the onlyEditMembers element and replaced it with contentOnly. The name was changed to allow extending the usage of the element to other tags besides @DexEdit and to other items besides classes. The new tool accepts both the old and new names of the element, so binary patches compiled for an older version of the tool will continue to apply using the new version. However, recompiling the patches will require adapting them to the name change.

The official soundtrack for DexPatcher v1.5.0 is Jody Wisternoff and James Grant's Anjunadeep 148.

v1.4.1

This release was coded with one less tendon in my body.

Changes:

• Added the --no-auto-ignore command line option to disable the implicit ignore of trivial default constructors. This option complements the *.importSymbols options of the Gradle plugins to ensure that all symbols potentially used in the patch are also declared and explicitly tagged there, trivial default constructors included. The motivation for this functionality is the planned @DexCheck tag.
• Added the -P command line option as a short alias for --path-root.
• Bumped multidexlib2 version to 2.2.1.r2.

$ dexpatcher --help
DexPatcher Version 1.4.1 by Lanchon
           https://dexpatcher.github.io/
usage: dexpatcher [<option> ...] [--output <patched-dex-or-dir>]
                  <source-dex-apk-or-dir> [<patch-dex-apk-or-dir> ...]
 -?,--help                    print this help message and exit
 -a,--api-level <n>           android api level (default: auto-detect)
    --annotations <package>   package name of DexPatcher annotations
                              (default: 'lanchon.dexpatcher.annotation')
    --compat-dextag           enable support for the deprecated DexTag
    --debug                   output debugging information
    --dry-run                 do not write output files (much faster)
 -J,--multi-dex-jobs <n>      multi-dex thread count (implies: -m -M)
                              (default: available processors up to 4)
 -M,--multi-dex-threaded      multi-threaded multi-dex (implies: -m)
 -m,--multi-dex               enable multi-dex support
    --max-dex-pool-size <n>   maximum size of dex pools (default: 65536)
    --no-auto-ignore          no trivial default constructor auto-ignore
 -o,--output <dex-or-dir>     name of output file or directory
 -P,--path-root <root>        output absolute paths of source code files
 -p,--path                    output relative paths of source code files
 -q,--quiet                   do not output warnings
    --stats                   output timing statistics
 -v,--verbose                 output extra information
    --version                 print version information and exit

v1.4.0

This release was coded at 10,000 feet.

Changes:

• Added @DexPrepend and @DexAppend, which are tags that add code to existing methods that return void. More information here.
• Added the ability to implicitly handle static constructors. More information here.
• Updated the tool documentation.

The official soundtrack for DexPatcher v1.4.0 is Todd Terje's live performance at Oya Festival 2014.

v1.3.1

Changes:

• Added the ability to implicitly ignore trivial default constructors. More information here.
• Updated the tool documentation.