New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to aapt2 #1689

Closed
iBotPeaches opened this Issue Dec 13, 2017 · 10 comments

Comments

Projects
None yet
2 participants
@iBotPeaches
Owner

iBotPeaches commented Dec 13, 2017

It is time. There are many things no longer working in aapt1 and this will only get worse as time goes on.

Sample: https://github.com/aosp-mirror/platform_frameworks_base/tree/master/libs/androidfw/tests/data/feature

@iBotPeaches iBotPeaches self-assigned this Dec 13, 2017

@iBotPeaches

This comment has been minimized.

Owner

iBotPeaches commented Dec 13, 2017

For example. A new application with 0x80 pkg id.

W: Package id out of range
W: A/ResourceType(40612): Package id out of range
@iBotPeaches

This comment has been minimized.

Owner

iBotPeaches commented Jan 28, 2018

Patches required for aapt2

  • MIUI rules (godzilla UI, etc). Still legacy with no announcement to drop, so must maintain
  • forced-package-id so we can force system apps to non-standard IDs
  • Allow uppercase resources
  • Add support for MCC/MNC
  • libc++ should be static built
  • Skip functions that modify apk for compat purposes, Apktool keeps as-is
  • Remove host dependency on AndroidManifest.xml
  • Ignore character checks on filenames
  • Remove "do"|"if" from reserved name
  • "attr" type does not need to be first in resource table, remove assertion
@iBotPeaches

This comment has been minimized.

Owner

iBotPeaches commented Jan 31, 2018

Preventing a complete utter disaster. Apktool will have both aapt1/aapt2 in it for a few releases. We will release a version with aapt2 inside, but keep the default at v1.

The next release will move that default to v2.

3 releases from that point, v1 will be removed and only available (with no confirmations) via --aapt flag.

@TheHighriser

This comment has been minimized.

TheHighriser commented Feb 1, 2018

That's great news! So when you talk about next release, you mean 2.3.2? Or are we talking about a major release like v3?

@iBotPeaches

This comment has been minimized.

Owner

iBotPeaches commented Feb 1, 2018

@TheHighriser My goal is that aapt2 will do everything the current Apktool can do, so no need for a breaking change version increase like v3, so it will be probably be v2.4.0.

My goal is a release every 3 months, if we approach that 3 month deadline and aapt2 is not ready. I will slice a v2.3.2 release with the pending changes unrelated to aapt2 and delay aapt2 until the next release.

@iBotPeaches

This comment has been minimized.

Owner

iBotPeaches commented Feb 13, 2018

Some good news is that aapt2 has some parameters that resolve some underlying issues in apktool.

 --package-id arg                                  Specify the package ID to use for this app. Must be greater or equal to
                                                   0x7f and can't be used with --static-lib or --shared-lib.

It seems we can force the pkg-id during a build instead of the hacky solutions we hacked into aapt1.

 --rename-manifest-package arg                     Renames the package in AndroidManifest.xml.
 --rename-instrumentation-target-package arg       Changes the name of the target package for instrumentation. Most useful
                                                   when used in conjunction with --rename-manifest-package.

We can rename the package during the build of aapt, but now we have an additional property. So it seems the manual "rewriting" of the AndroidManifest.xml is no longer needed in apktool. So we can remove that portion.

 --no-auto-version                                 Disables automatic style and layout SDK versioning.
 --no-version-vectors                              Disables automatic versioning of vector drawables. Use this only
                                                   when building with vector drawable support library.
 --no-version-transitions                          Disables automatic versioning of transition resources. Use this only
                                                   when building with transition support library.
 --no-resource-deduping                            Disables automatic deduping of resources with

Tons of properties that stop the automatic changing of resources. All of these should be disabled.

 --enable-sparse-encoding                          Enables encoding sparse entries using a binary search tree.
                                                   This decreases APK size at the cost of resource retrieval performance.

We can detect a "sparse" application, so we should add an additional apktool.yml flag, so we can match the original as best as possible. IE sparse if the application is sparse.

 --stable-ids arg                                  File containing a list of name to ID mapping.
 --emit-ids arg                                    Emit a file at the given path with a list of name to ID mappings,
                                                   suitable for use with --stable-ids.

Currently we use the public.xml file to keep resource ids. I don't know the format of the file that --stable-ids takes, but perhaps we can leverage this instead of public.xml.

More and more it seems switching to aapt2 while keeping aapt1 functional is difficult.

@iBotPeaches

This comment has been minimized.

Owner

iBotPeaches commented Feb 14, 2018

Changes not ported for various reasons to aapt2:

aapt2 has been successfully built on Linux & Mac with the needed changes. https://github.com/iBotPeaches/platform_frameworks_base/commits/apktool_8.1.0

@iBotPeaches

This comment has been minimized.

Owner

iBotPeaches commented Feb 14, 2018

aapt2 compile [options] -o arg files...

Options:
 -o arg                                            Output path
 --dir arg                                         Directory to scan for resources
 --pseudo-localize                                 Generate resources for pseudo-locales (en-XA and ar-XB)
 --no-crunch                                       Disables PNG processing
 --legacy                                          Treat errors that used to be valid in AAPT as warnings
 -v                                                Enables verbose logging
 -h                                                Displays this help menu
aapt2 link [options] -o arg --manifest arg files...

Options:
 -o arg                                            Output path.
 --manifest arg                                    Path to the Android manifest to build.
 -I arg                                            Adds an Android APK to link against.
 -A arg                                            An assets directory to include in the APK. These are unprocessed.
 -R arg                                            Compilation unit to link, using `overlay` semantics.
                                                   The last conflicting resource given takes precedence.
 --package-id arg                                  Specify the package ID to use for this app. Must be greater or equal to
                                                   0x7f and can't be used with --static-lib or --shared-lib.
 --java arg                                        Directory in which to generate R.java.
 --proguard arg                                    Output file for generated Proguard rules.
 --proguard-main-dex arg                           Output file for generated Proguard rules for the main dex.
 --no-auto-version                                 Disables automatic style and layout SDK versioning.
 --no-version-vectors                              Disables automatic versioning of vector drawables. Use this only
                                                   when building with vector drawable support library.
 --no-version-transitions                          Disables automatic versioning of transition resources. Use this only
                                                   when building with transition support library.
 --no-resource-deduping                            Disables automatic deduping of resources with
                                                   identical values across compatible configurations.
 --enable-sparse-encoding                          Enables encoding sparse entries using a binary search tree.
                                                   This decreases APK size at the cost of resource retrieval performance.
 -x                                                Legacy flag that specifies to use the package identifier 0x01.
 -z                                                Require localization of strings marked 'suggested'.
 -c arg                                            Comma separated list of configurations to include. The default
                                                   is all configurations.
 --preferred-density arg                           Selects the closest matching density and strips out all others.
 --product arg                                     Comma separated list of product names to keep
 --output-to-dir                                   Outputs the APK contents to a directory specified by -o.
 --no-xml-namespaces                               Removes XML namespace prefix and URI information from
                                                   AndroidManifest.xml and XML binaries in res/*.
 --min-sdk-version arg                             Default minimum SDK version to use for AndroidManifest.xml.
 --target-sdk-version arg                          Default target SDK version to use for AndroidManifest.xml.
 --version-code arg                                Version code (integer) to inject into the AndroidManifest.xml if none is
                                                   present.
 --version-name arg                                Version name to inject into the AndroidManifest.xml if none is present.
 --shared-lib                                      Generates a shared Android runtime library.
 --static-lib                                      Generate a static Android library.
 --no-static-lib-packages                          Merge all library resources under the app's package.
 --non-final-ids                                   Generates R.java without the final modifier. This is implied when
                                                   --static-lib is specified.
 --stable-ids arg                                  File containing a list of name to ID mapping.
 --emit-ids arg                                    Emit a file at the given path with a list of name to ID mappings,
                                                   suitable for use with --stable-ids.
 --private-symbols arg                             Package name to use when generating R.java for private symbols.
                                                   If not specified, public and private symbols will use the application's
                                                   package name.
 --custom-package arg                              Custom Java package under which to generate R.java.
 --extra-packages arg                              Generate the same R.java but with different package names.
 --add-javadoc-annotation arg                      Adds a JavaDoc annotation to all generated Java classes.
 --output-text-symbols arg                         Generates a text file containing the resource symbols of the R class in
                                                   the specified folder.
 --auto-add-overlay                                Allows the addition of new resources in overlays without
                                                   <add-resource> tags.
 --rename-manifest-package arg                     Renames the package in AndroidManifest.xml.
 --rename-instrumentation-target-package arg       Changes the name of the target package for instrumentation. Most useful
                                                   when used in conjunction with --rename-manifest-package.
 -0 arg                                            File extensions not to compress.
 --split arg                                       Split resources matching a set of configs out to a Split APK.
                                                   Syntax: path/to/output.apk:<config>[,<config>[...]].
                                                   On Windows, use a semicolon ';' separator instead.
 -v                                                Enables verbose logging.
 -h                                                Displays this help menu

Tired of looking for this information. Sorry to anyone subscribed to this issue, feel free to unsubscribe. Using as my personal memory bank as I jump between computers.

@iBotPeaches

This comment has been minimized.

Owner

iBotPeaches commented Feb 14, 2018

aapt 
p 
-v
--forced-package-id 127
--min-sdk-version 11
--target-sdk-version 22
--version-code 1
--version-name 1.0
--no-version-vectors
-F /var/folders/hs/tpnx1lpj03l_zxwcws1pklkc0000gp/T/APKTOOL6913196265868759183.tmp
-0 arsc
-I /Users/connortumbleson/Library/apktool/framework/1.apk
-S /Users/connortumbleson/Desktop/Apktool/Bugs/test/res
-M /Users/connortumbleson/Desktop/Apktool/Bugs/test/AndroidManifest.xml

Example execution of aapt1. Confusing or unknown parameters are as follows:

p = package
F = apk file
S = resource sources (list), multiple allowed
M = AndroidManifest.xml file
0 = Stored with no compression.

This maps in aapt2 for the compile stage to

➜  Desktop aapt2 compile --dir /Users/connortumbleson/Desktop/aapt2training/app-debug/res/ -o resources.arsc.flat -v                                                                                     
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-mdpi/a.png: note: compiling PNG.
note:  paletteSize=350 alphaPaletteSize=133 maxGrayDeviation=170 grayScale=false.
note: encoding PNG RGBA.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-mdpi/a.png: note: original PNG is smaller than crunched PNG, using original.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-mdpi/a.png: note: legacy=2329 new=2096.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-hdpi/a.png: note: compiling PNG.
note:  paletteSize=452 alphaPaletteSize=134 maxGrayDeviation=255 grayScale=false.
note: encoding PNG RGBA.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-hdpi/a.png: note: original PNG is smaller than crunched PNG, using original.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-hdpi/a.png: note: legacy=3322 new=3056.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-xxxhdpi/a.png: note: compiling PNG.
note:  paletteSize=1164 alphaPaletteSize=410 maxGrayDeviation=255 grayScale=false.
note: encoding PNG RGBA.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-xxxhdpi/a.png: note: legacy=8946 new=8946.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-xxhdpi/a.png: note: compiling PNG.
note:  paletteSize=860 alphaPaletteSize=307 maxGrayDeviation=255 grayScale=false.
note: encoding PNG RGBA.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-xxhdpi/a.png: note: original PNG is smaller than crunched PNG, using original.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-xxhdpi/a.png: note: legacy=6554 new=6464.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-xhdpi/a.png: note: compiling PNG.
note:  paletteSize=696 alphaPaletteSize=272 maxGrayDeviation=170 grayScale=false.
note: encoding PNG RGBA.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-xhdpi/a.png: note: original PNG is smaller than crunched PNG, using original.
/Users/connortumbleson/Desktop/aapt2training/app-debug/res/mipmap-xhdpi/a.png: note: legacy=4676 new=4569.

➜  Desktop aapt2 link -o final.apk --manifest /Users/connortumbleson/Desktop/aapt2training/app-debug/AndroidManifest.xml -I /Users/connortumbleson/Library/apktool/framework/1.apk -v resources.arsc.flat
/Users/connortumbleson/Library/apktool/framework/1.apk: note: loading include path.
note: linking package 'com.ibotpeaches.issue1719' using package ID 7f.
note: merging resource table resources.arsc.flat.
resources.arsc.flat: error: invalid compiled table.
error: failed parsing input.

Needs more research. I thought that was right.

@iBotPeaches

This comment has been minimized.

Owner

iBotPeaches commented Feb 14, 2018

Compile directory: treats the input file as a directory when --dir is specified. This will emit a zip of compiled files, one for each file in the directory. The directory must follow the Android resource directory structure (res/values-[qualifiers]/file.ext).

So when using directory

aapt2 compile --dir /Users/connortumbleson/Desktop/aapt2training/app-debug/res/ -o resources.zip -v

➜  Desktop unzip -l resources.zip 
Archive:  resources.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
     2264  12-31-1980 19:00   mipmap-mdpi_a.png.flat
     3224  12-31-1980 19:00   mipmap-hdpi_a.png.flat
     9120  12-31-1980 19:00   mipmap-xxxhdpi_a.png.flat
     6636  12-31-1980 19:00   mipmap-xxhdpi_a.png.flat
      244  12-31-1980 19:00   values_strings.arsc.flat
     4739  12-31-1980 19:00   mipmap-xhdpi_a.png.flat
---------                     -------
    26227                     6 files

➜  Desktop aapt2 link -o final.apk --manifest /Users/connortumbleson/Desktop/aapt2training/app-debug/AndroidManifest.xml -I /Users/connortumbleson/Library/apktool/framework/1.apk -v resources.zip
/Users/connortumbleson/Library/apktool/framework/1.apk: note: loading include path.
note: linking package 'com.ibotpeaches.issue1719' using package ID 7f.
note: merging archive resources.zip.
note: merging 'mipmap/a' from compiled file resources.zip@mipmap-hdpi_a.png.flat.
note: merging 'mipmap/a' from compiled file resources.zip@mipmap-xxhdpi_a.png.flat.
note: merging 'mipmap/a' from compiled file resources.zip@mipmap-xxxhdpi_a.png.flat.
note: merging 'mipmap/a' from compiled file resources.zip@mipmap-xhdpi_a.png.flat.
note: merging 'mipmap/a' from compiled file resources.zip@mipmap-mdpi_a.png.flat.
note: merging resource table resources.zip@values_strings.arsc.flat.
note: enabling pre-O feature split ID rewriting.
AndroidManifest.xml: note: writing to archive (keep_raw_values=false).
note: writing AndroidManifest.xml to archive.
note: writing res/mipmap-mdpi-v4/a.png to archive.
note: writing res/mipmap-hdpi-v4/a.png to archive.
note: writing res/mipmap-xhdpi-v4/a.png to archive.
note: writing res/mipmap-xxhdpi-v4/a.png to archive.
note: writing res/mipmap-xxxhdpi-v4/a.png to archive.
note: writing resources.arsc to archive.
➜  Desktop 

➜  Desktop unzip -l final.apk 
Archive:  final.apk
  Length      Date    Time    Name
---------  ---------- -----   ----
      672  12-31-1980 19:00   AndroidManifest.xml
     2096  12-31-1980 19:00   res/mipmap-mdpi-v4/a.png
     3056  12-31-1980 19:00   res/mipmap-hdpi-v4/a.png
     4569  12-31-1980 19:00   res/mipmap-xhdpi-v4/a.png
     6464  12-31-1980 19:00   res/mipmap-xxhdpi-v4/a.png
     8946  12-31-1980 19:00   res/mipmap-xxxhdpi-v4/a.png
     1228  12-31-1980 19:00   resources.arsc
---------                     -------
    27031                     7 files

:)

➜  Desktop aapt2 dump final.apk 
Package name=com.ibotpeaches.issue1719 id=7f

  type mipmap id=1 entryCount=1
    spec resource 0x7f010000 com.ibotpeaches.issue1719:mipmap/a
      (mdpi-v4) (file) res/mipmap-mdpi-v4/a.png
      (hdpi-v4) (file) res/mipmap-hdpi-v4/a.png
      (xhdpi-v4) (file) res/mipmap-xhdpi-v4/a.png
      (xxhdpi-v4) (file) res/mipmap-xxhdpi-v4/a.png
      (xxxhdpi-v4) (file) res/mipmap-xxxhdpi-v4/a.png

  type string id=2 entryCount=1
    spec resource 0x7f020000 com.ibotpeaches.issue1719:string/a
      () (string) "Issue1719"

Time to hook up Apktool with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment