Multiple ResTypeSpecs in a row, cannot be decoded. #1031

Closed
lucacapacci opened this Issue Sep 1, 2015 · 13 comments

Comments

Projects
None yet
4 participants
@lucacapacci

Apktool 2.0.1 crashes and produces no results with this apk https://play.google.com/store/apps/details?id=it.ingdirect.app

I: Using Apktool 2.0.1 on a.apk
I: Loading resource table...
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 9
    at brut.androlib.res.decoder.ARSCDecoder.readConfig(ARSCDecoder.java:189)
    at brut.androlib.res.decoder.ARSCDecoder.readType(ARSCDecoder.java:159)
    at brut.androlib.res.decoder.ARSCDecoder.readPackage(ARSCDecoder.java:116)
    at brut.androlib.res.decoder.ARSCDecoder.readTable(ARSCDecoder.java:78)
    at brut.androlib.res.decoder.ARSCDecoder.decode(ARSCDecoder.java:47)
    at brut.androlib.res.AndrolibResources.getResPackagesFromApk(AndrolibResources.java:538)
    at brut.androlib.res.AndrolibResources.loadMainPkg(AndrolibResources.java:63)
    at brut.androlib.res.AndrolibResources.getResTable(AndrolibResources.java:55)
    at brut.androlib.Androlib.getResTable(Androlib.java:64)
    at brut.androlib.ApkDecoder.setTargetSdkVersion(ApkDecoder.java:209)
    at brut.androlib.ApkDecoder.decode(ApkDecoder.java:92)
    at brut.apktool.Main.cmdDecode(Main.java:165)
    at brut.apktool.Main.main(Main.java:81)
@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
@iBotPeaches

iBotPeaches Sep 4, 2015

Owner

Yes, I confirm. Will look into this.

Owner

iBotPeaches commented Sep 4, 2015

Yes, I confirm. Will look into this.

@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
@iBotPeaches

iBotPeaches Oct 4, 2015

Owner

Okay, spent the morning on this one.

The problem shown above is just the failure point. The real problem occurs much early in the process. For example. entryCount reports 265 for the failure instance, but the config instance returns 9. Obviously storing 265 resources in 9 slots won't work, thus the above error. Manually setting the proper array size, allows apktool to complete but every resource is APKTOOL_DUMMY.

This means the failure is occurring way earlier. So I breakpointed after the first problem. We read a Type - https://github.com/android/platform_frameworks_base/blob/master/include/androidfw/ResourceTypes.h#L1246

and usually then we read the corresponding configuration after that - https://github.com/android/platform_frameworks_base/blob/master/include/androidfw/ResourceTypes.h#L1284

However, the type header is set to that of another type vs a configuration, so Apktool sets all the resources from that type to false values (APKTOOL_DUMMY, etc).

Owner

iBotPeaches commented Oct 4, 2015

Okay, spent the morning on this one.

The problem shown above is just the failure point. The real problem occurs much early in the process. For example. entryCount reports 265 for the failure instance, but the config instance returns 9. Obviously storing 265 resources in 9 slots won't work, thus the above error. Manually setting the proper array size, allows apktool to complete but every resource is APKTOOL_DUMMY.

This means the failure is occurring way earlier. So I breakpointed after the first problem. We read a Type - https://github.com/android/platform_frameworks_base/blob/master/include/androidfw/ResourceTypes.h#L1246

and usually then we read the corresponding configuration after that - https://github.com/android/platform_frameworks_base/blob/master/include/androidfw/ResourceTypes.h#L1284

However, the type header is set to that of another type vs a configuration, so Apktool sets all the resources from that type to false values (APKTOOL_DUMMY, etc).

@eriklu

This comment has been minimized.

Show comment
Hide comment
@eriklu

eriklu Oct 15, 2015

hi, ths your work.

the newest 2.0.2 version can work on LocusPro with the -r option.
but when I remove the -r option. the same exception occour again.

imac-2:test ao$ java -jar apktool_2.0.2.jar d LocusPro\ v3.10.3.apk -f
I: Using Apktool 2.0.2 on LocusPro v3.10.3.apk
I: Loading resource table...
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
    at brut.androlib.res.decoder.ARSCDecoder.readConfig(ARSCDecoder.java:189)
    at brut.androlib.res.decoder.ARSCDecoder.readType(ARSCDecoder.java:159)
    at brut.androlib.res.decoder.ARSCDecoder.readPackage(ARSCDecoder.java:116)
    at brut.androlib.res.decoder.ARSCDecoder.readTable(ARSCDecoder.java:78)
    at brut.androlib.res.decoder.ARSCDecoder.decode(ARSCDecoder.java:47)
    at brut.androlib.res.AndrolibResources.getResPackagesFromApk(AndrolibResources.java:544)
    at brut.androlib.res.AndrolibResources.loadMainPkg(AndrolibResources.java:63)
    at brut.androlib.res.AndrolibResources.getResTable(AndrolibResources.java:55)
    at brut.androlib.Androlib.getResTable(Androlib.java:65)
    at brut.androlib.ApkDecoder.setTargetSdkVersion(ApkDecoder.java:197)
    at brut.androlib.ApkDecoder.decode(ApkDecoder.java:96)
    at brut.apktool.Main.cmdDecode(Main.java:165)
    at brut.apktool.Main.main(Main.java:81)

eriklu commented Oct 15, 2015

hi, ths your work.

the newest 2.0.2 version can work on LocusPro with the -r option.
but when I remove the -r option. the same exception occour again.

imac-2:test ao$ java -jar apktool_2.0.2.jar d LocusPro\ v3.10.3.apk -f
I: Using Apktool 2.0.2 on LocusPro v3.10.3.apk
I: Loading resource table...
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
    at brut.androlib.res.decoder.ARSCDecoder.readConfig(ARSCDecoder.java:189)
    at brut.androlib.res.decoder.ARSCDecoder.readType(ARSCDecoder.java:159)
    at brut.androlib.res.decoder.ARSCDecoder.readPackage(ARSCDecoder.java:116)
    at brut.androlib.res.decoder.ARSCDecoder.readTable(ARSCDecoder.java:78)
    at brut.androlib.res.decoder.ARSCDecoder.decode(ARSCDecoder.java:47)
    at brut.androlib.res.AndrolibResources.getResPackagesFromApk(AndrolibResources.java:544)
    at brut.androlib.res.AndrolibResources.loadMainPkg(AndrolibResources.java:63)
    at brut.androlib.res.AndrolibResources.getResTable(AndrolibResources.java:55)
    at brut.androlib.Androlib.getResTable(Androlib.java:65)
    at brut.androlib.ApkDecoder.setTargetSdkVersion(ApkDecoder.java:197)
    at brut.androlib.ApkDecoder.decode(ApkDecoder.java:96)
    at brut.apktool.Main.cmdDecode(Main.java:165)
    at brut.apktool.Main.main(Main.java:81)

pwelyn pushed a commit to pwelyn/Apktool that referenced this issue Oct 15, 2015

越狱
Fix #1031
 java.lang.ArrayIndexOutOfBoundsException: 9
@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
@iBotPeaches

iBotPeaches Dec 2, 2015

Owner

Started looking at this. Notes for myself

https://github.com/iBotPeaches/platform_frameworks_base/blob/apktool-mm/include/androidfw/ResourceTypes.h#L1309

  • readEntry --> ResTable_config
  • readConfig --> ResTable_type
  • readType --> ReadTable_typeSpec

Seems our failure occurs in the first readConfig, where entryCount is reflecting wrong number of Resources.


entryCount is 260 on first Config. aapt d resources does not report any configuration with 260 items :(

Owner

iBotPeaches commented Dec 2, 2015

Started looking at this. Notes for myself

https://github.com/iBotPeaches/platform_frameworks_base/blob/apktool-mm/include/androidfw/ResourceTypes.h#L1309

  • readEntry --> ResTable_config
  • readConfig --> ResTable_type
  • readType --> ReadTable_typeSpec

Seems our failure occurs in the first readConfig, where entryCount is reflecting wrong number of Resources.


entryCount is 260 on first Config. aapt d resources does not report any configuration with 260 items :(

@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
@iBotPeaches

iBotPeaches Dec 6, 2015

Owner

Okay disregard previous comment by myself. We correctly read the first Type which is attr, and it reports 295 entries.

  type 0 configCount=1 entryCount=295

Likewise, aapt reports 295 entries with 1 configuration for this type. However, after reading the flags. Our next chunk is another type. Instead of being the expected config chunk.. This means all 295 resources are filled with APKTOOL_DUMMY. All data after this point is in unexpected places because we incorrectly read the remaining data in that chunk.

At this point either I don't understand this failure or a bug has been discovered in the root way apktool functions. If apktool does not find a configuration for a given type, it fills it with dummy data. According to the comment here - https://github.com/iBotPeaches/platform_frameworks_base/blob/apktool-mm/include/androidfw/ResourceTypes.h#L850

/**
 * A collection of resource data types within a package.  Followed by
 * one or more ResTable_type and ResTable_typeSpec structures containing the
 * entry values for each resource type.
 */

The problem Apktool has is it expects Type --> Config, Type --> Config, etc, etc. Currently this application goes Type --> Type, which isn't expected nor supported yet.

Owner

iBotPeaches commented Dec 6, 2015

Okay disregard previous comment by myself. We correctly read the first Type which is attr, and it reports 295 entries.

  type 0 configCount=1 entryCount=295

Likewise, aapt reports 295 entries with 1 configuration for this type. However, after reading the flags. Our next chunk is another type. Instead of being the expected config chunk.. This means all 295 resources are filled with APKTOOL_DUMMY. All data after this point is in unexpected places because we incorrectly read the remaining data in that chunk.

At this point either I don't understand this failure or a bug has been discovered in the root way apktool functions. If apktool does not find a configuration for a given type, it fills it with dummy data. According to the comment here - https://github.com/iBotPeaches/platform_frameworks_base/blob/apktool-mm/include/androidfw/ResourceTypes.h#L850

/**
 * A collection of resource data types within a package.  Followed by
 * one or more ResTable_type and ResTable_typeSpec structures containing the
 * entry values for each resource type.
 */

The problem Apktool has is it expects Type --> Config, Type --> Config, etc, etc. Currently this application goes Type --> Type, which isn't expected nor supported yet.

@iBotPeaches iBotPeaches self-assigned this Dec 8, 2015

@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
@iBotPeaches

iBotPeaches Dec 8, 2015

Owner
ibotpeaches@tombstone:~$ apktool d 1.89.0.apk
I: Using Apktool 2.0.3-e7c1bc-SNAPSHOT on 1.89.0.apk
I: Loading resource table...
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 186
    at brut.androlib.res.decoder.ARSCDecoder.readTableType(ARSCDecoder.java:190)
    at brut.androlib.res.decoder.ARSCDecoder.readTableTypeSpec(ARSCDecoder.java:159)
    at brut.androlib.res.decoder.ARSCDecoder.readTablePackage(ARSCDecoder.java:116)
    at brut.androlib.res.decoder.ARSCDecoder.readTableHeader(ARSCDecoder.java:78)
    at brut.androlib.res.decoder.ARSCDecoder.decode(ARSCDecoder.java:47)
    at brut.androlib.res.AndrolibResources.getResPackagesFromApk(AndrolibResources.java:544)
    at brut.androlib.res.AndrolibResources.loadMainPkg(AndrolibResources.java:63)
    at brut.androlib.res.AndrolibResources.getResTable(AndrolibResources.java:55)
    at brut.androlib.Androlib.getResTable(Androlib.java:66)
    at brut.androlib.ApkDecoder.setTargetSdkVersion(ApkDecoder.java:198)
    at brut.androlib.ApkDecoder.decode(ApkDecoder.java:96)
    at brut.apktool.Main.cmdDecode(Main.java:165)
    at brut.apktool.Main.main(Main.java:81)

Additionally Ingress 1.89.0 has this exact problem. Time to bump priority. This is a major major major bug now. Affecting more and more apps daily.

Owner

iBotPeaches commented Dec 8, 2015

ibotpeaches@tombstone:~$ apktool d 1.89.0.apk
I: Using Apktool 2.0.3-e7c1bc-SNAPSHOT on 1.89.0.apk
I: Loading resource table...
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 186
    at brut.androlib.res.decoder.ARSCDecoder.readTableType(ARSCDecoder.java:190)
    at brut.androlib.res.decoder.ARSCDecoder.readTableTypeSpec(ARSCDecoder.java:159)
    at brut.androlib.res.decoder.ARSCDecoder.readTablePackage(ARSCDecoder.java:116)
    at brut.androlib.res.decoder.ARSCDecoder.readTableHeader(ARSCDecoder.java:78)
    at brut.androlib.res.decoder.ARSCDecoder.decode(ARSCDecoder.java:47)
    at brut.androlib.res.AndrolibResources.getResPackagesFromApk(AndrolibResources.java:544)
    at brut.androlib.res.AndrolibResources.loadMainPkg(AndrolibResources.java:63)
    at brut.androlib.res.AndrolibResources.getResTable(AndrolibResources.java:55)
    at brut.androlib.Androlib.getResTable(Androlib.java:66)
    at brut.androlib.ApkDecoder.setTargetSdkVersion(ApkDecoder.java:198)
    at brut.androlib.ApkDecoder.decode(ApkDecoder.java:96)
    at brut.apktool.Main.cmdDecode(Main.java:165)
    at brut.apktool.Main.main(Main.java:81)

Additionally Ingress 1.89.0 has this exact problem. Time to bump priority. This is a major major major bug now. Affecting more and more apps daily.

@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
@iBotPeaches

iBotPeaches Dec 8, 2015

Owner

Another comment for myself. Sorry for notifications.

I was correct in (#1031 (comment)) comment. All the TypeSpecs are read in a row instead of going TypeSpec -> Config -> Entry. The new order is TypeSpec -> TypeSpec --> ... -> TypeSpec, then finally Config -> Entry -> Entry -> Entry --> Config -> Entry etc.

So I adapted the code to read all the TypeSpec, then move onto the Configs. This worked, but each Entry read had the wrong ids. This is because of a flaw in apktool, in which it expected the last read Type to be the Type the resources were a part of. So the last read type (id) was being used, but the Config being read was for the attr type which was read first.

Our decoder needs a tiny rewrite to support this change, without breaking previous apks. Still on a mission to figure out how to get this to happen myself in a smaller test apk. Building Marshmallow apks by myself in Android Studio does not trigger this change. AAPT would be responsible for this, so maybe I'm missing a configuration/parameter value that is causing it.

Research / fixes are ongoing.

Owner

iBotPeaches commented Dec 8, 2015

Another comment for myself. Sorry for notifications.

I was correct in (#1031 (comment)) comment. All the TypeSpecs are read in a row instead of going TypeSpec -> Config -> Entry. The new order is TypeSpec -> TypeSpec --> ... -> TypeSpec, then finally Config -> Entry -> Entry -> Entry --> Config -> Entry etc.

So I adapted the code to read all the TypeSpec, then move onto the Configs. This worked, but each Entry read had the wrong ids. This is because of a flaw in apktool, in which it expected the last read Type to be the Type the resources were a part of. So the last read type (id) was being used, but the Config being read was for the attr type which was read first.

Our decoder needs a tiny rewrite to support this change, without breaking previous apks. Still on a mission to figure out how to get this to happen myself in a smaller test apk. Building Marshmallow apks by myself in Android Studio does not trigger this change. AAPT would be responsible for this, so maybe I'm missing a configuration/parameter value that is causing it.

Research / fixes are ongoing.

@iBotPeaches iBotPeaches changed the title from error with specific apk to Multiple ResTypeSpecs in a row, cannot be decoded. Dec 8, 2015

@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
@iBotPeaches

iBotPeaches Dec 8, 2015

Owner

Okay, another update.

Patching the previous comment problem was a success. Types were correctly associated with the configurations they were part of, thus preserving the correct resource ids. In the case of Ingress 1.89.0, completely read the first block of resources. The [default] configuration of 32 attributes.

Next up was type 1 (drawables) with 15 configurations and 195 entries. This is where the problems started. The first config read was nodpi-v4 which is actually the 2nd configuration for this type according to aapt. When it should have read the default configuration first.

This is where things get strange. Reading the type returns 195 entries. Reading the first ConfigFlags should return the default config with 195 entries, but it didn't. So reading our int array resulted in tons of -1 values, thus going out of bounds.

Tis progress though

Owner

iBotPeaches commented Dec 8, 2015

Okay, another update.

Patching the previous comment problem was a success. Types were correctly associated with the configurations they were part of, thus preserving the correct resource ids. In the case of Ingress 1.89.0, completely read the first block of resources. The [default] configuration of 32 attributes.

Next up was type 1 (drawables) with 15 configurations and 195 entries. This is where the problems started. The first config read was nodpi-v4 which is actually the 2nd configuration for this type according to aapt. When it should have read the default configuration first.

This is where things get strange. Reading the type returns 195 entries. Reading the first ConfigFlags should return the default config with 195 entries, but it didn't. So reading our int array resulted in tons of -1 values, thus going out of bounds.

Tis progress though

@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
@iBotPeaches

iBotPeaches Dec 9, 2015

Owner

Okay another update x2.

Major progress.

$ apktool debug d 1.89.0.apk -f
I: Using Apktool 2.0.3-e7c1bc-SNAPSHOT on 1.89.0.apk
I: Loading resource table...
Exception in thread "main" brut.androlib.AndrolibException: Multiple resources: spec=0x7f0d0001 id/app_icon, config=[DEFAULT]
    at brut.androlib.res.data.ResType.addResource(ResType.java:63)
    at brut.androlib.res.data.ResType.addResource(ResType.java:56)
    at brut.androlib.res.decoder.ARSCDecoder.readEntry(ARSCDecoder.java:252)
    at brut.androlib.res.decoder.ARSCDecoder.readTableType(ARSCDecoder.java:220)
    at brut.androlib.res.decoder.ARSCDecoder.readTableTypeSpec(ARSCDecoder.java:166)
    at brut.androlib.res.decoder.ARSCDecoder.readTablePackage(ARSCDecoder.java:116)
    at brut.androlib.res.decoder.ARSCDecoder.readTableHeader(ARSCDecoder.java:78)
    at brut.androlib.res.decoder.ARSCDecoder.decode(ARSCDecoder.java:47)
    at brut.androlib.res.AndrolibResources.getResPackagesFromApk(AndrolibResources.java:544)
    at brut.androlib.res.AndrolibResources.loadMainPkg(AndrolibResources.java:63)
    at brut.androlib.res.AndrolibResources.getResTable(AndrolibResources.java:55)
    at brut.androlib.Androlib.getResTable(Androlib.java:66)
    at brut.androlib.ApkDecoder.setTargetSdkVersion(ApkDecoder.java:198)
    at brut.androlib.ApkDecoder.decode(ApkDecoder.java:96)
    at brut.apktool.Main.cmdDecode(Main.java:165)
    at brut.apktool.Main.main(Main.java:81)

The problem now, is that like previous comments I built a FIFO Queue (First In, First Out). Also known as a ArrayDeque in Java, leveraging addLast and removeFirst. However, I found after reading the configurations for the drawable type that since its sparse we might read the attributes then drawables, neither of which completely. So the current version of Apktool, reads the SpecType type, pops it off stack and reads the configurations (which may only be one, instead of all configurations for that TypeSpec). When it returns to the attr type, it actually pops the next element off the stack which is drawables. This conflict between mapping the correct TypeSpec to configurations is causing multiple resources.

Need to update the TypeSpec system to be some of index based list, reading the TableType typeId in order to know what TypeSpec to load.

Good News - Types & TypeSpecs are correctly load for both sparse and densely packed apks.

Owner

iBotPeaches commented Dec 9, 2015

Okay another update x2.

Major progress.

$ apktool debug d 1.89.0.apk -f
I: Using Apktool 2.0.3-e7c1bc-SNAPSHOT on 1.89.0.apk
I: Loading resource table...
Exception in thread "main" brut.androlib.AndrolibException: Multiple resources: spec=0x7f0d0001 id/app_icon, config=[DEFAULT]
    at brut.androlib.res.data.ResType.addResource(ResType.java:63)
    at brut.androlib.res.data.ResType.addResource(ResType.java:56)
    at brut.androlib.res.decoder.ARSCDecoder.readEntry(ARSCDecoder.java:252)
    at brut.androlib.res.decoder.ARSCDecoder.readTableType(ARSCDecoder.java:220)
    at brut.androlib.res.decoder.ARSCDecoder.readTableTypeSpec(ARSCDecoder.java:166)
    at brut.androlib.res.decoder.ARSCDecoder.readTablePackage(ARSCDecoder.java:116)
    at brut.androlib.res.decoder.ARSCDecoder.readTableHeader(ARSCDecoder.java:78)
    at brut.androlib.res.decoder.ARSCDecoder.decode(ARSCDecoder.java:47)
    at brut.androlib.res.AndrolibResources.getResPackagesFromApk(AndrolibResources.java:544)
    at brut.androlib.res.AndrolibResources.loadMainPkg(AndrolibResources.java:63)
    at brut.androlib.res.AndrolibResources.getResTable(AndrolibResources.java:55)
    at brut.androlib.Androlib.getResTable(Androlib.java:66)
    at brut.androlib.ApkDecoder.setTargetSdkVersion(ApkDecoder.java:198)
    at brut.androlib.ApkDecoder.decode(ApkDecoder.java:96)
    at brut.apktool.Main.cmdDecode(Main.java:165)
    at brut.apktool.Main.main(Main.java:81)

The problem now, is that like previous comments I built a FIFO Queue (First In, First Out). Also known as a ArrayDeque in Java, leveraging addLast and removeFirst. However, I found after reading the configurations for the drawable type that since its sparse we might read the attributes then drawables, neither of which completely. So the current version of Apktool, reads the SpecType type, pops it off stack and reads the configurations (which may only be one, instead of all configurations for that TypeSpec). When it returns to the attr type, it actually pops the next element off the stack which is drawables. This conflict between mapping the correct TypeSpec to configurations is causing multiple resources.

Need to update the TypeSpec system to be some of index based list, reading the TableType typeId in order to know what TypeSpec to load.

Good News - Types & TypeSpecs are correctly load for both sparse and densely packed apks.

@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
@iBotPeaches

iBotPeaches Dec 10, 2015

Owner

Okay another update x3

$ apktool debug d 1.89.0.apk -f
I: Using Apktool 2.0.3-e7c1bc-SNAPSHOT on 1.89.0.apk
I: Loading resource table...
I: Decoding AndroidManifest.xml with resources...
I: Loading resource table from file: /home/ibotpeaches/apktool/framework/1.apk
I: Regular manifest package...
I: Decoding file-resources...
I: Decoding values */* XMLs...
Exception in thread "main" brut.androlib.err.UndefinedResObject: resource spec: 0x7f0d0001
    at brut.androlib.res.data.ResPackage.getResSpec(ResPackage.java:59)
    at brut.androlib.res.data.ResTable.getResSpec(ResTable.java:65)
    at brut.androlib.res.data.ResTable.getResSpec(ResTable.java:61)
    at brut.androlib.res.data.value.ResReferenceValue.getReferent(ResReferenceValue.java:57)
    at brut.androlib.res.data.value.ResEnumAttr.serializeBody(ResEnumAttr.java:56)
    at brut.androlib.res.data.value.ResAttr.serializeToResValuesXml(ResAttr.java:64)
    at brut.androlib.res.AndrolibResources.generateValuesFile(AndrolibResources.java:502)
    at brut.androlib.res.AndrolibResources.decode(AndrolibResources.java:252)
    at brut.androlib.Androlib.decodeResourcesFull(Androlib.java:136)
    at brut.androlib.ApkDecoder.decode(ApkDecoder.java:102)
    at brut.apktool.Main.cmdDecode(Main.java:165)
    at brut.apktool.Main.main(Main.java:81)

Per the last comment. TypeSpecs are now stored in a HashMap, indexed by the TypeId. During read, the TypeId of the TableType is used to reference a TypeSpec. That TypeSpec ID is used for building the correct resource id.

The next issue we encountered was handling INVALID TYPE lines. Some APKs have lines like such

      INVALID TYPE CONFIG FOR RESOURCE 0x01010267

I'm still not quite sure why those occur, but they do. To handle this we generate a bogus FALSE value with that resource id. However, the logic and code in Apktool for handling the what we call "Missing Res Specs" was dependent on a densely packed order of types. IE - looking for missing resources after reading a TypeSpec. We now know that TypeSpecs are sparse, so the logic for Missing Res Specs is wrong.

Multiple errors occurred with this logic as resources were duplicated. IE a bogus resource was created for a legitimate resource. A temporary and hacky fix, which was nothing more than disabling the Missing Res Specs was put into place. We were left with the stacktrace shown above. We correctly read the entire ResTable. Hurrah :)

While decoding files back to human readable form, we encounter an error while following a reference on a file. So this resource doesn't exist. Why?

Well its obvious. The "Missing Res Specs" are disabled.

I wonder what that bad resource is.

      INVALID TYPE CONFIG FOR RESOURCE 0x7f0d0001

Yep, one of those :)

I just need to patch MissingResSpecs back in Apktool, and make sure it works and knock on wood. This is fixed.

Owner

iBotPeaches commented Dec 10, 2015

Okay another update x3

$ apktool debug d 1.89.0.apk -f
I: Using Apktool 2.0.3-e7c1bc-SNAPSHOT on 1.89.0.apk
I: Loading resource table...
I: Decoding AndroidManifest.xml with resources...
I: Loading resource table from file: /home/ibotpeaches/apktool/framework/1.apk
I: Regular manifest package...
I: Decoding file-resources...
I: Decoding values */* XMLs...
Exception in thread "main" brut.androlib.err.UndefinedResObject: resource spec: 0x7f0d0001
    at brut.androlib.res.data.ResPackage.getResSpec(ResPackage.java:59)
    at brut.androlib.res.data.ResTable.getResSpec(ResTable.java:65)
    at brut.androlib.res.data.ResTable.getResSpec(ResTable.java:61)
    at brut.androlib.res.data.value.ResReferenceValue.getReferent(ResReferenceValue.java:57)
    at brut.androlib.res.data.value.ResEnumAttr.serializeBody(ResEnumAttr.java:56)
    at brut.androlib.res.data.value.ResAttr.serializeToResValuesXml(ResAttr.java:64)
    at brut.androlib.res.AndrolibResources.generateValuesFile(AndrolibResources.java:502)
    at brut.androlib.res.AndrolibResources.decode(AndrolibResources.java:252)
    at brut.androlib.Androlib.decodeResourcesFull(Androlib.java:136)
    at brut.androlib.ApkDecoder.decode(ApkDecoder.java:102)
    at brut.apktool.Main.cmdDecode(Main.java:165)
    at brut.apktool.Main.main(Main.java:81)

Per the last comment. TypeSpecs are now stored in a HashMap, indexed by the TypeId. During read, the TypeId of the TableType is used to reference a TypeSpec. That TypeSpec ID is used for building the correct resource id.

The next issue we encountered was handling INVALID TYPE lines. Some APKs have lines like such

      INVALID TYPE CONFIG FOR RESOURCE 0x01010267

I'm still not quite sure why those occur, but they do. To handle this we generate a bogus FALSE value with that resource id. However, the logic and code in Apktool for handling the what we call "Missing Res Specs" was dependent on a densely packed order of types. IE - looking for missing resources after reading a TypeSpec. We now know that TypeSpecs are sparse, so the logic for Missing Res Specs is wrong.

Multiple errors occurred with this logic as resources were duplicated. IE a bogus resource was created for a legitimate resource. A temporary and hacky fix, which was nothing more than disabling the Missing Res Specs was put into place. We were left with the stacktrace shown above. We correctly read the entire ResTable. Hurrah :)

While decoding files back to human readable form, we encounter an error while following a reference on a file. So this resource doesn't exist. Why?

Well its obvious. The "Missing Res Specs" are disabled.

I wonder what that bad resource is.

      INVALID TYPE CONFIG FOR RESOURCE 0x7f0d0001

Yep, one of those :)

I just need to patch MissingResSpecs back in Apktool, and make sure it works and knock on wood. This is fixed.

@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
@iBotPeaches

iBotPeaches Dec 10, 2015

Owner

Okay another update x4.

$ apktool debug d 1.89.0.apk -f
I: Using Apktool 2.0.3-e7c1bc-SNAPSHOT on 1.89.0.apk
I: Loading resource table...
I: Decoding AndroidManifest.xml with resources...
I: Loading resource table from file: /home/ibotpeaches/apktool/framework/1.apk
I: Regular manifest package...
I: Decoding file-resources...
I: Decoding values */* XMLs...
I: Baksmaling classes.dex...
I: Copying assets and libs...
I: Copying unknown files...
I: Copying original files...

We did it :)

The current fix for MissingResSpecs is dirty and needs some clean up, because of a few things.

  1. You can't assume all missing resources in a configuration are actually busted. Some configurations (ie -en) might not have some resources and fall back to default. Therefore it doesn't make sense to mark those resources as DUMMY. So in our case, we make them DUMMY resources. If another resID comes along with the same ID as a DUMMY resource, then we remove the DUMMY resource and replace it with the correct one. This allows the sparse reading of ResTable to work with adding in DUMMY resources.
  2. I've touched code everywhere. ResPackage, ResResSpec, ResType, ResTypeSpec, ARSCDecoder, ExtMXSerializer, ResXMLEncoders. I need to clean up some of the fixes, because I think I've slowed down the execution time of some functions.
  3. I have about 15 commits unpushed. I need to clean up history, and make sure backward compat is absolutely working.
  4. I need to create a unit-test for this sparse ResTable.
Owner

iBotPeaches commented Dec 10, 2015

Okay another update x4.

$ apktool debug d 1.89.0.apk -f
I: Using Apktool 2.0.3-e7c1bc-SNAPSHOT on 1.89.0.apk
I: Loading resource table...
I: Decoding AndroidManifest.xml with resources...
I: Loading resource table from file: /home/ibotpeaches/apktool/framework/1.apk
I: Regular manifest package...
I: Decoding file-resources...
I: Decoding values */* XMLs...
I: Baksmaling classes.dex...
I: Copying assets and libs...
I: Copying unknown files...
I: Copying original files...

We did it :)

The current fix for MissingResSpecs is dirty and needs some clean up, because of a few things.

  1. You can't assume all missing resources in a configuration are actually busted. Some configurations (ie -en) might not have some resources and fall back to default. Therefore it doesn't make sense to mark those resources as DUMMY. So in our case, we make them DUMMY resources. If another resID comes along with the same ID as a DUMMY resource, then we remove the DUMMY resource and replace it with the correct one. This allows the sparse reading of ResTable to work with adding in DUMMY resources.
  2. I've touched code everywhere. ResPackage, ResResSpec, ResType, ResTypeSpec, ARSCDecoder, ExtMXSerializer, ResXMLEncoders. I need to clean up some of the fixes, because I think I've slowed down the execution time of some functions.
  3. I have about 15 commits unpushed. I need to clean up history, and make sure backward compat is absolutely working.
  4. I need to create a unit-test for this sparse ResTable.
@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
Owner

iBotPeaches commented Dec 14, 2015

@opysanni

This comment has been minimized.

Show comment
Hide comment
@opysanni

opysanni Feb 7, 2017

can i use the apktool to decode or decompile multiple apk file at once, and how?

opysanni commented Feb 7, 2017

can i use the apktool to decode or decompile multiple apk file at once, and how?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment