Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

iOS 15 new shared cache format. #15

Closed
0cyn opened this issue Jun 8, 2021 · 15 comments
Closed

iOS 15 new shared cache format. #15

0cyn opened this issue Jun 8, 2021 · 15 comments

Comments

@0cyn
Copy link
Contributor

0cyn commented Jun 8, 2021

latest revision has completely changed the format of the shared cache which may end up requiring a rewrite or seperate project to tackle. Here's a spot to compile info on the topic.
@arandomdev
Copy link
Owner

arandomdev commented Jun 8, 2021 •
edited
Loading

Okay, I believe this was introduced in iOS 15 beta. The Dyld Shared Cache (DSC) seems to be split into separate files, i.e.

  • dyld_shared_cache_arm64
  • dyld_shared_cache_arm64.1
  • dyld_shared_cache_arm64.2
  • dyld_shared_cache_arm64.3
  • dyld_shared_cache_arm64.symbols

Looking at the header of each of these, we can find out a few things.

dyld_shared_cache_arm64 
dyld_shared_cache_arm64
        magic: b'dyld_v1   arm64'
        mappingOffset: 456
        mappingCount: 1
        imagesOffset: 0
        imagesCount: 0
        dyldBaseAddress: 0
        codeSignatureOffset: 1070710784
        codeSignatureSize: 8372224
        slideInfoOffsetUnused: 0
        slideInfoSizeUnused: 0
        localSymbolsOffset: 0
        localSymbolsSize: 0
        uuid: [83, 218, 0, 137, 88, 97, 52, 244, 172, 33, 253, 129, 13, 125, 99, 19]
        cacheType: 1
        branchPoolsOffset: 0
        branchPoolsCount: 0
        accelerateInfoAddr: 0
        accelerateInfoSize: 0
        imagesTextOffset: 75832
        imagesTextCount: 2352
        patchInfoAddr: 8667632592
        patchInfoSize: 1361752
        otherImageGroupAddrUnused: 0
        otherImageGroupSizeUnused: 0
        progClosuresAddr: 0
        progClosuresSize: 0
        progClosuresTrieAddr: 0
        progClosuresTrieSize: 0
        platform: 2
        formatVersion: 0
        dylibsExpectedOnDisk: 0
        simulator: 0
        locallyBuiltCache: 0
        builtFromChainedFixups: 0
        padding: 0
        sharedRegionStart: 6442450944
        sharedRegionSize: 2236121088
        maxSlide: 113328128
        dylibsImageArrayAddr: 0
        dylibsImageArraySize: 0
        dylibsTrieAddr: 8667529216
        dylibsTrieSize: 103376
        otherImageArrayAddr: 0
        otherImageArraySize: 0
        otherTrieAddr: 0
        otherTrieSize: 0
        mappingWithSlideOffset: 512
        mappingWithSlideCount: 1
dyld_shared_cache_arm64.1 
dyld_shared_cache_arm64.1
        magic: b'dyld_v1   arm64'
        mappingOffset: 456
        mappingCount: 1
        imagesOffset: 0
        imagesCount: 0
        dyldBaseAddress: 0
        codeSignatureOffset: 586416128
        codeSignatureSize: 4587520
        slideInfoOffsetUnused: 0
        slideInfoSizeUnused: 0
        localSymbolsOffset: 0
        localSymbolsSize: 0
        uuid: [158, 168, 231, 25, 184, 22, 50, 47, 187, 173, 79, 221, 157, 76, 52, 209]
        cacheType: 1
        branchPoolsOffset: 0
        branchPoolsCount: 0
        accelerateInfoAddr: 0
        accelerateInfoSize: 0
        imagesTextOffset: 75832
        imagesTextCount: 2352
        patchInfoAddr: 0
        patchInfoSize: 0
        otherImageGroupAddrUnused: 0
        otherImageGroupSizeUnused: 0
        progClosuresAddr: 0
        progClosuresSize: 0
        progClosuresTrieAddr: 0
        progClosuresTrieSize: 0
        platform: 2
        formatVersion: 0
        dylibsExpectedOnDisk: 0
        simulator: 0
        locallyBuiltCache: 0
        builtFromChainedFixups: 0
        padding: 0
        sharedRegionStart: 7513161728
        sharedRegionSize: 0
        maxSlide: 0
        dylibsImageArrayAddr: 0
        dylibsImageArraySize: 0
        dylibsTrieAddr: 0
        dylibsTrieSize: 0
        otherImageArrayAddr: 0
        otherImageArraySize: 0
        otherTrieAddr: 0
        otherTrieSize: 0
        mappingWithSlideOffset: 512
        mappingWithSlideCount: 1
dyld_shared_cache_arm64.2 
dyld_shared_cache_arm64.2
        magic: b'dyld_v1   arm64'
        mappingOffset: 456
        mappingCount: 4
        imagesOffset: 0
        imagesCount: 0
        dyldBaseAddress: 0
        codeSignatureOffset: 276037632
        codeSignatureSize: 2162688
        slideInfoOffsetUnused: 0
        slideInfoSizeUnused: 0
        localSymbolsOffset: 0
        localSymbolsSize: 0
        uuid: [159, 214, 128, 158, 37, 36, 55, 108, 135, 193, 111, 161, 170, 112, 69, 63]
        cacheType: 1
        branchPoolsOffset: 0
        branchPoolsCount: 0
        accelerateInfoAddr: 0
        accelerateInfoSize: 0
        imagesTextOffset: 76168
        imagesTextCount: 2352
        patchInfoAddr: 0
        patchInfoSize: 0
        otherImageGroupAddrUnused: 0
        otherImageGroupSizeUnused: 0
        progClosuresAddr: 0
        progClosuresSize: 0
        progClosuresTrieAddr: 0
        progClosuresTrieSize: 0
        platform: 2
        formatVersion: 0
        dylibsExpectedOnDisk: 0
        simulator: 0
        locallyBuiltCache: 0
        builtFromChainedFixups: 0
        padding: 0
        sharedRegionStart: 8099577856
        sharedRegionSize: 0
        maxSlide: 0
        dylibsImageArrayAddr: 0
        dylibsImageArraySize: 0
        dylibsTrieAddr: 0
        dylibsTrieSize: 0
        otherImageArrayAddr: 0
        otherImageArraySize: 0
        otherTrieAddr: 0
        otherTrieSize: 0
        mappingWithSlideOffset: 680
        mappingWithSlideCount: 4
dyld_shared_cache_arm64.3 
dyld_shared_cache_arm64.3
        magic: b'dyld_v1   arm64'
        mappingOffset: 456
        mappingCount: 2
        imagesOffset: 0
        imagesCount: 0
        dyldBaseAddress: 0
        codeSignatureOffset: 202293248
        codeSignatureSize: 1589248
        slideInfoOffsetUnused: 0
        slideInfoSizeUnused: 0
        localSymbolsOffset: 0
        localSymbolsSize: 0
        uuid: [245, 34, 192, 200, 68, 6, 50, 231, 187, 94, 165, 39, 97, 46, 114, 21]
        cacheType: 1
        branchPoolsOffset: 0
        branchPoolsCount: 0
        accelerateInfoAddr: 0
        accelerateInfoSize: 0
        imagesTextOffset: 75944
        imagesTextCount: 2352
        patchInfoAddr: 0
        patchInfoSize: 0
        otherImageGroupAddrUnused: 0
        otherImageGroupSizeUnused: 0
        progClosuresAddr: 0
        progClosuresSize: 0
        progClosuresTrieAddr: 0
        progClosuresTrieSize: 0
        platform: 2
        formatVersion: 0
        dylibsExpectedOnDisk: 0
        simulator: 0
        locallyBuiltCache: 0
        builtFromChainedFixups: 0
        padding: 0
        sharedRegionStart: 8476278784
        sharedRegionSize: 0
        maxSlide: 0
        dylibsImageArrayAddr: 0
        dylibsImageArraySize: 0
        dylibsTrieAddr: 0
        dylibsTrieSize: 0
        otherImageArrayAddr: 0
        otherImageArraySize: 0
        otherTrieAddr: 0
        otherTrieSize: 0
        mappingWithSlideOffset: 568
        mappingWithSlideCount: 2
dyld_shared_cache_arm64.symbols 
dyld_shared_cache_arm64.symbols
        magic: b'dyld_v1   arm64'
        mappingOffset: 456
        mappingCount: 1
        imagesOffset: 0
        imagesCount: 0
        dyldBaseAddress: 0
        codeSignatureOffset: 460079104
        codeSignatureSize: 3604480
        slideInfoOffsetUnused: 0
        slideInfoSizeUnused: 0
        localSymbolsOffset: 16384
        localSymbolsSize: 460062720
        uuid: [44, 127, 111, 44, 107, 123, 62, 64, 173, 193, 47, 206, 7, 168, 149, 236]
        cacheType: 0
        branchPoolsOffset: 0
        branchPoolsCount: 0
        accelerateInfoAddr: 0
        accelerateInfoSize: 0
        imagesTextOffset: 0
        imagesTextCount: 0
        patchInfoAddr: 0
        patchInfoSize: 0
        otherImageGroupAddrUnused: 0
        otherImageGroupSizeUnused: 0
        progClosuresAddr: 0
        progClosuresSize: 0
        progClosuresTrieAddr: 0
        progClosuresTrieSize: 0
        platform: 2
        formatVersion: 0
        dylibsExpectedOnDisk: 0
        simulator: 0
        locallyBuiltCache: 0
        builtFromChainedFixups: 0
        padding: 0
        sharedRegionStart: 0
        sharedRegionSize: 0
        maxSlide: 0
        dylibsImageArrayAddr: 0
        dylibsImageArraySize: 0
        dylibsTrieAddr: 0
        dylibsTrieSize: 0
        otherImageArrayAddr: 0
        otherImageArraySize: 0
        otherTrieAddr: 0
        otherTrieSize: 0
        mappingWithSlideOffset: 512
        mappingWithSlideCount: 1

Basically different elements are separated, for example, all the local symbols are stored in the ".symbols" file. Also by examining the dyld source code, it's likely that the mapping info (located with mappingOffset) is placed directly after the header. Currently, the size of the header is 320 bytes, but the mappingOffset is set to 456, which means that we are missing some fields in the new header.

Unfortunately, until Apple open sources their new format, It would be hard to add support for this.

@blacktop
Copy link

blacktop commented Jun 9, 2021

Got 3 new header fields so far - https://github.com/blacktop/ipsw/blob/master/hack/extras/Dyld.bt (14 more to go)

  • imagesOffset and imagesCount moved to the very bottom for some reason
  • numSplits added to tell you how many dsc.1, dsc.2 etc files there are

@arandomdev
Copy link
Owner

Interesting! Did you disassemble dyld to get the size of the new fields? Until the actual release of the code, I guess I can look further into it.

@blacktop
Copy link

blacktop commented Jun 9, 2021

I was going to do that eventually, I am mostly using 010 Editor to jump around and make guesses at what the fields are, if they fields made sense as uint64 or uint32 etc is a try and guess procedure right now, but it got me 3 fields so far ;) There are so MANY new fields, this seems like a pretty big change, but it is very strange that they would move imagesOffset and imagesCount to the bottom and still keep the old field around?? An associate on twitter took the linkers WWDC 2021 lab and they said the in-memory cache will always look the same, but the on-disk might change (meaning that this new format is in flux??) 🤷

@arandomdev
Copy link
Owner

As for keeping the old field around, I think it's for some cross-compatibility so that older dyld loaders could still make sense of newer caches. Like it would be better for an older loader to read the imageOffset as 0 instead of a new value meant for a different field. But given apple's method of bundling both the loader and the cache together in the IPSW file, it wouldn't make too much of a difference.

Also, it doesn't make sense that they would just move the dyld_cache_image_info structure without some reason. Maybe they did something similar to dyld_cache_mapping_info and dyld_cache_mapping_and_slide_info, where this new image info structure contains more fields.

If possible can you link the Twitter conversation or elaborate more on it?

@blacktop
Copy link

blacktop commented Jun 9, 2021

https://twitter.com/_saagarjha/status/1402345391706677259

@blacktop
Copy link

blacktop commented Jun 9, 2021

got another field: (and correct name for "numSplits"

<SNIP>
    uint32 numSubCaches;     // number of dyld_shared_cache .1,.2,.3 files
    Uuid symbolSubCacheUUID; // unique value for .symbols sub-cache
<SNIP>

@arandomdev
Copy link
Owner

Nice! how are you getting these field names?

@blacktop
Copy link

blacktop commented Jun 9, 2021

RE of dsc_extractor.bundle

There is also a comparison of subcache UUIDs at the uint32 right above the numSubCaches field, but I am confused if it is a "partial UUID?" or just an ID? but there are all the same for the dsc and the dsc.1, dsc.2 etc. but it refers to it as a UUID which needs 16bytes ??

@blacktop
Copy link

Got a few more fields, but they are kinda strange, this format feels un-finalized 🤷

@arandomdev
Copy link
Owner

It might be a beta thing. They might not adopt this format for early iOS 15 releases.

@zcutlip zcutlip mentioned this issue Aug 18, 2021
Closed
@zhouxing5311 zhouxing5311 mentioned this issue Sep 22, 2021
Closed
@blacktop
Copy link

I've been meaning to blog about my findings for a while, but I've realized I write english like a complete moron. In the mean time I hope this makes sense to people?

NationalSecurityAgency/ghidra#3666 (comment)

@arandomdev
Copy link
Owner

This is really helpful, I’ll see what I can do with DyldExtractor when I get the chance.

@arandomdev
Copy link
Owner

I uploaded a beta branch here. It contains all the fixes for the new format, I'm just doing a lot of bug fixes now.

@arandomdev
Copy link
Owner

iOS 15 caches are now supported with version 2.0.0!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@blacktop @arandomdev @0cyn