Opening large archives on iOS yields "Cannot allocate memory" error from NSData

[logancollins]

On some devices (but not all), attempting to open a large archive (> 500 MB) results in NSData falling to initialize using NSDataReadingMappedAlways with a "Cannot allocate memory" generic Cocoa error.

Apparently this is a known regression in iOS 7 that has not been fixed in iOS 8 (see http://openradar.io/14388203).

This prevents archives larger than this size to be read for decompression. One likely solution could be to modify behavior to use NSFileHandle for accessing the file data.

I can reproduce this issue on an iPod touch (5th gen) and iPad 3, but not an iPhone 5s.

[pixelglow]

Thanks for your bug report!

I can't reproduce this on my own iPad 3 -- please provide the minimal code to trigger this together with iOS, device identifiers and link to an offending zip file, something similar to the following.

My configuration was as follows:

• iPad 3 (Model MC705X/A).
• iOS 7.0.6
• zipzap version 8.0 (latest)
• zipinfo stevejobs.zip (700+ MB file)

Archive:  /Volumes/Tuscany/stevejobs.zip   729954092 bytes   3 files
-rwxrwxrwx  2.1 unx 734685184 bX defN  2-Oct-13 13:00 stevejobs.avi
drwxrwxr-x  2.1 unx        0 bx stor 17-Sep-14 11:25 __MACOSX/
-rw-rw-rw-  2.1 unx     1527 bX defN  2-Oct-13 13:00 __MACOSX/._stevejobs.avi
3 files, 734686711 bytes uncompressed, 729953634 bytes compressed:  0.6%

• Code (in -[AppDelegate application:didFinishLaunchingWithOptions:)

NSError* err = nil;
ZZArchive* archive = [ZZArchive archiveWithURL:[[NSBundle mainBundle] URLForResource:@"stevejobs" withExtension:@"zip"] error:&err];

• Code architecture is arm7 (read this from the Link step next to -arch)
• Linked to SDK 7.1.

At a breakpoint after the code, err was still nil and ZZArchive contains a valid object.

[logancollins]

The configuration I can reproduce this most reliably with:

A 612 MB zip of 240 GIF images linked here:
https://www.dropbox.com/s/9cyvqb18y5sjsdh/GIFs.zip?dl=0

• iPod touch (5th generation)
• iOS 8 GM seed
• zipzap 8.0 (github latest)
• zipinfo GIFs.zip

Archive:  /Users/logan/Dropbox/Fail Whale/GIFs.zip   612588235   511
drwxr-xr-x  2.1 unx        0 bx stor 15-Sep-14 11:27 GIFs/
-rw-r--r--  2.1 unx   155340 bX defN 23-Jul-14 22:32 GIFs/ahhhh.gif
drwxrwxr-x  2.1 unx        0 bx stor 16-Sep-14 09:41 __MACOSX/
drwxrwxr-x  2.1 unx        0 bx stor 16-Sep-14 09:41 __MACOSX/GIFs/
-rw-r--r--  2.1 unx      317 bX defN 23-Jul-14 22:32 __MACOSX/GIFs/._ahhhh.gif
-rw-r--r--  2.1 unx  1950163 bX defN  6-Jun-13 10:12 GIFs/Ahhh… no.gif
-rw-r--r--  2.1 unx      317 bX defN  6-Jun-13 10:12 __MACOSX/GIFs/._Ahhh… no.gif
(truncated for sanity)

• Code:

// Open archive
NSError *nsError = nil;
ZZArchive *archive = [[ZZArchive alloc] initWithURL:[NSURL fileURLWithPath:sourcePath] options:@{ZZOpenOptionsCreateIfMissingKey: @NO} error:&nsError];
if ( nsError != nil )
{
    success = NO;
    *error = nsError;
}

At this point, nsError was a the following and archive was nil:

Error Domain=com.pixelglow.zipzap Code=0 "The operation couldn’t be completed. (com.pixelglow.zipzap error 0.)" UserInfo=0x17870f20 {NSUnderlyingError=0x17870970 "The operation couldn’t be completed. (Cocoa error 256.)"}

[logancollins]

Tracing into loadCanMiss:error: within zip zap, the underlying error is:

Error Domain=NSCocoaErrorDomain Code=256 "The operation couldn’t be completed. (Cocoa error 256.)" UserInfo=0x17870970 {NSFilePath=/private/var/mobile/Containers/Shared/AppGroup/799C88D2-63E8-490B-99DF-42AEED9682B0/Documents/GIFs.zip, NSUnderlyingError=0x165072e0 "The operation couldn’t be completed. Cannot allocate memory"}

[pixelglow]

Hmmm… I can't repro with any of my iOS 7 machines: iPad 3, iPad Mini Retina Display or iPhone 5. I don't yet have a machine running iOS 8.

Did you get this error with a minimal project or when you insert your code into -[AppDelegate application:didFinishLaunchingWithOptions]? It's possible that you were already running low in memory when you tried to open the archive.

If you're still getting the memory at the start, try doing a mmap with various options to see if it returns -1 i.e. failure to memory map. It may also be that NSDataReadingMappedAlways doesn't actually do mmap'ing for whatever reason. If so, I may move the code over to use mmap directly.

[phoenix42]

Hi,

I'm getting the same issue, previously on iOS 7.x and now on iOS 8. Zip file is 597Mb and was created by ZipZap. Smaller files open correctly.

Error occurs on iPad mini generation 1. I'll try the mmap too. Is there any theoretical limit to mmap size on the platform? Presumably there's some big address space limit but the memory manager should be paging in and out as required.

[phoenix42]

Ok, have tried the following:

const char *path = [[archivePath path]cStringUsingEncoding:[NSString defaultCStringEncoding]];

FILE *f = fopen( path, "rb" );
fseek( f, 0, SEEK_END );
int len = (int)ftell( f );
fseek( f, 0, SEEK_SET );

void *raw = mmap( 0, len, PROT_READ, MAP_SHARED, fileno( f ), 0 );

if ( raw == MAP_FAILED ) {
    printf( "MAP_FAILED. errno=%d", errno ); // Here it says 12, which is ENOMEM.
}

It fails with the error no memory. Is it possible to map only the sections required rather than the whole file?

[pixelglow]

I've done some tests with iOS 8.0, and have some ideas for a way forward.

First the tests, which tries to exhaust the addressable memory on the device.

int ff = open(path, O_RDONLY);
int i;
for (i = 0; i < 10000; ++i)
{
    void* addr;
    if ((addr = mmap(0, 1638400, PROT_READ, MAP_SHARED, ff, 0 ))== (void*)-1)
        break;
}
NSLog(@"%u", i);

On my devices:

• iPad 2 on iOS 8.0: 614 (1GB)
• iPad 3 on iOS 7.x: 1065 (1.74GB)
• iPad Mini with Retina Display on iOS 8.0 (64-bit): 1265 (2.07GB)

On the simulator:

• iPhone 5 on iOS 8.0: 2097 (3.4GB)
• iPhone 6 on iOS 8.0 (64-bit): 10000 (>16.38GB)

My analysis:

• It looks like the devices are artificially limiting the amount of addressable memory available to each process. I can't see anything in the iOS SDK or even lower level BSD/POSIX level that would increase the addressable memory e.g. I tried setrlimit but it doesn't seem to affect the process, presumably because it's not run as superuser.
• The addressable memory doesn't really correspond to the physical memory available. After all, the benefit of mmap is that it only pages in and occupies physical memory when needed.
• If we mmap smaller chunks we can eventually access more memory, but not significantly more e.g. the iPad Mini in 64-bit mode is still limited to something like 2GB, compared to over 16GB in simulator mode.
• All the device memory address limits are well below the theoretical limit for standard zip files (4GB).

[pixelglow]

Absent Apple "fixing" iOS devices to allow more memory addresses, we could:

• Back the archive with a cache of smaller memory maps, which can then be purged as needed.
  • Note that this can't really be a NSCache or similar object that reacts to real memory pressure, but something that reacts to virtual memory pressure.
  • We can't make the smaller memory maps fit exactly into each zip entry, since memory maps have to start at 4096-byte (or 16384-byte) offsets from the start of the file and zip entries could start anywhere.
• Only read the metadata into memory, and abandon memory maps entirely (except perhaps when returning stored, unencrypted data).
• Figure out some way of bumping the memory address limit in the process.

Not an ideal situation.

[phoenix42]

I think you're right.

My opinion is that the library should be able to handle large files whichever device is being used. The memory map should allow that, but isn't behaving consistently between models. And it's not easy to predict when it won't work. I'm working on an App which needs to allow import/export of large files for backup purposes. My nightmare scenario is that someone could export a file which they couldn't later import!

I was until recently a Windows developer so I'm a little new to the platform, so please excuse me if this is a silly question, but can't we map a compliant window around each file as we decode/encode it and release once done? I really like the access to an NSData* for an archive entry, and I use it to encrypt on at that level for security.

[pixelglow]

... can't we map a compliant window around each file as we decode/encode it and release once done? I really like the access to an NSData* for an archive entry, and I use it to encrypt on at that level for security.

Not a silly question at all!

For reading, we allow random access to individual entries, so it's not easy to predict when data is no longer needed. Using a memory map keeps this efficient since the system pager should dump parts of the map under memory pressure. The library already does do some wrapping with decryption/decompressing but the bottom layer of the stack is still backed by the memory map. Quite likely, I'll have to rewire that bottom layer to read straight from the zip file instead. The central directory entries can continue to live in the memory map, since we're necessarily limiting that to at most a 64K trailer.

[phoenix42]

That sounds pretty reasonable to me. It's a shame the mapping isn't more robust on the platform. It's the standard answer on windows for lots of complex stuff.

My encryption operates at the NSData level so that should be unaffected as long as you still pass that back. I have made a tiny change to expose the zip comment as I needed somewhere to put the encryption IV and salt but I can reintegrate that again!

[chrisze]

Glen, I can reproduce the problem on iPhone 5s and 6 Plus, when writing a new zip file with this code:

NSError *error = nil;
ZZArchive* archive = [[ZZArchive alloc] initWithURL:filePathURL options:@{ ZZOpenOptionsCreateIfMissingKey : @YES } error:&error];

NSMutableArray *entries = [[NSMutableArray alloc] init];
NSArray *fileList = [[NSFileManager defaultManager] contentsOfDirectoryAtPath:folderPath error:nil];

for (NSString *fileName in fileList)
{
    ZZArchiveEntry *entry = [ZZArchiveEntry archiveEntryWithFileName:fileName compress:NO dataBlock:^ NSData * (NSError **error)
    {
        NSString *path = [folderPath stringByAppendingPathComponent:fileName];
        return [NSData dataWithContentsOfMappedFile:path];
    }];

    [entries addObject:entry];
    [archive updateEntries:entries error:&error];

    if (error || self.isCancelled) return;
}

Note: This code runs inside the main function of NSOperation. Because updateEntries can take a long time, we call it after each iteration so that we can stop should the operation get canceled (I assume there isn't a better way to do it, is there?).

Anyhow, this sample reproduces the problem consistently with about 100 images (~2-4 MB each).

[pixelglow]

@chrisze While you can update entries in a loop like that, it's pretty inefficient. In particular, every time through the loop you are compressing/storing the previous entries all over again, which adds to the slowness. You really have two choices:

• In the loop, compose the entire entries. After the loop, call updateEntries:error once only. However if there's an error, it will just atomically abort i.e. the archive remains unchanged.
• In the loop, get the archive's current entries, make a new array by adding your new entry, then call updateEntries:error: with that array. You could arrange to call updateEntries:error: in batches to balance between efficiency and safety.

Why is the 2nd choice more efficient than your code? If the array passed to updateEntries:error: has existing entries from the archive in front, it will skip writing those entries out again.

[chrisze]

Thanks @pixelglow. That's a good idea. It would be awesome to have a method that can take just one entry and can be efficiently called in a loop. If you are archiving large number of files, it is important to be able to cancel the operation.

[phoenix42]

What's the ETA for a fix for this? Is there anything else I can do to help?