Encrypt/Decrypt large files will cause memory usage spike

[xianwen]

First of all, in the README, this line of code in "Async and Streams":

decryptor = [[RNEncryptor alloc] initWithSettings:kRNCryptorAES256Settings
                                         password:@"blah"
                                          handler: ...];

The variable name should be "encryptor".

Secondly, when trying out this code myself, I used a 1.9G huge file to test the memory issue. And the app will always crash due to memory spike.

I profiled it, and found that as the encryption goes, the memory usage keeps going up, never goes down.

After a closer look, I see that it's this call that increases memory usage every time for 48 KB:

-[_NSPlaceholderData initWithBytes:length:copy:deallocator:]

So, I added a symbolic breakpoint for this function, and found that it's called by RNCryptorEngine.m at line 93 via

return [buffer subdataWithRange:NSMakeRange(0, dataOutMoved)];

So, looks like when using the Async and Streams method in ReadMe to encrypt a file, the memory usage will keep going up due to unknown issues related to this line of code.

I'm not quite familiar with RNCryptor yet. Is this a bug of RNCryptor? How can I fix this issue

Thanks a lot!

[xianwen]

I also tried to enwrap the block inside an autoreleasepool, but it doesn't work either.

The following is the code I used, mostly the same as the Async and Streams section of ReadMe:

- (void) encryptFileAtPath: (NSString *)inputFilePath toPath: (NSString *)encryptedFilePath withKey: (NSString *)key
{
// Make sure that this number is larger than the header + 1 block.
// 33+16 bytes = 49 bytes. So it shouldn't be a problem.
int blockSize = 32 * 1024;

NSInputStream *inputStream = [NSInputStream inputStreamWithFileAtPath: inputFilePath];
NSOutputStream *encryptedStream = [NSOutputStream outputStreamToFileAtPath: encryptedFilePath append:NO];

[inputStream open];
[encryptedStream open];

// We don't need to keep making new NSData objects. We can just use one repeatedly.
__block NSMutableData *data = [NSMutableData dataWithLength:blockSize];
__block RNEncryptor *encryptor = nil;

dispatch_block_t readStreamBlock = ^{
    @autoreleasepool {
        [data setLength:blockSize];
        NSInteger bytesRead = [inputStream read:[data mutableBytes] maxLength:blockSize];
        if (bytesRead < 0) {
            // Throw an error
        }
        else if (bytesRead == 0) {
            [encryptor finish];
        }
        else {
            [data setLength:bytesRead];
            [encryptor addData:data];
            NSLog(@"Sent %ld bytes to encryptor", (unsigned long)bytesRead);
        }
    }
};

encryptor = [[RNEncryptor alloc] initWithSettings:kRNCryptorAES256Settings
                                         password:key
                                          handler:^(RNCryptor *cryptor, NSData *data) {
                                              @autoreleasepool {
                                                  NSLog(@"Encryptor recevied %ld bytes", (unsigned long)data.length);
                                                  [encryptedStream write:data.bytes maxLength:data.length];
                                                  if (cryptor.isFinished) {
                                                      [encryptedStream close];
                                                      // call my delegate that I'm finished with decrypting
                                                  }
                                                  else {
                                                      // Might want to put this in a dispatch_async(), but I don't think you need it.
                                                      readStreamBlock();
                                                  }
                                              }
                                          }];

// Read the first block to kick things off
readStreamBlock();
}

[carllindberg]

You may want to make that internal call dispatch_async(). I.e.:
dispatch_async(dispatch_get_main_queue(), readStreamBlock);
I think that allows the memory to be collected as it goes, at least in my quick test.

[carllindberg]

The other way would be to put in @autoreleasepool { ... } around the content in the dispatch_async() block in RNEncryptor's addData: method. It seems like with the above pattern, any objects autoreleased in there do not get released until the end of the process. But if the call to readStreamBlock() in the sample code above is made async (as the comment suggests it might be good to do), then the system autorelease pools get flushed more correctly.

[iPermanent]

I think you need to use NSFileHandler to encrypt the file data by pieces, you can encrypt the piece data by [NSFileHandler seekTo...] to get pieces of data, and encrypt them, then you can append it to a file, also you can try NSOutputStream to write file. If you get the whole data, it will out of memory with no doubt.

[rnapier]

I believe the Swift version has more reliable memory usage, and no longer has a misleading example.

[fagerbua]

For anyone coming in here via Google; I solved this problem by applying the semaphore solution suggested on Stack Overflow:
https://stackoverflow.com/questions/14481605/dispatch-queues-and-asynchronous-rncryptor/14571838#14571838

Using code like that my iPad mini successfully encrypted a 2 GB file in one minute, with no memory issues.