EXC_RESOURCE (RESOURCE_TYPE_MEMORY: high watermark memory limit exceeded) (limit=30 MB) from inside an iOS Widget extension

[acecilia]

Hi 👋

I am using this library to decompress some selected files that have a specific prefix from an archive. I an doing it from inside a Widget extension, in iOS. The total decompressed size of the files to decompress is 4MB, the decompressed archive is 120MB, and the archive itself is 10MB. The decompression code is as follows:

func decompress(prefix: String, from archiveUrl: URL, to outputUrl: URL) throws {
    let archivePath = try Path(archiveUrl.path)
    let archivePathInStream = try InStream(path: archivePath)

    let decoder = try Decoder(stream: archivePathInStream, fileType: .sevenZ, delegate: self)

    guard try decoder.open() else {
        throw PLzmaServiceError.couldNotOpenDecoder
    }

    let numberOfArchiveItems = try decoder.count()
    let selectedItemsDuringIteration = try ItemArray(capacity: numberOfArchiveItems)

    for itemIndex in 0..<numberOfArchiveItems {
        let item = try decoder.item(at: itemIndex)
        let path = try String(describing: item.path())
        if path.hasPrefix("\(prefix)/") {
            try selectedItemsDuringIteration.add(item: item)
        }
    }

    let extracted = try decoder.extract(
        items: selectedItemsDuringIteration,
        to: try Path(outputUrl.path)
    )

    guard extracted else {
        throw PLzmaServiceError.couldNotExtract
    }
}

During that operations I get a runtime exception saying EXC_RESOURCE (RESOURCE_TYPE_MEMORY: high watermark memory limit exceeded) (limit=30 MB). Seems like the Widget extension has some memory limits enforced by iOS, and the decompression is exceeding them. As a result, the OS kills the process in the middle of the decompression operation.

Is there any way to configure the decompression so it uses less memory?

Thanks!

[OlehKulykov]

Hi,
first of all, thank you for a informative description.
As for 'configure the decompression so it uses less memory' - yes, it's possible to set the minimum size of memory buffers for IO before decoder creation.
Example:

PLzmaSDK.decoderReadSize = 1024
PLzmaSDK.decoderWriteSize = 1024
PLzmaSDK.streamReadSize = 1024
PLzmaSDK.streamWriteSize = 1024

Also, you can disable thread-safety by placing #define LIBPLZMA_THREAD_UNSAFE 1 at the top of libplzma.h file or globaly for POD.

But, all of this will not fully help you. Cause the biggest ammount of memory is archive's dictionary. And to reduce the size of dictionary, you need to create/use archive with compression level <= 3.
Example: 7za a -mx3 test.7z install-arm64-minimal-20230910T213208Z.iso
Please refer to: https://documentation.help/7-Zip/method.htm 'x=[0 | 1 | 5 | 7 | 9 ] Sets level of compression of 7z. There you'll find dictionary sizes.

[acecilia]

Thanks for detailed context @OlehKulykov 🙏 Just so you know, I have very limited knowledge about compression, so some of my questions may be off.

As for 'configure the decompression so it uses less memory' - yes, it's possible to set the minimum size of memory buffers for IO before decoder creation.

Is there any calculation I can use to understand what is contributing to the memory consumption? Example:

• For an archive of size X, using dictionary size of Y, the memory consumption during decompression can be estimated as <some formula> = Z MB

Also, you can disable thread-safety by placing #define LIBPLZMA_THREAD_UNSAFE 1 at the top of libplzma.h file or globaly for POD.

How does thread safety affect memory consumption? How are those two related?

But, all of this will not fully help you. Cause the biggest ammount of memory is archive's dictionary. And to reduce the size of dictionary, you need to create/use archive with compression level <= 3.
Example: 7za a -mx3 test.7z install-arm64-minimal-20230910T213208Z.iso
Please refer to: https://documentation.help/7-Zip/method.htm 'x=[0 | 1 | 5 | 7 | 9 ] Sets level of compression of 7z. There you'll find dictionary sizes.

Thanks! I am currently using level 9, as I need to have the smallest size possible. Will play with the compression level value (the main problem I see is that reducing the compression level results in a bigger archive)

Couple more questions if you can 🙏

• Would the usage of a file OutStream help in this case? (I already tried to set it up, but did not manage yet because I could not find any example of it. All examples of streams I could find in the repository are using memory streams, not file streams)
• For my usecase, I would like to maintain the compression level so the archive size is as small as possible. I could accept a slower decompression on exchange for less memory consumption, is there a way to configure this?

[OlehKulykov]

Hi,
Q/A:

1. Is there any calculation I can use to understand what is contributing to the memory consumption? Example:

• Almost no. You should take the ammount of memory for dictionary depending on selected compression level plus avarange memory for your UI code, ext.

2. How does thread safety affect memory consumption? How are those two related?

• Less code -> less memory. Mutexes are class fields, fileds/mutexes & processing code could be ignored/disabled by preprocessor definition & not compiled into the app binary.
  No class field -> no extra memory for the whole class. No processing code -> less size of the app binary -> less memory to run the app.

3. Yes, lower compression level -> smaller dictionary -> less memory to decompress -> bigger size of the archive, no matter which type of archive.

4. OutStream. Out'File'Stream -> write decoded/decompressed parts directly to physical file. Out'Memory'Stream -> write decoded/decompressed parts to heap memory.
   'Public interface' for streams is same. The difference in a way of storing the content of stream.
   P.S.: The design of 'public interface' allows even to write decoded/decompressed parts to HTTPs, FTP, etc. via client code & custom implementation.

5. 'I would like to maintain the compression level so the archive size is as small as possible'.

• You can Google a bit: https://www.google.com/search?q=compression+algorithm+without+dictionary
• Or try to update your task/needs to a single file inside an 'arhive', something like tar.xz or your own format based on on a single compressed data/stream (NOT an arhive with files, dirs inside).

Good luck.

[OlehKulykov]

P.S.: Some info about this SDK.
The main programming language is C++, both patched original 7-zip SDK and code 'cross platform way how to use it' - this SDK.
Since C++ libraries are a huge pain ..., there is a 'C language bindings' - all functionality available in pure C via extra code, i.e. C bindings.

Reminder from README: the structure of SDK for iOS looks like:

• Main C++ implementation -> C bindings -> Swift
• Main C++ implementation -> Objective-C

In case of a very limited memory, depending on your skills(I don't know), for your specific needs, you can implement next:

1. Decide what do you need. From the comments you have to decode/extract some archive items by prefix from a big archive which contains 'something interesting/needed for you' & rest.
2. As I see, the app's language is Swift.
3. Make a local version of Cocoapod or local package for Apple's Package Manger. It's MIT - everything is fine.
4. Remove from local version of POD | APM all swift and Obj-C code. Keep in mind reference.
   4.1. !!! At this point you are using the only 'libplzma.a'. My recommendation is: use static lib to avoid unused code!!!.
5. Disable TAR & thread-safety via preprocessor definition 'LIBPLZMA_NO_TAR=1' -> '#define LIBPLZMA_NO_TAR 1', 'LIBPLZMA_THREAD_UNSAFE=1' -> '#define LIBPLZMA_THREAD_UNSAFE 1'
   5.1. Keep in mind: less code -> less memory.
   5.2. You are good in C. Create your own Swift class/func for your specific needs & use directly lib's C bindings(take a look in Swift reference).
   5.3. You are good in C++. Create your own Obj-C class/func(*.mm)+Bridging, disable C bindings by preprocessor definition 'LIBPLZMA_NO_C_BINDINGS=1' -> '#define LIBPLZMA_NO_C_BINDINGS 1' (take a look in Obj-C reference).
6. In case of issues with SDK, please create an issue here in GitHub.
   6.1. In case of some specific/weird using of the SDK, please use email from public headers.

[acecilia]

Thank you so much for this great amount of context. It is very much enlightening, appreciate it a lot

4.1. !!! At this point you are using the only 'libplzma.a'. My recommendation is: use static lib to avoid unused code!!!.

Thanks! I am already doing this and linking PLzmaSDK statically 👍

I will try the alternatives you mentioned and write here the results when I have them 🙏