Potential memory leak on Apple silicon (3gb fitz.Pixmap cache)

[paultheman]

Please provide all mandatory information!

Describe the bug (mandatory)

A clear and concise description of what the bug is.

I have noticed that upon calling the following function there is a memory leak that grows to about 3gb and not more.
The fitz.Tools().store_size does not go over 256mb, see the screenshot below.

If fitz.Tools().store_shrink(10) is called the problem is fixed. (any argument will do...)

def create_img_from_pdf(self) -> Image.Image:
        """ Converts a page from a PDF doc to PIL.Image
        """

        #fitz.Tools().store_shrink(10)

        pdf = fitz.Document(self.f_path)
        first_page:fitz.Page = pdf.load_page(self.current_page.get())
        pix:fitz.Pixmap = first_page.get_pixmap()
        pdf.close()

        print(f'store_size -> {fitz.Tools().store_size}')

        return Image.frombytes("RGB", [pix.width, pix.height], pix.samples)

To Reproduce (mandatory)

Explain the steps to reproduce the behavior, For example, include a minimal code snippet, example files, etc.

• Create multiple fitz.Pixmap elements and see that the memory consumption goes over 256mb.

For problems when building or installing PyMuPDF, give the full output of the build/install command so that, for example, all pip/compiler/linker errors/warnings can be seen.

Expected behavior (optional)

Describe what you expected to happen (if not obvious).

• The maximum memory consumption should be 256mb, as per documentation https://pymupdf.readthedocs.io/en/latest/tools.html#Tools.store_size

Screenshots (optional)

If applicable, add screenshots to help explain your problem.

Your configuration (mandatory)

• Operating system, potentially version and bitness
• Python version, bitness
• PyMuPDF version, installation method (wheel or generated from source).
  For example, the output of print(sys.version, "\n", sys.platform, "\n", fitz.__doc__) would be sufficient (for the first two bullets).

3.11.3 (main, Apr 7 2023, 20:13:31) [Clang 14.0.0 (clang-1400.0.29.202)]
darwin
PyMuPDF 1.22.3: Python bindings for the MuPDF 1.22.0 library.
Version date: 2023-05-10 00:00:01.
Built for Python 3.11 on darwin (64-bit).

Additional context (optional)

Add any other context about the problem here.

[JorjMcKie]

The fitz.TOOLS.store_size is the internal cache size of the base library MuPDF.
This is independent from the Python memory occupied by the app.
Your size displays (so far) show a maximum of below 230 MB. In so far, the assertions of the documentation are met.

The fact that you can regain stroage at all (via shrinking store) means that memory is recoverable, there just was no reason yet to free it.
This is not a leak.

I tried a few things like this one:

def make_img(doc):
    for page in doc:
        pix=page.get_pixmap()
        img= Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
        print(page.number, round(fitz.TOOLS.store_size/1024/1024,2))

This function I called with larger files like the Adobe manuals. Also increased pixmap DPI etc.
I never saw cache store sizes even close to 256 MB - even after processing thousands of pages.

[paultheman]

Thank you for taking the time to answer.
I believed that there was a memory leak because of the huge memory consumption difference without and with fitz.TOOLS.store_shrink.

    def create_img_from_pdf(self) -> Image.Image:
        """ Converts a page from a PDF doc to PIL.Image
        """

        # fitz.TOOLS.store_shrink(10)
        pdf = fitz.Document(self.f_path)
        first_page:fitz.Page = pdf.load_page(self.current_page.get())
        pix:fitz.Pixmap = first_page.get_pixmap()
        pdf.close()
        print(round(fitz.TOOLS.store_size/1024/1024,2))

        return Image.frombytes("RGB", [pix.width, pix.height], pix.samples)

Running the function with fitz.TOOLS.store_shrink(10) leads to a maximum Python memory consumption of around 100mb.

Without fitz.TOOLS.store_shrink(10) Python managed to get to a maximum memory consumption of 3.8gb.

[JorjMcKie]

@paultheman - oops, what made you close this?

[paultheman]

Sorry, i thought it was not a bug being related to python memory consumption.

[JorjMcKie]

I have changed my mind:
Let's move this to "Discussions". There is a lack of explanaining this satisfactorily - which is not an issue per se.