Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wand doesn't free resources at all and eats all available memory #312

Closed
homm opened this issue Nov 5, 2016 · 11 comments
Closed

Wand doesn't free resources at all and eats all available memory #312

homm opened this issue Nov 5, 2016 · 11 comments
Milestone

Comments

@homm
Copy link
Contributor

@homm homm commented Nov 5, 2016

In [1]: from wand.image import Image

In [2]: for _ in range(500):
   ...:     im = Image(width=2560, height=1600)
   ...:     del im
   ...:     
python: unable to acquire cache view `No such file or directory' @ fatal/cache-view.c/AcquireAuthenticCacheView/121.

gc.collect() inside loop also doesn't help.

@homm
Copy link
Contributor Author

@homm homm commented Nov 5, 2016

In opposite, Pillow works very well and very fast:

In [1]: from PIL import Image
In [2]: %time for _ in range(500): Image.new('RGBA', (2560, 1600))

CPU times: user 1.96 s, sys: 16 ms, total: 1.98 s
Wall time: 1.98 s

@flux627
Copy link

@flux627 flux627 commented Mar 22, 2017

This bug is crippling my app right now. Did you ever find a fix?

@emcconville emcconville added this to the Wand 0.6.0 milestone Dec 19, 2019
@vegarsti
Copy link
Contributor

@vegarsti vegarsti commented Feb 28, 2020

This doesn't cause an error on my machine with Wand 0.5.9 and ImageMagick 7.0.9

@homm
Copy link
Contributor Author

@homm homm commented Feb 28, 2020

For some reason, rather than allocate memory, Wand now pollutes /tmp folder. Also, creating the empty image is only half-working now (it doesn't throw an error, but still leads to leads impossibility of further work). But the root of the problem the same: Wand doesn't free resources implicitly in the language with automatic memory management and garbage collection.

You can try this if you have a big enough image:

from wand.image import Image

for i in range(500):
    im = Image(filename='./imgs/Bangkok.jpg')
    im = None

@vegarsti
Copy link
Contributor

@vegarsti vegarsti commented Feb 28, 2020

How big does it need to be? Could you upload the image you used there?

@emcconville
Copy link
Owner

@emcconville emcconville commented Feb 29, 2020

Hey,

I've made a lot of progress with this issue, but haven't had free time to test thoroughly. If folks have experience with memory profiling, I'd love some help testing the 0.6-dev branch.

The allocation issues have been address, and garbage collection seems to behave as expected. It is still possible to run out of memory before gc has a chance to clean-up, so the with Image(...) as im context format is still preferred.

As for the /tmp artifacts -- The 0.6.0 release will implement Python's atexit routine, and will clean up the temporary files. I haven't been able to test, but intermediate files may still hang around if the process errors/warns, as ImageMagick is expecting the user to inspect frames/pages.

I can make a pull-request for the fixes if folks want to evaluate the code-changes, or make a beta package available if folks can dedicate some hardware time.

Ping me directly for more info, or comment here. 🤷‍♂️

@vegarsti
Copy link
Contributor

@vegarsti vegarsti commented Feb 29, 2020

That's great to hear! I'd love to help with a code review. I don't have any experience with memory management, though.

The 0.6.0 release will implement Python's atexit routine, and will clean up the temporary files.

Does that mean that everything in /tmp will be deleted, or just the files created by Wand?

@emcconville emcconville mentioned this issue Mar 1, 2020
@emcconville
Copy link
Owner

@emcconville emcconville commented Mar 1, 2020

The 0.6.0 release will implement Python's atexit routine, and will clean up the temporary files.

Does that mean that everything in /tmp will be deleted, or just the files created by Wand?

ImageMagick creates temporary files (not Wand), and it will only delete files created by it's own process.

@emcconville
Copy link
Owner

@emcconville emcconville commented May 13, 2020

Seems fix. It's still possible for folks to get into trouble, so ensure policy & resource configurations reflect hardware limits.

@prakhar2017
Copy link

@prakhar2017 prakhar2017 commented May 17, 2020

Is it fixed ?

@casperbh96
Copy link

@casperbh96 casperbh96 commented Nov 30, 2021

This is still an issue - but it is not Wand, it is ImageMagick. There is a requirement of 8 bytes per pixel in the Q16 build of ImageMagick - and it seems that ImageMagick only clears the previous memory after processing the next image. For example, I am processing big images that require 6600*5100*8*2 = 536 MB per page of a PDF. We multiply by two because ImageMagick does not clear the cache before having processed the second image.

However, there is a solution - albeit a bit slower. Use your disk to your advantage and store anything extra that cannot fit into memory in a temporary folder by the use of their policy file. REMEMBER to clear your temporary folder after every run (ImageMagick does not handle this).

Find more information about this:

  1. https://imagemagick.org/script/security-policy.php
  2. https://stackoverflow.com/a/26849250/7910473

Here is my example policy file that works for large images and PDFs:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE policymap [
  <!ELEMENT policymap (policy)+>
  <!ATTLIST policymap xmlns CDATA #FIXED ''>
  <!ELEMENT policy EMPTY>
  <!ATTLIST policy xmlns CDATA #FIXED '' domain NMTOKEN #REQUIRED
    name NMTOKEN #IMPLIED pattern CDATA #IMPLIED rights NMTOKEN #IMPLIED
    stealth NMTOKEN #IMPLIED value CDATA #IMPLIED>
]>
<policymap>
  <policy domain="resource" name="temporary-path" value="/tmp/imagemagick"/>
  <policy domain="resource" name="memory" value="1GiB"/>
  <policy domain="resource" name="map" value="512MiB"/>
  <policy domain="resource" name="width" value="16KP"/>
  <policy domain="resource" name="height" value="16KP"/>
  <policy domain="resource" name="area" value="128MB"/>
  <policy domain="resource" name="disk" value="5GiB"/>
- <policy domain="cache" name="synchronize" value="True"/>
  <policy domain="delegate" rights="none" pattern="URL" />
  <policy domain="delegate" rights="none" pattern="HTTPS" />
  <policy domain="delegate" rights="none" pattern="HTTP" />
  <!-- in order to avoid to get image with password text -->
  <policy domain="path" rights="none" pattern="@*"/>
  <!-- disable ghostscript format types -->
  <policy domain="coder" rights="none" pattern="PS" />
  <policy domain="coder" rights="none" pattern="PS2" />
  <policy domain="coder" rights="none" pattern="PS3" />
  <policy domain="coder" rights="none" pattern="EPS" />
  <policy domain="coder" rights="read|write" pattern="PDF" />
  <policy domain="coder" rights="none" pattern="XPS" />
</policymap>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants