Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory issues #396

Closed
mxmlnkn opened this Issue Mar 8, 2017 · 6 comments

Comments

Projects
None yet
7 participants
@mxmlnkn
Copy link

mxmlnkn commented Mar 8, 2017

Wasn't sure whether to open 3 issues for these.

I want to convert 119 700x950px jpg images each 81K large into a gif, so in total 9.5M. But when I try to do that I get

convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `000081.jpg' @ error/cache.c/OpenPixelCache/3945.
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `000082.jpg' @ error/cache.c/OpenPixelCache/3945.
...
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `000118.jpg' @ error/cache.c/OpenPixelCache/3945.
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `000119.jpg' @ error/cache.c/OpenPixelCache/3945.

Meaning it fails after the 80th image for some reason.
So in my opinion a first performance bug is already, that ImageMagick seems to cache all images first instead of just streaming them. Also why does this exceed my memory limit in the first place. It seems to cache them uncompressed.

Looking at the limits I get:

identify -list resource
    Resource limits:
      Width: 16KP
      Height: 16KP
      Area: 128MP
      Memory: 256MiB
      Map: 512MiB
      Disk: 1GiB
      File: 768
      Thread: 4
      Throttle: 0
      Time: unlimited

So another problem I have is that for some reason memory is limited to 256MiB, but I have 16GiB in total and 10GiB free:

free -m
                  total        used        free      shared  buff/cache   available
    Mem:          15935        5265        2154         122        8515       10418
    Swap:             0           0           0

Also I have a bit more disk space

df -h
    Filesystem              Size  Used Avail Use% Mounted on
    /dev/sdc1                95G   78G   13G  87% /

So I tried setting it higher with an old code snippet I found here

identify -limit memory 8000mb -list resource
    Resource limits:
      Width: 16KP
      Height: 16KP
      Area: 128MP
      Memory: 7.81KiB
      Map: 512MiB
      Disk: 1GiB
      File: 768
      Thread: 4
      Throttle: 0
      Time: unlimited

So instead of setting it to 8000 MB like I intended it sets it to 8000 Byte. In my opinion the unit specifier shouldn't be case-sensitive. And even if you want to enforce a case-sensitive CLI, there should be a warning or even error message if the unit specifier could not be recognized!

It works in uppercase:

identify -limit memory 80MB -list resource
    Resource limits:
      Memory: 76.29MiB

BUT trying to set it higher than the default 256MiB does not work for some weird reason which is the most pressing bug for me:

identify -limit memory 8000MiB -list resource
    Resource limits:
      Memory: 256MiB
@mikayla-grace

This comment has been minimized.

Copy link

mikayla-grace commented Mar 8, 2017

See https://www.imagemagick.org/script/architecture.php#tera-pixel. You likely want to use the registry:temporary-path define and point to an area on disk with lots of free space. See https://www.imagemagick.org/script/architecture.php#cache to learn more about how the pixel cache works. See https://www.imagemagick.org/script/security-policy.php to discover how your security policy may override your command-line resource settings.

@mxmlnkn

This comment has been minimized.

Copy link
Author

mxmlnkn commented Mar 8, 2017

Thank you, increasing the memory limit in /etc/ImageMagic-6/policy.xml works for me.

@dlemstra dlemstra closed this Mar 16, 2017

@soubhikchatterjee

This comment has been minimized.

Copy link

soubhikchatterjee commented Aug 2, 2017

For anyone who is still facing this problem even after increasing the memory limit, try commenting out the limit tags for policymap. Eg:

<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <!-- <policy domain="resource" name="memory" value="2GiB"/> -->
  <!-- <policy domain="resource" name="map" value="4GiB"/> -->
  <!-- <policy domain="resource" name="area" value="1GB"/> -->
  <!-- <policy domain="resource" name="disk" value="16EB"/> -->
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <!-- <policy domain="resource" name="thread" value="4"/> -->
  <!-- <policy domain="resource" name="throttle" value="0"/> -->
  <!-- <policy domain="resource" name="time" value="3600"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->
  <policy domain="cache" name="shared-secret" value="passphrase"/>
  <policy domain="coder" rights="none" pattern="EPHEMERAL" />
  <policy domain="coder" rights="none" pattern="URL" />
  <policy domain="coder" rights="none" pattern="HTTPS" />
  <policy domain="coder" rights="none" pattern="MVG" />
  <policy domain="coder" rights="none" pattern="MSL" />
  <policy domain="coder" rights="none" pattern="TEXT" />
  <policy domain="coder" rights="none" pattern="SHOW" />
  <policy domain="coder" rights="none" pattern="WIN" />
  <policy domain="coder" rights="none" pattern="PLT" />
  <policy domain="path" rights="none" pattern="@*" />
</policymap>
@coaxial

This comment has been minimized.

Copy link

coaxial commented Sep 4, 2017

For some reason, my ImageMagick came with very low limits:

Resource limits:
  Width: 16KP
  Height: 16KP
  Area: 128MP
  Memory: 256MiB
  Map: 512MiB
  Disk: 1GiB
  File: 768
  Thread: 4
  Throttle: 0
  Time: unlimited

I changed /etc/ImageMagick-6/policy.xml to more sensible defaults:

Resource limits:
  Width: 128KP
  Height: 128KP
  Area: 1.0737GP
  Memory: 2GiB
  Map: 4GiB
  Disk: 8GiB
  File: 768
  Thread: 4
  Throttle: 0
  Time: unlimited

and it doesn't choke anymore with the new limits when I try to manipulate more than a few dozen scanned pages.

@Bzzz

This comment has been minimized.

Copy link

Bzzz commented Sep 23, 2018

Indeed very helpful to get an MP4 converted into single PNGs. The error message could be a bit more precise, though...IM should know which limit has been reached and needs to be increased for a successful run.

@Ortega-Dan

This comment has been minimized.

Copy link

Ortega-Dan commented Feb 15, 2019

For some reason, my ImageMagick came with very low limits:

Resource limits:
  Width: 16KP
  Height: 16KP
  Area: 128MP
  Memory: 256MiB
  Map: 512MiB
  Disk: 1GiB
  File: 768
  Thread: 4
  Throttle: 0
  Time: unlimited

I changed /etc/ImageMagick-6/policy.xml to more sensible defaults:

Resource limits:
  Width: 128KP
  Height: 128KP
  Area: 1.0737GP
  Memory: 2GiB
  Map: 4GiB
  Disk: 8GiB
  File: 768
  Thread: 4
  Throttle: 0
  Time: unlimited

and it doesn't choke anymore with the new limits when I try to manipulate more than a few dozen scanned pages.

Thank you so much

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.