Memory issues #396

mxmlnkn · 2017-03-08T11:26:37Z

Wasn't sure whether to open 3 issues for these.

I want to convert 119 700x950px jpg images each 81K large into a gif, so in total 9.5M. But when I try to do that I get

convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `000081.jpg' @ error/cache.c/OpenPixelCache/3945.
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `000082.jpg' @ error/cache.c/OpenPixelCache/3945.
...
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `000118.jpg' @ error/cache.c/OpenPixelCache/3945.
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `000119.jpg' @ error/cache.c/OpenPixelCache/3945.

Meaning it fails after the 80th image for some reason.
So in my opinion a first performance bug is already, that ImageMagick seems to cache all images first instead of just streaming them. Also why does this exceed my memory limit in the first place. It seems to cache them uncompressed.

Looking at the limits I get:

identify -list resource
    Resource limits:
      Width: 16KP
      Height: 16KP
      Area: 128MP
      Memory: 256MiB
      Map: 512MiB
      Disk: 1GiB
      File: 768
      Thread: 4
      Throttle: 0
      Time: unlimited

So another problem I have is that for some reason memory is limited to 256MiB, but I have 16GiB in total and 10GiB free:

free -m
                  total        used        free      shared  buff/cache   available
    Mem:          15935        5265        2154         122        8515       10418
    Swap:             0           0           0

Also I have a bit more disk space

df -h
    Filesystem              Size  Used Avail Use% Mounted on
    /dev/sdc1                95G   78G   13G  87% /

So I tried setting it higher with an old code snippet I found here

identify -limit memory 8000mb -list resource
    Resource limits:
      Width: 16KP
      Height: 16KP
      Area: 128MP
      Memory: 7.81KiB
      Map: 512MiB
      Disk: 1GiB
      File: 768
      Thread: 4
      Throttle: 0
      Time: unlimited

So instead of setting it to 8000 MB like I intended it sets it to 8000 Byte. In my opinion the unit specifier shouldn't be case-sensitive. And even if you want to enforce a case-sensitive CLI, there should be a warning or even error message if the unit specifier could not be recognized!

It works in uppercase:

identify -limit memory 80MB -list resource
    Resource limits:
      Memory: 76.29MiB

BUT trying to set it higher than the default 256MiB does not work for some weird reason which is the most pressing bug for me:

identify -limit memory 8000MiB -list resource
    Resource limits:
      Memory: 256MiB

The text was updated successfully, but these errors were encountered:

mikayla-grace · 2017-03-08T12:07:02Z

See https://www.imagemagick.org/script/architecture.php#tera-pixel. You likely want to use the registry:temporary-path define and point to an area on disk with lots of free space. See https://www.imagemagick.org/script/architecture.php#cache to learn more about how the pixel cache works. See https://www.imagemagick.org/script/security-policy.php to discover how your security policy may override your command-line resource settings.

mxmlnkn · 2017-03-08T13:13:39Z

Thank you, increasing the memory limit in /etc/ImageMagic-6/policy.xml works for me.

soubhikchatterjee · 2017-08-02T05:12:04Z

For anyone who is still facing this problem even after increasing the memory limit, try commenting out the limit tags for policymap. Eg:

<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <!-- <policy domain="resource" name="memory" value="2GiB"/> -->
  <!-- <policy domain="resource" name="map" value="4GiB"/> -->
  <!-- <policy domain="resource" name="area" value="1GB"/> -->
  <!-- <policy domain="resource" name="disk" value="16EB"/> -->
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <!-- <policy domain="resource" name="thread" value="4"/> -->
  <!-- <policy domain="resource" name="throttle" value="0"/> -->
  <!-- <policy domain="resource" name="time" value="3600"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->
  <policy domain="cache" name="shared-secret" value="passphrase"/>
  <policy domain="coder" rights="none" pattern="EPHEMERAL" />
  <policy domain="coder" rights="none" pattern="URL" />
  <policy domain="coder" rights="none" pattern="HTTPS" />
  <policy domain="coder" rights="none" pattern="MVG" />
  <policy domain="coder" rights="none" pattern="MSL" />
  <policy domain="coder" rights="none" pattern="TEXT" />
  <policy domain="coder" rights="none" pattern="SHOW" />
  <policy domain="coder" rights="none" pattern="WIN" />
  <policy domain="coder" rights="none" pattern="PLT" />
  <policy domain="path" rights="none" pattern="@*" />
</policymap>

coaxial · 2017-09-04T01:55:48Z

For some reason, my ImageMagick came with very low limits:

Resource limits:
  Width: 16KP
  Height: 16KP
  Area: 128MP
  Memory: 256MiB
  Map: 512MiB
  Disk: 1GiB
  File: 768
  Thread: 4
  Throttle: 0
  Time: unlimited

I changed /etc/ImageMagick-6/policy.xml to more sensible defaults:

Resource limits:
  Width: 128KP
  Height: 128KP
  Area: 1.0737GP
  Memory: 2GiB
  Map: 4GiB
  Disk: 8GiB
  File: 768
  Thread: 4
  Throttle: 0
  Time: unlimited

and it doesn't choke anymore with the new limits when I try to manipulate more than a few dozen scanned pages.

Bzzz · 2018-09-23T19:31:17Z

Indeed very helpful to get an MP4 converted into single PNGs. The error message could be a bit more precise, though...IM should know which limit has been reached and needs to be increased for a successful run.

Ortega-Dan · 2019-02-15T14:31:45Z

For some reason, my ImageMagick came with very low limits:
Resource limits:
  Width: 16KP
  Height: 16KP
  Area: 128MP
  Memory: 256MiB
  Map: 512MiB
  Disk: 1GiB
  File: 768
  Thread: 4
  Throttle: 0
  Time: unlimited
I changed /etc/ImageMagick-6/policy.xml to more sensible defaults:
Resource limits:
  Width: 128KP
  Height: 128KP
  Area: 1.0737GP
  Memory: 2GiB
  Map: 4GiB
  Disk: 8GiB
  File: 768
  Thread: 4
  Throttle: 0
  Time: unlimited
and it doesn't choke anymore with the new limits when I try to manipulate more than a few dozen scanned pages.

Thank you so much

ORESoftware · 2019-04-10T22:26:13Z

HOW TO CONVERT TO HIGHER MEMORY USING THE COMMAND LINE NOT CONFIG FILE?

@mxmlnkn

Bzzz · 2019-04-11T07:15:02Z

HOW TO POLITELY ASK QUESTIONS ON THE INTERWEBS?

lee-tts · 2019-04-19T05:31:27Z

Given this has been necro'd anyway, I will add a friendly warning:

The answer given by soubhikchatterjee above to comment out the policy lines is very effective—so much so that your system might totally lock up if you exceed resource capacity.

mxmlnkn · 2019-04-19T15:31:13Z

Given this has been necro'd anyway, I will add a friendly warning:

The answer given by soubhikchatterjee above to comment out the policy lines is very effective—so much so that your system might totally lock up if you exceed resource capacity.

An out-of-memory killer will prevent this if installed.

lee-tts · 2019-04-20T07:07:39Z

An out-of-memory killer will prevent this if installed.

I am on Ubuntu which doesn't really have a working oom killer as far as I can tell looking at answers on SE. Your comment did get me to dig into that though and I've hopefully set it to be slightly more functional/aggressive (setting vm.oom_kill_allocating_task=1 — hopefully that's right).

jidanni · 2019-05-31T04:04:13Z

It would be great, if the message shown to the user, would go on to say
"Try ...."

Not only would it save the user hours of searching for the answer,
if what it tells the user to try also fails, then he knows it is a bug.

EndlessCheng · 2019-06-12T08:00:34Z

For anyone who is still facing this problem even after increasing the memory limit, try commenting out the limit tags for policymap. Eg:

<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <!-- <policy domain="resource" name="memory" value="2GiB"/> -->
  <!-- <policy domain="resource" name="map" value="4GiB"/> -->
  <!-- <policy domain="resource" name="area" value="1GB"/> -->
  <!-- <policy domain="resource" name="disk" value="16EB"/> -->
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <!-- <policy domain="resource" name="thread" value="4"/> -->
  <!-- <policy domain="resource" name="throttle" value="0"/> -->
  <!-- <policy domain="resource" name="time" value="3600"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->

Remove all of these would be a bad idea.

The main problem is the area config, which means "Use up to _ disk space before failure"

(memory means "Use up to _ of memory before doing mmap" and map means "Use up to _ mmaps before caching to disk")

nmz787 · 2019-12-22T22:12:59Z

Why is display trying to use disk space anyway??? I'm trying to load a 7GB PBM image, about 59GigaPixels, and commenting out the policy lines indeed removes the error... but I see that my disk space is running out. I can easily delete disk items, but I've got 64GB RAM... so I'm confused why my disk is being used at all.

mos790 · 2020-03-21T10:53:49Z

For me the different steps works :

open terminal : sudo nano /etc/ImageMagick-6/policy.xml
Enter your password
You should be in nano file, change the line with "ressource memory" to
<policy domain="resource" name="memory" value="5000MiB"/>
Save the nano file : ctrl + o and enter to confirm
Up the memory if necessary

Hope it will help !

https://blog.bigbinary.com/2018/09/12/configuring-memory-allocation-in-imagemagick.html ImageMagick/ImageMagick#396

liar666 · 2020-03-28T12:17:28Z

Hi there,

Encountered the problem, increased limits as advised here... But now I get
libgomp: Thread creation failed: Resource temporarily unavailable

Any ideas ?

urban-warrior · 2020-03-28T12:28:52Z

Speculation, but it looks like you may be running out of memory. Reduce your cache memory limit by policy, environment variable, or command-line option as appropriate. This will force the pixel cache on disk and spare the memory for system tasks such as thread creation. See https://imagemagick.org/script/architecture.php#tera-pixel.

DethRaid · 2021-03-20T23:22:09Z

Thank god for this thread! Neither the ImageMagick error message and the ImageMagick docs even implied that there's a policy.xml file that silently prevents me from setting the resource limits "too high"

fmw42 · 2021-03-20T23:35:26Z

See site map to find https://imagemagick.org/script/resources.php and in it https://imagemagick.org/source/policy.xml. However, if you are on a shared system, only the owner of the system who has root privileges can make the limits higher. You can restrict them, but not relax them.

dlemstra closed this as completed Mar 16, 2017

ianzitow mentioned this issue Apr 12, 2018

convert-im6.q16: cache resources exhausted vvo/gifify#99

Closed

ukos-git mentioned this issue Jul 1, 2018

memory for imagemagick usnish/biochemical-pathways-poster#5

Open

srussking mentioned this issue Aug 30, 2018

Large .psb files VoidVolker/MagickSlicer#8

Open

gjost mentioned this issue Nov 8, 2018

Access files not made for large files denshoproject/ddr-cmdln#111

Open

stefanct mentioned this issue Jan 15, 2019

Allowed values of temporary-path #1449

Closed

stemiwe mentioned this issue Mar 13, 2019

import fails because of imagemagicks memory limit academic-moodle-cooperation/moodle-mod_offlinequiz#96

Closed

eliotjordan mentioned this issue May 6, 2019

Increase memory limits for ImageMagick pulibrary/figgy#2979

Closed

eliotjordan mentioned this issue Oct 11, 2019

ImageMagick resource limit values are not respected pulibrary/princeton_ansible#924

Closed

niklashhh mentioned this issue Feb 13, 2020

Cache resources exhausted when converting to gif asciinema/asciicast2gif#74

Closed

jcsrb mentioned this issue Mar 12, 2020

Running out of Memory on ImageMagick convert step stop quietly (code 0) axa-group/Parsr#347

Closed

bseeger mentioned this issue Mar 20, 2020

Ensure ImageMagick limits are configured correctly jhu-idc/iDC-general#34

Closed

pdelaby added a commit to intoo-solutions/imgpush that referenced this issue Mar 23, 2020

Tried to solve cache problem by increasing values of ImgMagick

3c37789

https://blog.bigbinary.com/2018/09/12/configuring-memory-allocation-in-imagemagick.html ImageMagick/ImageMagick#396

pdelaby added a commit to intoo-solutions/imgpush that referenced this issue Mar 23, 2020

Tried to solve cache problem by increasing values of ImgMagick

6245092

https://blog.bigbinary.com/2018/09/12/configuring-memory-allocation-in-imagemagick.html ImageMagick/ImageMagick#396

pdelaby mentioned this issue Mar 26, 2020

Cache fix hauxir/imgpush#14

Merged

satyamtg mentioned this issue Jun 8, 2020

Cache resources exhausted error with Imagemagick openzim/sotoki#147

Closed

sbs20 mentioned this issue Nov 24, 2020

Convert: cache resources exhausted on large images sbs20/scanservjs#97

Closed

thasmo mentioned this issue Nov 25, 2020

Resizing does not work on certain settings xperseguers/t3ext-image_autoresize#47

Closed

VoidVolker mentioned this issue Mar 11, 2021

cache resources exhausted VoidVolker/MagickSlicer#13

Closed

derneuere mentioned this issue Mar 21, 2021

Handle thumbnail creation of video files correctly LibrePhotos/librephotos#211

Closed

BenjaminHabert mentioned this issue Oct 26, 2021

WIP: Prepare data ramp-kits/follicles_detection#4

Merged

6 tasks

sbs20 mentioned this issue Mar 21, 2022

Batch scan with many pages crashed while saving sbs20/scanservjs#414

Closed

radar3301 mentioned this issue Sep 8, 2022

Inconsistent memory issues per execution #5541

Closed

iamtekson mentioned this issue May 9, 2023

convert tool shipped with ASP is buggy NeoGeographyToolkit/StereoPipeline#384

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory issues #396

Memory issues #396

mxmlnkn commented Mar 8, 2017

mikayla-grace commented Mar 8, 2017

mxmlnkn commented Mar 8, 2017

soubhikchatterjee commented Aug 2, 2017

coaxial commented Sep 4, 2017 •

edited

Bzzz commented Sep 23, 2018

Ortega-Dan commented Feb 15, 2019

ORESoftware commented Apr 10, 2019 •

edited

Bzzz commented Apr 11, 2019

lee-tts commented Apr 19, 2019

mxmlnkn commented Apr 19, 2019

lee-tts commented Apr 20, 2019 •

edited

jidanni commented May 31, 2019

EndlessCheng commented Jun 12, 2019 •

edited

nmz787 commented Dec 22, 2019

mos790 commented Mar 21, 2020

liar666 commented Mar 28, 2020

urban-warrior commented Mar 28, 2020

DethRaid commented Mar 20, 2021

fmw42 commented Mar 20, 2021

Memory issues #396

Memory issues #396

Comments

mxmlnkn commented Mar 8, 2017

mikayla-grace commented Mar 8, 2017

mxmlnkn commented Mar 8, 2017

soubhikchatterjee commented Aug 2, 2017

coaxial commented Sep 4, 2017 • edited

Bzzz commented Sep 23, 2018

Ortega-Dan commented Feb 15, 2019

ORESoftware commented Apr 10, 2019 • edited

Bzzz commented Apr 11, 2019

lee-tts commented Apr 19, 2019

mxmlnkn commented Apr 19, 2019

lee-tts commented Apr 20, 2019 • edited

jidanni commented May 31, 2019

EndlessCheng commented Jun 12, 2019 • edited

nmz787 commented Dec 22, 2019

mos790 commented Mar 21, 2020

liar666 commented Mar 28, 2020

urban-warrior commented Mar 28, 2020

DethRaid commented Mar 20, 2021

fmw42 commented Mar 20, 2021

coaxial commented Sep 4, 2017 •

edited

ORESoftware commented Apr 10, 2019 •

edited

lee-tts commented Apr 20, 2019 •

edited

EndlessCheng commented Jun 12, 2019 •

edited