New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sequence not clearing from memory, sequence.destroy not working #237

Closed
Nertskull opened this Issue May 7, 2015 · 8 comments

Comments

4 participants
@Nertskull

Nertskull commented May 7, 2015

If I do this

for root, dirs, files in os.walk(myDir):
  for myFile in files:
    with Image(filename=myFile) as img:
      with Image(image=img) as main:
        print main.sequence[0].width

or even explicity destroy the image like this

for root, dirs, files in os.walk(myDir):
  for myFile in files:
    with Image(filename=myFile) as img:
      with Image(image=img) as main:
        first = True
        for frame in main.sequence:
           if first:
             print frame.width
             first = False
           frame.destroy()

I end up with images piling up in memory (and then on disk in /tmp) until i reach a segmentation fault. It appears the ImageSingle stuff is not getting cleared.

I posted about it here, in case that helps.:
https://stackoverflow.com/questions/30100956/python-wand-sequence-not-clearing-from-memory/30106428#30106428

@emcconville

This comment has been minimized.

Owner

emcconville commented May 8, 2015

Thanks Nertskull. Can you run the following, and post output?

python -mwand.version
@Nertskull

This comment has been minimized.

Nertskull commented May 8, 2015

Yeah, here it is.

0.4.0

@valdarin

This comment has been minimized.

valdarin commented May 15, 2015

I have also run into this issue recently. I spent a little bit of time troubleshooting and was able to narrow it down to something that is happening inside Sequence.__getitem__().

this is fine:

with Image(blob=blob) as img:
    print len(img.sequence)

but this will not free up the allocated memory:

with Image(blob=blob) as img:
    print img.sequence[0]

I can even specify that it only holds onto the memory for the specific sequence item that you interact with. If you have images in the sequence and you only examine one of them, it will free the rest up.

Sorry I am not able to provide any more detail. Hope that is useful.

@dahlia

This comment has been minimized.

Collaborator

dahlia commented May 15, 2015

Every single image in a sequence has to be accessed using with, as like Image. Have you tried this?:

with Image(blob=blob) as img:
    with img.sequence[0] as single_image:
        print single_image
@valdarin

This comment has been minimized.

valdarin commented May 15, 2015

Yea, we're using that way. I was trying to simplify for the sake of the example.

@dahlia dahlia added the bug label May 15, 2015

@emcconville

This comment has been minimized.

Owner

emcconville commented May 18, 2015

All (@Nertskull & @valdarin),

Please clone/checkout my development branch issue-237, and verify if the issue still exists.

Proposed Solution

When wand.image.Image.sequence is invoked, a low-level MagickCore routine was called to clone image data into a new instance. This solution deallocates a temporary resource after use, but I haven't gone over MagickWand source to verify if data in question is expected to be constant reference, or copied-value blob. So further testing will be needed.

Discovery

For folks wondering how I identified this, I used Apple's Instruments.app (a gui for ldb + batteries included.)

Using the example provided with Python 3.4 + ImageMagick 6.8.5 (both compiled with debug flags for backtrace & breakpoint support.)

Memory usage + leaks where visible
Memory leaks

The bt quickly identified CloneImage was the cause of memory consumption. A simple DestroyImage seems to work.

Memory usage after patch
fixed

Please help and verify.

@dahlia

This comment has been minimized.

Collaborator

dahlia commented May 18, 2015

Cool. 👍

dahlia added a commit that referenced this issue May 18, 2015

Merge pull request #239 from emcconville/issue-237
Proposed fix for issue #237 - Sequence not clearing from memory
@dahlia

This comment has been minimized.

Collaborator

dahlia commented May 18, 2015

#237 fixed this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment