Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sequence not clearing from memory, sequence.destroy not working #237

Closed
Nertskull opened this issue May 7, 2015 · 8 comments
Closed

Sequence not clearing from memory, sequence.destroy not working #237

Nertskull opened this issue May 7, 2015 · 8 comments
Labels
bug

Comments

@Nertskull
Copy link

@Nertskull Nertskull commented May 7, 2015

If I do this

for root, dirs, files in os.walk(myDir):
  for myFile in files:
    with Image(filename=myFile) as img:
      with Image(image=img) as main:
        print main.sequence[0].width

or even explicity destroy the image like this

for root, dirs, files in os.walk(myDir):
  for myFile in files:
    with Image(filename=myFile) as img:
      with Image(image=img) as main:
        first = True
        for frame in main.sequence:
           if first:
             print frame.width
             first = False
           frame.destroy()

I end up with images piling up in memory (and then on disk in /tmp) until i reach a segmentation fault. It appears the ImageSingle stuff is not getting cleared.

I posted about it here, in case that helps.:
https://stackoverflow.com/questions/30100956/python-wand-sequence-not-clearing-from-memory/30106428#30106428

@emcconville
Copy link
Owner

@emcconville emcconville commented May 8, 2015

Thanks Nertskull. Can you run the following, and post output?

python -mwand.version
@Nertskull
Copy link
Author

@Nertskull Nertskull commented May 8, 2015

Yeah, here it is.

0.4.0

@valdarin
Copy link

@valdarin valdarin commented May 15, 2015

I have also run into this issue recently. I spent a little bit of time troubleshooting and was able to narrow it down to something that is happening inside Sequence.__getitem__().

this is fine:

with Image(blob=blob) as img:
    print len(img.sequence)

but this will not free up the allocated memory:

with Image(blob=blob) as img:
    print img.sequence[0]

I can even specify that it only holds onto the memory for the specific sequence item that you interact with. If you have images in the sequence and you only examine one of them, it will free the rest up.

Sorry I am not able to provide any more detail. Hope that is useful.

@dahlia
Copy link
Collaborator

@dahlia dahlia commented May 15, 2015

Every single image in a sequence has to be accessed using with, as like Image. Have you tried this?:

with Image(blob=blob) as img:
    with img.sequence[0] as single_image:
        print single_image
@valdarin
Copy link

@valdarin valdarin commented May 15, 2015

Yea, we're using that way. I was trying to simplify for the sake of the example.

@dahlia dahlia added the bug label May 15, 2015
@emcconville
Copy link
Owner

@emcconville emcconville commented May 18, 2015

All (@Nertskull & @valdarin),

Please clone/checkout my development branch issue-237, and verify if the issue still exists.

Proposed Solution

When wand.image.Image.sequence is invoked, a low-level MagickCore routine was called to clone image data into a new instance. This solution deallocates a temporary resource after use, but I haven't gone over MagickWand source to verify if data in question is expected to be constant reference, or copied-value blob. So further testing will be needed.

Discovery

For folks wondering how I identified this, I used Apple's Instruments.app (a gui for ldb + batteries included.)

Using the example provided with Python 3.4 + ImageMagick 6.8.5 (both compiled with debug flags for backtrace & breakpoint support.)

Memory usage + leaks where visible
Memory leaks

The bt quickly identified CloneImage was the cause of memory consumption. A simple DestroyImage seems to work.

Memory usage after patch
fixed

Please help and verify.

@dahlia
Copy link
Collaborator

@dahlia dahlia commented May 18, 2015

Cool. 👍

dahlia added a commit that referenced this issue May 18, 2015
Proposed fix for issue #237 - Sequence not clearing from memory
@dahlia
Copy link
Collaborator

@dahlia dahlia commented May 18, 2015

#237 fixed this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants