Memory leak moving from 9.0.0 to 9.1.0

[replabrobin]

I maintain reportlab most likely badly. A user reported a memory leak problem running a large chart generation job with reportlab.
I have dealt with many leaks over the years mostly from the C extensions in our software. I was suprised to find that the leak occurs only when I update Pillow from pillow 9.0.0 to 9.1.0; PIL/Pillow has always had a very stable API so our dependence is now Pillow>=9.0.0. A reportlab contributor said that Pillow<9.0.0 had known security issues.

I observed the problem on windows amd64 and linux x86_64. Only the pillow package was changed. reportlab uses Pillow to convert bitmaps to standard formats. In this case we are just saving png.

I'm not sure I can replicate the error in a fully open source manner, but would certainly like to try and find the source of the problem. I will try to create a simpler test example.

Is anything known about memory leaks in pillow; what tools should I be trying to use? In linux htop provides updated memory usage as the script runs on windows the process inspector shows a steep graph for memory usage.

[wiredfool]

The gold standard for memory leaks is valgrind, but that generally requires a reproducible test case so that it can be run with instrumentation. Do you have an idea of the pillow operations that are being called, and approximately how many times?

[replabrobin]

I think we are just saving the image with reportlab's renderPM.py; the internal graphics state is just an RGB buffer drawn to by bits of libart_lgpl.

The code looks like

	def toPIL(self):
		im = _getImage().new('RGB', size=(self._gs.width, self._gs.height))
		(getattr(im,'frombytes',None) or getattr(im,'fromstring'))(self._gs.pixBuf)
		return im

	def saveToFile(self,fn,fmt=None):
		im = self.toPIL()
.........
		im.save(fn,fmt,**configPIL)

I will have to check what the example stuff is actually doing to set up the configPIL keyword arguments and if the filename is a simple string. The test script is writing out 1134 images. I'll do some simpler testing tomorrow; after a dentist visit :(

[replabrobin]

I'm almost sure this is not a leak issue in Pillow 9.1.0; by experiment I found that inserting a gc.collect() in the generation loop every 10th chart prevents the huge memory usage which is seen without it. On the other hand it does seem as though pillow 9.1.0 is using more memory somewhere and that creates the garbage pileup which creates the problem. I do notice that pillow 9.1.0 is much faster so this is perhaps caused my some memory / time tradeoff.

pillow 9.1.0
last chart 30 reached

real	0m7.580s
user	0m7.346s
sys	0m0.226s

pillow 9.0.0
last chart 30 reached

real	0m13.893s
user	0m13.664s
sys	0m0.201s

[radarhere]

I wonder if this is the same problem as #6448. Would you be able to test #6456 and see if it fixes your situation?

[replabrobin]

I can certainly just apply that patch and test with my one file example. Give me an hour or so.

[replabrobin]

Thanks radarhere; I can confirm that the patch from #6456 does remove the large growth in memory I saw with my test example. It also seems to keep the speedup I saw between 9.0.0 & 9.1.0; hopefully this will end up in a release.

[radarhere]

The PR has been merged now, so it will be part of Pillow 9.3.0, due out on October 15