Image.save() does not close file pointer

[rustyis]

What did you do?

I'm modifying a large number of single-page tiffs as NumPy arrays and saving each to file individually via Image.fromarray. For example:

for src_path in img_list:
    np_array = cv2.imread(src_path, 0)
    with Image.fromarray(np_array) as img:
        img = img.convert('1')
        img.save(dst_path, format='tiff')

What did you expect to happen?

I expected the full list of images to be read into arrays in turn by OpenCV & NumPy and saved as TIFFs by Pillow.

What actually happened?

After exactly 1020 successful iterations, I get the following error:

Converting 1021 of 5173: bort_005.png
Traceback (most recent call last):
  File "C:\TestEnv\Pillowtest\test.py", line 94, in <module>
    main(sys.argv[1:])
  File "C:\TestEnv\Pillowtest\test.py", line 39, in main
    img.save(dst_path, format='tiff')
  File "C:\Python34\lib\site-packages\PIL\Image.py", line 1893, in save
    save_handler(self, fp, filename)
  File "C:\Python34\lib\site-packages\PIL\TiffImagePlugin.py", line 1430, in _save
    _fp = os.dup(fp.fileno())
OSError: [Errno 24] Too many open files

Press enter to continue.

What versions of Pillow and Python are you using?

Python v3.4.4 (x64) on Windows 7
Pillow 4.2.1

What have you tried?

I have tried enclosing img.save() in a context manager, as well as explicitly closing the file pointer (if that's even possible or relevant here?), e.g.

with open(dst_path, 'wb') as image_file:
    img.save(image_file, format='tiff')
    img.close()
    image_file.close()

I have also seen and reviewed the following issues, and have not been able to successfully implement their workarounds. This case seems to be different due to the lack of calling either Image.open() or Image.load().

• #477: 'ResourceWarning: unclosed file (for Python 3)
• #835: ResourceWarning: unclosed file when calling save...
• #1144: Image.load() does not close file pointer
• #1237: 'Too many open files' error

[rustyis]

Skipping the CV2/NumPy process entirely and using PIL alone, like this:

with open(src_path, 'rb') as image_file:
    with Image.open(image_file) as img:
        img = img.convert('1')
        img.save(dst_path, format='tiff')

Produces the same error, but with a different traceback:

Converting 1021 of 3274: bobathy_010.png
Traceback (most recent call last):
  File "C:\TestEnv\Pillowtest\test.py", line 94, in <module>
    main(sys.argv[1:])
  File "C:\TestEnv\Pillowtest\test.py", line 39, in main
    img.save(dst_path, format='tiff')
  File "C:\Python34\lib\site-packages\PIL\Image.py", line 1890, in save
    fp = builtins.open(filename, "w+b")
OSError: [Errno 24] Too many open files: 'C:\\TestEnv\\Pillowtest\\_output\\bobathy_010.tif'

Press enter to continue.

[wiredfool]

The tracebacks aren't going to be much use, since they are triggering on the last attempt to open a file, not which files remain unclosed.

Can you get a list of open files from the OS somewhere?

I suspect that the trouble is still in the open side.

Can you try this:

with open(src_path, 'rb') as image_file:
    with Image.open(image_file) as img:
        img = img.convert('1')

And on one of the images, check if img.readonly and img.map are set?

[rustyis]

I don't fully know what I'm doing, but using Process Explorer I can't find any indication that the process has more than one file handle open to an image at any given time. In fact, it explicitly shows each file handle exiting in turn (turning red in the handles frame) before a new one is opened.

And on one of the images, check if img.readonly and img.map are set?

The value of img.readonly changes from 1 to 0 after img = img.convert('1'), while img.map throws an AttributeError both before and after.

[wiredfool]

So does it run to completion without the save? Or does it eventually die in opening?

[rustyis]

My god, I'm sorry, this isn't a Pillow issue at all. I thought I had a sterile test environment, but I was still wrapping this whole process in a context manager to suppress "deep" stdout and stderr content.

The instance created by this context manager was redirecting stdout and stderr to null files and then failing to properly clean up after itself.

Thank you for your time and help.