ImageFolder dataset: "too many open files" error

[shaibagon]

Hi,
I am using ImageFolder dataset to train on imagenet.
I repeatedly get "too many open files" OSError after training for several hours.
I suspect the issue comes from [here][1]:

def pil_loader(path):
    # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
    with open(path, 'rb') as f:
        img = Image.open(f)
        return img.convert('RGB')  # <--

The return is inside with ... clause, thus (I suspect) the file handle f is not closed properly when the function returns.
Shouldn't the function be

def pil_loader(path):
    # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
    with open(path, 'rb') as f:
        img = Image.open(f)
    return img.convert('RGB')  # <--  not indented

Thanks,
-Shai
[1]: https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py#L156

cc @pmeier

[vfdev-5]

@shaibagon could you tested this modification on imagenet again and see if problem disappears ?

[shaibagon]

@vfdev-5 I'm afraid this is not so easy to do the way pycharm is installed on my machine. I'll try and report back. (this is why I opened an "issue" rather than a PR...)

[vfdev-5]

@shaibagon could you provide a traceback please if you still have it ?

[shaibagon]

@vfdev-5 I am sorry, but because of the remote environment I am working at it is very difficult to provide additional information. I am also suspecting I have not located to problem correctly.

Bottom line:

running for long time (day+) using ImageFolder data results with OSError "too many open files".

I will do my best to shed more light on this issue. meanwhile, I don't think there's much to do.

Other people who stumble upon this error - please try and provide more information.

[vfdev-5]

@shaibagon I tested the following and no error found:

from PIL import Image

def pil_loader(path):
    # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
    with open(path, 'rb') as f:
        img = Image.open(f)
        return img.convert('RGB')

import tempfile
import os

a = None

with tempfile.TemporaryDirectory() as tmp:    
    print("Temp dir: ", tmp)
    for i in range(10000000):
        if i % 50000 == 0:
            print("{}".format(i), end=" ")
        img = Image.new('RGB', size=(10, 10))
        p = os.path.join(tmp, "test_{}.png".format(i))
        img.save(p)
        a = pil_loader(p)

[shaibagon]

@vfdev-5 okay. that means I failed to spot the origin of this annoying OSError. However, it still occurs occasionally. Is it possible to leave this issue open hoping other user will be able to help reproduce it?

[vfdev-5]

@shaibagon could you post some of your code that produces the problem ?

Is it possible to leave this issue open hoping other user will be able to help reproduce it?
Sure, that it can be kept open for a while until mantainers decide something on it.

Actually, there is another option to not to use PIL to open images - replace default loader, see here
You can try other libraries like opencv or skimage just to open the image and then convert it to PIL (this could be a bit slower...) :

import cv2
from PIL import Image

def image_loader(path):
    img = cv2.imread(path)
    assert img is not None
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    return Image.fromarray(img)    

or even test it directly without with open()... as PIL issue 835 seems to be resolved:

from PIL import Image

def read_image(filepath):
    return Image.open(filepath)

[shaibagon]

Is this related?

[fmassa]

@shaibagon I don't think it's related.
What's happening is that for some reason PIL is not properly closing the files.
I'm not 100% sure of what the fix is, because there has already been 2 PRs trying to fix related behavior, see #357 and #176

I'll have a closer look at this.

[temerick]

Has there been any progress on this? I'm noticing this same problem on the current version (0.3.0).

[pmeier]

@shaibagon @temerick Is this still relevant?

[HJWoods]

This is still an issue, as far as I can see the solution suggested by the OP should fix this. This is a pretty big problem for any projects that train for a long time or use multiple training/test runs, it is pretty much guaranteed that you will hit the limit at some point