Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too many files open in world.iter_nbt (/w fix) #77

Open
Someone3x7 opened this issue Dec 15, 2014 · 2 comments
Open

Too many files open in world.iter_nbt (/w fix) #77

Someone3x7 opened this issue Dec 15, 2014 · 2 comments

Comments

@Someone3x7
Copy link

With a large world world.iter_nbt caused a too many files open exception since it never closes the region files. The simple fix is to add a few lines to close the region files in world.py at line 95 so it looks like this:

def iter_nbt(self):
    """
    Return an iterable list of all NBT. Use this function if you only
    want to loop through the chunks once, and don't need the block or data arrays.
    """
    # TODO: Implement BoundingBox
    # TODO: Implement sort order
    for region in self.iter_regions():
        for c in region.iter_chunks():
            yield c
        if hasattr(region.file,'fileobj') and region.file.fileobj: # <- added
            region.file.fileobj.close() # <- added
        region.file.close() # <- added

edit: Updated fix after encountering again in an even larger world.

@macfreek
Copy link
Collaborator

Thanks for the report and the patch, I'll have a look at it shortly. (I wanted to have a look this weekend, but didn't get around it yet. For now, let me just acknowledge your report).

Out of curiousity, how large were these worlds?

Besides your patch, it would be good to write a test case to check if all files are open. However, it seems that the resource module can only list the maximum number of open files, but not the current amount of open files.

@Someone3x7
Copy link
Author

The last few worlds had a little over 1,000,000 chunks each. Takes my old i5 couple days to parse one. I didn't want to get too deep into correcting the issue as I'm loosing my mind slowly and tend to make stupid mistakes with code these days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants