"Too Many Files Open" Error - Files Not Being Closed?

[brc591]

I've been using the ANNOY Python API to build and save several index tables in order to load and reuse subsets of these in another script (at most about 400 in a subset at a time). The library is wonderfully fast, but I've been encountering "too many files open" errors after some time. I'm loading each index once, doing stuff with it, then unloading it before the loop continues onwards with the next index table.

Here is a partial code snippet that reproduces the error by reloading the same index table (510 x 64) over and over (run the loop multiple times if the error doesn't show up immediately):

from annoy import AnnoyIndex
index_table = AnnoyIndex( 64,metric='angular' )
for i in xrange(10000):
    index_table.load( index_table_filename )
    print i  # just seeing when it failed
    index_table.unload( index_table_filename )

After some sleuthing into the GitHub source code, I notice that unload() unmaps the file from memory, but doesn't close the file opened using load(). Would it be possible to get file closing functionality incorporated into the Python wrapper somehow please? Or am I missing something vital in using ANNOY?

Thanks for all the hard work on the API!

[erikbern]

Yes that seems correct, glancing at the code, the fd file is never closed in the unload function.

Do you want to put together a pull request for this?

[erikbern]

would be awesome if you can include that code sample as a unit test

[brc591]

I haven't used GitHub before aside from browsing through code and downloading projects, so please bear with me.

What is it that you're asking/wanting me to do?

[erikbern]

It should take me a few minutes, so I can do it.. give me 5 min

[erikbern]

something is weird with my python installation – i'll take a look later

[brc591]

Sorry for the late reply, but just wanted to say thank you very much for the fix! Just tested it out on my end and it's running without qualms here. So thank you again!