Out of memory under OS X when adding items to index

[herrbuerger]

First of all, thanks for open-sourcing this. The landscape of ANN solutions is quite sparse, so it's great to see something happening :)

I've been doing small tests under OS X with a few dozen items so far. Since the results looked very promising I decided to throw some more data at annoy.
However, during the first run, I directly ran out of memory and had to restart my notebook (16GB RAM). The test involved iterating over 500 items, each containing one hundred 128-dimensional vectors (SIFT image descriptors).

I assumed that the vectors are too large and therefore reduced the dimensions to 64 and later 32. This didn't have any effect on the memory usage though. I then decided to reduce the number of items down to 50 and then 25. This didn't make a difference as well.

Are there any known problems with annoy under OS X?
Also, can you give recommendations regarding a system configuration where annoy is known to run well? I'd be happy run this inside a VM if necessary.

Thanks,
Chris

[a1k0n]

That's very strange; could you share the code you're using to generate and to use the indexes? When does it run out of memory -- when creating the index or after saving it and using it for ANN lookups?

How much memory does it use in your small test? It shouldn't be very much if it's only 500 items, even if they're 128 dimensional items.

Annoy definitely works under Linux (Debian or Ubuntu is what we use) and I thought it worked fine on OS X as well.

[herrbuerger]

While stripping down the code to get a minimal working example, I found the actual cause of the problem.

I was generating quite long integers that I have used as identifiers (see https://gist.github.com/herrbuerger/5f22ee159d0df800c844). Once I had changed them back to smaller numbers, everything went smoothly.

Here is a minimal working script to reproduce the issue: https://gist.github.com/herrbuerger/e4452d55ffb03a53d423

[herrbuerger]

I have found another issue regarding the identifiers, not sure this is related. If not, a separate issue would probably make sense:

https://gist.github.com/herrbuerger/6cbeada92901a282c957

If you check the output of the script, you will see that annoy shows identifiers which have never been used. Took me a while to catch this :)

[erikbern]

We should probably couple Annoy with a hashtable so that you can put in arbitrary stuff. But yeah, until then, you will see those issues :(

[erikbern]

I'm not sure if there is a fix to this – what do you suggest?

[herrbuerger]

A hashtable would be great. This would definitely solve my problems.

Still, I have a working example with the integers now as well and the results look pretty good. However, I will have to create a lookup table, which will probably be a challenge itself depending on the number of items.

[erikbern]

There's a lot of file-based fast hashtables, including Spotify's own: https://github.com/spotify/sparkey – other ones I'm aware of are bsddb and Tokyo Cabinet.

[erikbern]

Closing this issue since it's not a "bug" per se – maybe the documentation should be more clear