Use LMDB for storage

[erikbern]

@LongbinChen pointed out that LMDB would be a good way to support a series of things.

http://symas.com/mdb/microbench/

If we swap out the storage to rely in LMDB we would get a lot of features for free

• Ability to use any key, not just integers
• Out-of-core data (dataset doesn't fit in RAM)
• Decouple index structure from nodes (could potentially set _K independently and optimize for it).
• Incremental add_item (and deletes) would be much easier to implement

I'm not sure how tricky this would be, but probably not more than 5-10h. Mentioning it in case someone wants to give it a shot :)

[LongbinChen]

best way to celebrate the labor day is to work. I will work on this for this weekend and hopefully come out something.

[erikbern]

Nice! Let me know if you need any help!

On Sunday, September 6, 2015, Longbin Chen notifications@github.com wrote:

best way to celebrate the labor day is to work. I will work on this for
this weekend and hopefully come out something.

—
Reply to this email directly or view it on GitHub
#96 (comment).

[erikbern]

Did you end up doing anything with this? I've been thinking more and more about this and might spend some time on it

[LongbinChen]

I added something that are not working yet. I can make a pull request and
so you can see what I have done.

and feel free to take from there.

On Wed, Sep 9, 2015 at 12:38 PM, Erik Bernhardsson <notifications@github.com

wrote:

Did you end up doing anything with this? I've been thinking more and more
about this and might spend some time on it

—
Reply to this email directly or view it on GitHub
#96 (comment).

[LongbinChen]

pull request made.

On Wed, Sep 9, 2015 at 1:41 PM, Lbchen Chen lbchen@gmail.com wrote:

I added something that are not working yet. I can make a pull request and
so you can see what I have done.

and feel free to take from there.

On Wed, Sep 9, 2015 at 12:38 PM, Erik Bernhardsson <
notifications@github.com> wrote:

Did you end up doing anything with this? I've been thinking more and more
about this and might spend some time on it

—
Reply to this email directly or view it on GitHub
#96 (comment).

[LongbinChen]

do you still have the access to spotify/annoy?

On Wed, Sep 9, 2015 at 2:06 PM, Lbchen Chen lbchen@gmail.com wrote:

pull request made.

On Wed, Sep 9, 2015 at 1:41 PM, Lbchen Chen lbchen@gmail.com wrote:

I added something that are not working yet. I can make a pull request and
so you can see what I have done.

and feel free to take from there.

On Wed, Sep 9, 2015 at 12:38 PM, Erik Bernhardsson <
notifications@github.com> wrote:

Did you end up doing anything with this? I've been thinking more and
more about this and might spend some time on it

—
Reply to this email directly or view it on GitHub
#96 (comment).

[erikbern]

Yeah I do. I think with LMDB it might be worth considering moving this somewhere else though – it's a big non-compatible change anyway

[LongbinChen]

and houzz would be happy to host the repo if you like.

On Wed, Sep 9, 2015 at 5:19 PM, Erik Bernhardsson notifications@github.com
wrote:

Yeah I do. I think with LMDB it might be worth considering moving this
somewhere else though – it's a big non-compatible change anyway

—
Reply to this email directly or view it on GitHub
#96 (comment).

[rikenmehta03]

@LongbinChen Are above mentioned support implemented or someone working on that..?

[erikbern]

not afaik