iterator performance

[whyrusleeping]

Trying this out, and i'm quite happy with read and write performance, but i've noticed that iterating through items in the datastore is significantly slower than what we're currently using. Is there going to be work on improving that performance?

[manishrjain]

Yeah, sorted key-value iteration is slow. We just haven't been able to get the same kind of read throughput from Go as expected. I wanted to try building an async io library in Go, but haven't had a chance. Might do it soon.

There're 2 ways to achieve performance with iteration:

• If you don't need values for all the keys, then Iterate only over keys, and retrieve values separately.
• If you don't care about sorted iteration, we could build an API which would give you keys and values in random order really fast.

[whyrusleeping]

Yeah, i don't really care about it being sorted, and i also don't care about values. If we could get a fast iteration of unordered keys, that would be really great

[manishrjain]

You can just set FetchValues option when creating iterator to false. It would iterate over keys in sorted order blazing fast.

Try it out. And if it gives you the performance you want, feel free to close this issue.

[manishrjain]

I presume this is working for you. Feel free to reopen if it doesn't.