Streaming API

[jacobrask]

I haven't checked the code thoroughly, but since search-index uses levelup, a streaming API shouldn't be impossible?

Indexing from an fs or http stream makes a lot of sense to me.

[fergiemcdowall]

Yes- a very sensible idea!

[fergiemcdowall]

There is now a streaming replication API, see the snapshot function

[fergiemcdowall]

Thats all the streaming that will be implemented for the time being.

Indexing: Because search-index uses tf-idf to work out relevance, and because tf-idf is indexed fastest in batches, it doesnt really make sense to stream, since all of the batch must be known before processing can take place. HOWEVER- when there are other ways to define sorting, then streaming may make sense.

Searching: Search results are also returned in batches with associated metadata. If there was some use case where gigantic search results should be returned and then streamed through then it would be worth implementing this feature. There would need to be a good use case though.

At the moment REPLICATION is a great use case for streaming, and the streaming API for replication is currently in place