How to get the data?

[ciro-maciel]

@tristanls liked your project, congratulations!

I'm looking at your documentation and trying to understand the architecture. I'm not finding anything on how to get the data.

How to get the data?

Thank you

Ciro

[tristanls]

Hey Ciro,

I never got around to implementing consolidating the data intervals nor reading from the written data intervals. The data is stored in a LevelDB-backed storage, that's what alldata-storage-leveldb uses. In particular, here is where storage is created - https://github.com/tristanls/alldata-storage-leveldb/blob/master/index.js#L344 and a write to it https://github.com/tristanls/alldata-storage-leveldb/blob/master/index.js#L361. Reading the data would involve opening up the underlying store using levelup.open. And then reading accordingly, probably via a levelup.createReadStream.

In case you're curious, there is a similar system that has been implemented end-to-end that I saw a presentation on. It has similar design elements as alldata, and is called OK Log. Check it out: OK Log video, repo link here.

I hope this helps.

Cheers,

Tristan

Edit: Added reference to levelup.createReadStream to demo streaming all the data from an interval, once opened.

[tristanls]

For a visual explanation of the architecture alldata implements, you can see a slide show explaining high-level concepts, starting here.

Edit: Updated link (accidentally linked to start of presentation instead of specific section).

[ciro-maciel]

Hello @tristanls,

I'm going to study the material you've sent and I'll return.

Thanks for the answer!

Ciro

[ciro-maciel]

Hello @tristanls,

Thank you for your point of view and the ample and detailed material, fantastic!

I believe you have understood about allData working, its documentation is great.

I have some questions / ideas, what would be your opinion about it:

• for a search on the data, do you think it's the best way to do a direct search on the local LevelDB instance?
• regarding fulltext searches, create the indexes in these locals instances and perform the search?

Ciro

[tristanls]

Hello,

I think if alldata were fully implemented using the ideas described in the slides I linked before, then there might end up no room for the indices on the machines that store the data. I would expect the machines to eventually become full and migrate themselves into read-only cluster. Although, this only implies that persistent storage is "full", memory might be available and if your indices fit into memory then that might work.

Another consideration is that the data in alldata is stored in-order, so sequential access via something like levelup.createReadStream would probably be most "performant" (for some arbitrary definition of "performant"). Doing random access/searches against in-order data might be less "performant". Then again, if you created indexes in memory, it might work just fine.

Cheers,

Tristan

[tristanls]

Here's some info more reliable than my opinion about performance of leveldb :)
https://github.com/google/leveldb#performance

[ciro-maciel]

Understood @tristanls,

The numbers on the levelDB performande are interesting!

Your presentation is important, very instructive.

I found this mechanism (DHTs - bucket storage) very interesting, once I reserve more time I will start to study these structures more.

Thank you.

Ciro

[tristanls]

Regarding bucket storage, here's an implementation that's used in a bunch of production DHTs: k-bucket. As of v3.0.3, it is optimized to use less heap, but if you look at the code prior to v3.0.3, it is a very literal implementation of the presentation slides.

[tristanls]

By the way,

I'm happy to hear you find these helpful, thanks for letting me know.

Cheers,

Tristan