Decentralized search

[feross]

I used this slide in my JSConf Asia 2014 talk today:

It would be awesome if we could crack the decentralized search problem.

cc @missinglink

[jure]

This is relevant to my interests as well, for more info look at http://juretriglav.si/an-open-distributed-search-engine-for-science/ https://github.com/ScholarNinja/extension

The project's DHT is currently turned off, because there was a bug in Chrome at that point (memory leak + 100% CPU). I haven't looked at it in depth since, due to lack of available time, but it was promised to be resolved in September's stable Chrome.

But yeah, we should totally crack decentralized search! :) Count me in.

[iShift]

look at tribbler - torrent client with search engine

[miketamis]

look in to Cubit http://www.cs.cornell.edu/~bwong/cubit/

On Sat, Nov 22, 2014 at 5:31 AM, iShift notifications@github.com wrote:

look at tribbler - torrent client with search engine

—
Reply to this email directly or view it on GitHub
#186 (comment).

[feross]

https://en.wikipedia.org/wiki/BitTorrent#Decentralized_keyword_search

[metacoin]

Blockchain-based solution may be viable, or perhaps http://maidsafe.net/

[eklem]

We have just that idea with search-index and Norch (NOde seaRCH).
https://github.com/fergiemcdowall/search-index
https://github.com/fergiemcdowall/norch

For now, the search engine uses levelDB as an index, and that means anything that can talk to levelUP will work. https://github.com/rvagg/node-levelup Now, @fergiemcdowall is working on getting the core of the search engine, search-index, to run in the browser with browserify.

So far, there's a replication-function for when you have a server, but wants to move the index over to another server or client (browser). But we had the thought on making it run with an index from a torrent-file.

I'm not that technical, being an interaction designer, so maybe @fergiemcdowall can explain better?

[Latrasis]

There's always Namecoin and https://github.com/okTurtles/dnschain: Uses a Blockchain to set a decentralized DNS. Oh and Ethereum of course: https://github.com/ethereum/

Would you mean leveraging existing tech such as torrents+dht?

[feross]

From the p2p-hackers mailing list:

There's a really neat solution to this problem called Hyperspace Hashing which has you model your data in a set of different dimensions (i.e. a euclidian hyperspace), and locate a point within the hyperspace based on the hashes of the inputs. You then break the hyperspace down into a finite number of subspaces, and partition those among the available servers. The familiar hash ring can be seen as the one dimensional version of this concept.

It doesn't quite fit the full text search scenario you described, but it does help expressing more powerful queries than just k/v lookup:

http://www.cs.cornell.edu/people/egs/papers/hyperdex-sigcomm.pdf
http://www.slideshare.net/DECK36/c36-hyperdexmeetupdec2013v2nogfx

• Tony Arcieri

[urbien]

perhaps if we relax one requirement for a de-centralized search, like immediacy of answers, then we can solve the problem. Here is a sketch for an algorithm:

Take a look at the decentralized search idea by Bill Gross (Idealab). The essence of it is:

• index data on each machine/node, and keep index there
• broadcast a search query
• have users manually confirm the return of search results

Broadcast can be done using Ethereum's Whisper or equivalent DHT-based mechanism and can be optimized by each node publishing the topics that it allows searches for.
Manual confirmation for searches can be saved with user's permissions and can be used later without bring it to user's attention.

[niczy]

I was trying to build a chrome app to browse content with webtorrent (Each file is some content, with a collection of magnet links). So the user can navigate from one file to another.

But seems DHT is not supported in browser? which makes it not much different than traditional content serving from a server.

Any plan to support DHT in the browser?

Thanks!

[eklem]

What if Webtorrent had a built in version of Norch ? Part of it is already browserified: search-index.

[feross]

@niczy Even without DHT support in the browser, the content still gets delivered P2P which is very different than serving the content from a server.

[feross]

This is a cool topic, but out-of-scope for WebTorrent. Closing issue, but feel free to continue discussion.

[lmatteis]

Also, Tribler seems to have tackled this issue: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6038729

[lmatteis]

Another approach seems to use the blockchain to store magnet links: http://mercbay.com/#faq

[rom1504]

http://btdigg.org/about/ seems interesting on that topic, sadly it doesn't seem to be opensource

[anexit1]

This is off the wall but you could create a flowing side bar. If we can stream video we could centrally snag info. Basically a scrolling window where if someone see's something they like it will display the hash and the file name. The webtorrent will come and go depending if the browser(s) are open or not. Could certainly dump this info to a small text file and make hash's search able. Of course, that may have a impact on file security.. key-gen anyone?

[mskogly]

A local index sounds good, but letting a user choose cwhich categories to "host", i.e to limit the local index to contenttypes of interrest.

[lock]

This thread has been automatically locked because it has not had recent activity. To discuss futher, please open a new issue.