Result / index sharing

[asciimoo]

It would be good to find a way to allow Hister users to share parts of their indexes with others. This way, users can (automatically?) contribute the pages they index - building a shared search engine over time. Also, it could be used to share datasets with others.

There are lots of questions about the implementation:

• What would be a user friendly, yet fully secure and privacy respecting way to submit data?
• Where should we handle the collected data? (Centralized or P2P or let users share on their own platforms?)
• How could we verify that the submitted data is valid and it does not contain anything malicious?

Probably there are more questions around this feature. Please let me know if you know other issues or if you have answers/suggestions.

[thiswillbeyourgithub]

Hi,

I got interested in the mwmbl search engine some time ago and even contributed somewhat. It's "minimalistic" approach made it appear to me as a good candidate for something that could simply be a docker container that would both expose a search page, crawl the web, and contribute it to some DHT-like index.

I'm bringing this up because I raised a few questions on actual technical problems to solve to take this on and think it would be relevant here:

• Make Mwmbl fully decentralized? mwmbl/mwmbl#259
• Thoughts on the crawler reputation problem mwmbl/mwmbl#260

I would still love to see this kind of decentralized-selfhosted-p2p search engine exist but mwmbl's dev ended up not responsive enough to not make me lose interest somewhat (no judgement here).

Very curious on your thoughts about all this.

PS: also, can't resist this unrelated but potentially useful read: Building a web search engine from scratch in two months with 3 billion neural embeddings

[asciimoo]

Great suggestions and valid concerns about trust/curation in both issues you referenced.

I don't have a strong opinion about what would be the best direction for Hister yet, but no matter what solutions we choose, I'd like to keep it fully optional and as conservative as possible - providing the users/admins full control over who they want to share/receive results with/from.

PS: also, can't resist this unrelated but potentially useful read: Building a web search engine from scratch in two months with 3 billion neural embeddings

Yes, this is a nice blog post. I even included it in the demo Hister instance: https://demo.hister.org/