Add support for P2P protocols (dat, IPFS,...)

[brechtcs]

Over at the Beaker project there's some talk about creating a search engine for dat sites. I think instead of creating an entirely new project for dat, it might be preferable to try to support it in YaCy.

The tricky thing would be to integrate it in YaCy's Java codebase, as dat currently is only implemented in nodejs. But I think it could work by checking if the dat CLI is available in PATH, and if yes calling that from the Java code.

A similar argument could be made for IPS sites I assume, but I'm not familiar enough with that ecosystem to consider implementation details.

[arni077]

Ipfs has already search engine. Ipfs-search.com
Yacy is very inefficient search engine.

[raspher]

Support for IPFS shouldn't be an issue for now. No popular browser it's supporting it. IPFS is still in beta anyway, as the filecoin...

What's the sense of adding feature, crawl ipfs links when their hashing method can change? All content would be unusable and peers will have junk, not usefull data...

[Clay-Ferguson]

@raspher IPFS hashes are not going to change. IPFS has already become the world-standard for P2P data sharing, and it's more likely that the sun is going to supernova than that IPFS hashing algorithm will "change". That would render all of IPFS data inaccessible and the risk of that is now zero, although back in 2018 when you posted that things weren't as final.

Bottom line: IPFS is ready. The world has accepted it. Getting it into Yacy should be a top priority imo.

[swedneck]

@Clay-Ferguson Uh, anyone can easily change the hashing algorithm used when adding data to IPFS, as well as a host of other parameters. It's also far from unlikely that they'll change the default hashing algorithm for go-ipfs in a future release.

However, what i think you're getting at is that an existing IPFS address can't suddenly be rendered unusable, so long as someone is providing the data for it.

[Clay-Ferguson]

IPFS was made to support different hashes only to ensure it's capable of handling possible futures where perhaps even quantum computing might render SHA-256 obsolete. Saying that's a reason not to trust the hashes is misleading at best. Like 99% of the world's strong hashes are currently on SHA-256, including Bitcoin. If SHA-256 becomes obsolete all Bitcoin and most blockchains would become insecure, for example.

Also, more importantly, the main reason IPFS was invented was to solve this issue. That is, once a piece of content is identified by a hash you can trust that hash to identify the content forever.

Now, as for the storage of the actual content, search engines (like Lucene, or even Google) generally do not store the actual data but links to the data plus index data (which is not the content). So IPFS solves the risk of those links ever "going bad", in as good a way as mankind possibly could. There's no way humans can guarantee someone is always storing a copy of file X, but what IPFS does (like BitTorrent) is it means as long as just ONE peer is still storing it, it's accessible to the entire world.

[swedneck]

I never said you shouldn't trust hashes, just that it's absolutely false that the default hashing algorithm will never change.

[Clay-Ferguson]

My reason for replying to you @swedneck was mainly to be sure the YaCy community at large was aware of the information I posted, since what I was replying to was leaning towards not trusting IPFS, and for reasons that were either wrong, misleading, or both.