Make Frostwire more IPFS-friendly by better handling IPFS URLs used as webseeds

[hollownights]

Right now, when Frostwire is presented with a public IPFS gateway URL used as a webseed, it, as expected, tries to connect to it, but downloading a file hosted on IPFS through a public gateway is less than ideal, specially for large files. With that said, I propose that Frostwire should better handle public IPFS gateway URLs by doing the following: when Frostwire detects a public IPFS gateway URL used as webseed, it should automatically rewrite it to a path-like URL served by the localhost or, if it detects a native IPFS URL using the “ipfs:” protocol, rewrite it as a subdomain gateway URL. After rewriting it and making the request, wait for a 206 (maybe 200 also?) HTTP response. If it comes, just let the connection proceeds; if it doesn’t, then ditch the localhost URL and go back to the public gateway URL or, if it was a native URL, just discard it. The URLs would look like this:

https://gateway-host.tld/ipfs/{cid} → http://localhost:8080/ipfs/{cid}

ipfs://{cidv1} → http://{cidv1}.ipfs.localhost:8080

This behavior could be further improved (specially for native URLs) by actually talking with the IPFS software (through command line or API), but I’m trying to keep it simple for now. Such a change, combined with the option to automatically import a downloaded file/folder to a local IPFS node (#997), would make for better interoperability between a decentralized web protocol (IPFS) and a decentralized but “files-only protocol” like BitTorrent.

P.S.: I am not a developer, so I’m here as a user to propose such a feature, discuss it and test if it gets implemented.

[gubatron]

I think the huge issue with this is the port number.

Say we had a light webserver on that port, something like 8080 is very popular, there's a chance it's taken and the user needs to configure another port in the settings.

In the case the port is not the standard 8080, how does another peer know on which port to point the URL to?

I think it's better to leverage a local IPFS instance. I'm looking into just embedding our own IPFS node, there's a light rust implementation we could use to create an executable FrostWire can manage via http commands, the same way we manage our "telluride" video downloader which runs as a standalone http server.

[hollownights]

I'll paste here what I said on #997:

I think you got so excited with such a feature that you are taking it a little too far already haha. No, it isn't a webserver and it really is as "dumb" as it reads: did Frostwire find a IPFS-like URL being used as webseed? If yes, rewrite the URL to try and get the file through IPFS.

If, lets say, right now you install IPFS desktop and get these torrent files, that have IPFS public gateway URLs as webseeds, and then manually changed those URLs so they look like these:

https://gateway-host.tld/ipfs/{cid} → http://localhost:8080/ipfs/{cid}

ipfs://{cidv1} → http://{cidv1}.ipfs.localhost:8080

... then Frostwire would already be able to get the files from IPFS. Just like that.

[gubatron]

so I see the torrent file has webseeds on cloudflare-ipfs.com/'s gateway

If I understand you, when FrostWire downloads and seeds a file that happens to have IPFS seeds, it should serve these files via http/https at localhost:8080/ipfs/{cid}

This will not always work because there's a ton of other apps and localhost servers that go to 8080, so let's say frostwire tries to start its HTTP server and bind it to port 8080 and its taken, the logical thing would be to error and tell the user:

"Hey, you already are using port 8080, if you want me to serve the file as an IPFS gateway let me know of another port"

so the user goes and sets the port and it's all good.

How will other IPFS clients or Web clients, or even other FrostWire clients try to fetch from us if we haven't specified anywhere that our IPFS/HTTP port for such files is on port (not 8080) ?

I think it's just better to tell a local IPFS daemon, hey announce this file however you can announce it, via your internal DHT, so if that some gateway is looking for it, we'll be able to respond to it so that other gateway can relay it for us, given we're able to break past all the firewall issues and what not.

[hollownights]

This issue is really just about getting the files, not serving then on a server/IPFS. Just rewrite the URLs to get the files and everything else is business as usual, no server created by Frostwire.

[gubatron]

I see, what you want is to check for that file first on a local IPFS instance instead of burdening the remote ones.
Easy if we have that "Settings > IPFS" dialog letting frostwire know before hand where to look for a local (or remote) gateway.

[gubatron]

Dev Note: webseed handling is done by the libtorrent library, would probably have to modify the torrent handler to just add more webseeds before letting libtorrent start the transfer.

[hollownights]

I see, what you want is to check for that file first on a local IPFS instance instead of burdening the remote ones.

Exactly!

Easy if we have that "Settings > IPFS" dialog letting frostwire know before hand where to look for a local (or remote) gateway.

Yeah, it could be like that, but considering the fact that there is no real interaction with any IPFS software, I don't think this feature needs to wait for any integration with IPFS.

would probably have to modify the torrent handler to just add more webseeds

Well, if you just add more webseeds (the ones for the localhost), then you would be burdening the public gateways anyway...

[gubatron]

ok, it could be like a "best effort" try,

"Check if there's a local IPFS HTTP server on 8080... if so, add it as the top priority webseed"

This should be a quick hack.

[gubatron]

Well, if you just add more webseeds (the ones for the localhost), then you would be burdening the public gateways anyway...

Not necessarilly, only if the underlying torrent engine needs to get pieces from more than one of the seeds. It may or it may not hit all the webseeds.

[gubatron]

...or we can make a setting for policies on how to handle IPFS webseeds:
(0) Prioritize local IPFS seeds
(X) Try ONLY local IPFS webseeds

[hollownights]

Not necessarilly, only if the underlying torrent engine needs to get pieces from more than one of the seeds. It may or it may not hit all the webseeds.

Yeah, I guess it would work, specially because public IPFS gateways usually take a long time to answer.

...or we can make a setting for policies on how to handle IPFS webseeds:

Yes, even without any integration with IPFS, a "Settings > IPFS" menu could be used to let the user say "I have a local IPFS node, so just ditch any public IPFS gateway URL used as webseed and only try to connect with the localhost".