Use original gateway URL for content routing (async fetch of blocks and cars)

[askender]

tl;dr: #1042 (comment)

---

Describe the bug and To Reproduce
If we are using ipfs-companion, some gateway url cannot be openned. It only try to find the content at local ipfs node, never retry the gateway url itself.

To Reproduce
Steps to reproduce the behavior:

1. set ipfs-companion well for Chrome, and running a local ipfs daemon.
2. open https://bafybeiae5c2ip2xxd5vvbnteu2sxntnmvgqqyt3we2s3hzbbcfifdsz3ya.ipfs.infura-ipfs.io/ using Chrome, the url cannot be opened. (It can be openned because someone pinned it now.)
3. Close the ipfs-companion, then retry to open https://bafybeiae5c2ip2xxd5vvbnteu2sxntnmvgqqyt3we2s3hzbbcfifdsz3ya.ipfs.infura-ipfs.io/ again, it can be opened now.
4. It seems the CID is not added to DHT, a ipfs gateway url with cid cannot opened. But the url is ok without ipfs-companion
5. any options of ipfs-companion cannot fix this problem.

Expected behavior
A url like https://bafybeiae5c2ip2xxd5vvbnteu2sxntnmvgqqyt3we2s3hzbbcfifdsz3ya.ipfs.infura-ipfs.io/ can be opened even if it is not in local ipfs node.

Screenshots
No.

Desktop (please complete the following information):

• OS: Mac
• Browser: chrome
• Version: 96.0.4664.110 (arm64)

[askender]

Detail:
https://community.infura.io/t/do-you-need-an-infura-account-to-use-infura-ipfs/3422/2 I didn't use the account yet. As doc said https://infura.io/docs/ipfs#section/Authentication/Using-Javascript with a new Infura Authorization header. Everything else stays the same, as the Infura API is fully compatible with IPFS.
So,
Option1: without auth, Infura don't pin data in DHT, and even we use their http-gateway, if people use ipfs companion, that gateway url never opened, if people don't use ipfs companion, that gateway-url can be opened, everything is fine. So embarrassing. (But it seems it can be solved in ipfs-companion side. A timeout and retry the gateway url itself. )
Option2, we pay for Infura ipfs service, but because we running in browser and not web2 backend, it seems other people can use our account.
Option3, we should find another free gateway which will auto-pin, or we host a ipfs gateway service ourself. As we are in China, host ipfs gateway maybe dangerous for ourselves.

So we still think ipfs-companion can solve the problem by retry the gateway url itself after fail to found in local ipfs node. (use gateway whitelist set by user, not only one default gateway)

Thanks!

[SgtPooki]

@lidel to respond on here with further information

[lidel]

@askender I think you should contact Infura and explain your needs to them:

• if they don't announce CIDs on DHT then it is a bug on their end
• if you need to use their gateway with some auth scheme - this is a feature request for them (e.g. other services are experimenting with UCAN, which removes the need for sharing API keys)

---

As for what we can do in IPFS Companion – we won't add any special handling per gateway. Gateways are generic interface that aims to remove vendor lock-in. We also can't silently decrease integrity guarantees: users choose to use local IPFS gateway to ensure hash of every block and CID is verified.

Potential feature request: async fetch of block and CAR from the original public gateway

My understanding is that a realistic feature request here is to leverage a list of public gateways for content routing when loading data via local gateway takes a long time (more than n seconds), and fetch them as raw Block or CAR (in a way that still verifies every block).

This is a sensible feature request, but is not possible to do without Verifiable HTTP Gateway Responses (ipfs/in-web-browsers#128).

• go-ipfs 0.13 will support raw block and CAR stream responses (feat(gateway): Block and CAR response formats kubo#8758), which will allow ipfs-companion to detect slow load, and fetch blocks from any gateway in a trustless fashion, allowing local gateway to complete request.
• ideally ipfs-companion would have a list of public gateways, and use them as a fallback. to make that efficient we would need Gateway: fast check if CID is in local datastore cache (only-if-cached) kubo#8783 to prioritize gateways which have the data in their local cache.

tl;dr blocked until

• go-ipfs 0.13
• MV2: refactor: necromancy to restore MV2 build #1054
• MV3: 2023

[lidel]

Verifiable gateway responses are now possible: https://docs.ipfs.tech/reference/http/gateway/#trustless-verifiable-retrieval

Companion should be able to monitor URLs and log when a public gateway URL is requested, then detect when top-level document load takes more than a few seconds, and accelerate it by requesting blocks over HTTP from the original gateway.

This should be enabled by default, with opt-out toggle on Settings page.