Native IPFS support in Brave #10220
Description
Overview
Before the work described in this issue, Brave supports IPFS through the installation of the IPFS Companion extension. A user can install this extension from the Chrome Web Store, or by toggling a switch in chrome://settings in the Extensions section.
Once IPFS Companion is installed, the user can go to IPFS Companion’s settings and make it use “External + chrome.sockets (experimental)”. This avoids the user needing to manually install a local daemon, and uses js-ipfs instead; however, js-ipfs is behind go-ipfs on several fronts. It would be ideal to use go-ipfs instead of js-ipfs.
Our objective is to seamlessly connect Brave users to the IPFS network, including allowing Brave users to easily use dApps that run on IPFS. This spec describes the handling and installation of the go-ipfs daemon via use of the Brave component update server. The functionality will feel built-in and not require the user to separately install a daemon for full IPFS support out of the box.
Targeted Platforms
Desktop (Windows, macOS, Linux)
Description
The work can be broken down into the following sections:
- Native protocol support
- IPFS onboarding experience
- Installing and updating go-ipfs daemon component
- IPFS keyed service and proxy configuration
- Processes
- Port numbers
- Non top level navigations
- Settings
- Diagnostic page - brave://ipfs
- HTTP gateway fallback
- Interstitial page when there are no peers
- Admin policy
- The role of IPFS Companion and new chrome.ipfs API
- Landing and shipping code behind a flag
- Privacy-Preserving Product Analytics (P3A) (for those that have P3A on)
- Guest windows, Tor windows, and incognito windows
- Cache clearing (1.20.x and beyond)
Native protocol support
The following 2 protocols will be handled by Brave automatically:
ipfs:// and ipns://
When the user navigates to a URL such as:
ipfs://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq/wiki/Vincent_van_Gogh.html
If IPFS is not yet configured, the user will have the IPFS page loaded through the gateway https://dweb.link. The IPFS onboarding experience mentioned below will ask the user if they’d like to install the full node and serve it through a local gateway instead. The ipfs:// and ipns:// URLs in the URL bar will not be preserved unless a local node is used.
If the IPFS full node is installed, the content will be loaded through that node.
Other URL types which would activate this same UI are described here:
https://github.com/ipfs/in-web-browsers/blob/master/ADDRESSING.md
When a user requests via a base58 encoding:
ipfs://QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR
It gets redirected to gateway path:
https://dweb.link/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR
And the gateway does the DNS-safe conversion and redirects to base32:
https://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi.ipfs.dweb.link/
The base32 conversion can be done directly as an optimization.
IPFS onboarding experience
An infobar will be displayed when the user navigates to a gateway URL with URL pattern */ipfs/* or */ipns/*:
 |
If the user selects to “Enable IPFS” the go-ipfs component will be installed and the daemon will be used for all future requests. After installation, the infobar goes away and future URL loads will load through the local node.
If the user selects “Settings”, they will be redirected to chrome://settings and they will see the advanced settings available such as using a gateway.
Installing and updating go-ipfs daemon component
The go-ipfs daemon binary will be packaged into a crx file and installable via the component updater. There will be a different package per platform (Windows, macOS, and Linux x64) each with a different component extension ID.
Updates will occur as part of the normal component update flow in Brave described here: https://github.com/brave/brave-browser/wiki/Component-Extensions
If it is installed, you can see it installed here: brave://components/ Use find in page for IPFS.
When a new go-ipfs version is released, it doesn’t need to always be updated in Brave. We’ll start with having a process of Protocol Labs asking Brave to update. When Brave receives a request to update, we’ll run a Jenkins job which packages, signs, and publishes a new component. The binary will be pulled from Protocol Labs at https://dist.ipfs.io/go-ipfs.
ipfs.bravesoftware.com should be used so that if an antivirus wants to block IPFS, then it can do so without blocking Brave completely.
Code for brave-core-crx-packager can be found here: https://github.com/brave/brave-core-crx-packager/search?q=ipfs
IPFS keyed service and proxy configuration
A new Chromium keyed service and related code will be added inside of brave/components/ipfs/ to manage the go-ipfs daemon.
A SOCKS5 proxy is not needed, we instead map URL traffic to http://{cid}.ipfs.localhost:<port>
Default ports are changed as to not conflict with a previously installed go-ipfs node on a user's machine.
Processes
There are 3 processes that come into play:
- Browser process
- Utility: IPFS Service process (visible from Chromium task manager)
- go-ipfs process (visible outside of Chromium)
All of the Brave code that is added is browser process level code.
Brave will also start the utility process which is sandboxed and who's sole purpose is to manage the go-ipfs process.
Both the Utility process and go-ipfs process will only be started when IPFS is being used. It will be lazily started.
Similar to how Tor is handled, there is handling for if go-ipfs crashes.
Port numbers
go-ipfs starts 3 different TCP listening sockets:
- API port
- Gateway port
- Swarm port
Brave Nightly, Dev, Beta, and Release each use different port numbers for each of those 3 sockets. So they should all be able to be used at the same time. The ports used are also not default, so they shouldn't conflict with a user that already has go-ipfs installed manually.
Non top level navigations
Subresources are allowed to use the ipfs:// and ipns:// schemes, and the handling can be found here.
Subresources with the ipfs and ipns scheme can only be loaded from within a top level scheme that is also ipfs and ipns. So for example, an http page cannot load an ipfs scheme image. However an ipfs scheme can load http resources.
Settings
In addition to the toggle that already exists in brave://settings, we will have:
- A drop down which allows the change how Brave resolves
ipfsandipnsURLS. The user can select betweenAsk,Gateway,Local node,Disabled. TheLocal nodeoption only shows up when it is installed. - A setting which will allow the user to select which gateway to use
- A setting which allows the user to fallback to the public gateway when a local node cannot be accessed.
- A setting which will allow the user to turn on automatic redirection of IPFS gateway resources to use the local node instead.
Diagnostic page - brave://ipfs
To help troubleshoot IPFS in Brave, and to see daemon status, a new Chromium WebUI page is added.
This page will show if the IPFS daemon is started, and allow a user to start / stop the IPFS daemon.
It will also show the API, gateway, and swarm addresses.
A user will be able to get at standard output from the go-ipfs daemon inside this page. (Not implemented yet)
The user will have the ability to pass environment variables the user specifies, so that things like log verbosity can be changed. (Not implemented yet)
This page will also get API info from the local API endpoint:
https://docs-beta.ipfs.io/reference/http/api/
In particular for example, the number of peers:
https://docs-beta.ipfs.io/reference/http/api/#api-v0-swarm-peers
HTTP gateway fallback
If the user chooses in chrome://settings to use a gateway instead of a local go-ipfs node, then all requests to ipfs:// and similar will be redirected to the configured IPFS gateway.
Interstitial page when there are no peers
If there are no peers but an IPFS or IPNS URL is accessed, an interstitial page (such as bad SSL certificate) will be displayed telling the user that the node doesn’t seem to be started or there are no connected peers. An option will be given on the page to use the default gateway fallback and to continue to do so automatically in the future when there are no peers.
Admin policy
A top level IPFS enabled setting will be added and tied to an admin policy to completely disable all IPFS detection and functionality.
Browser tests will be added to browser/policy/brave_policy_browsertest.cc
The role of IPFS Companion and new chrome.ipfs API
IPFS Companion will be compatible but not required to be installed. We’ll leave it to Protocol Labs to optionally enhance IPFS Companion but we’ll assist via adding APIs that would be helpful.
Installing IPFS Companion will be a separate action and will not be bundled with the installation of the go-ipfs binary. Users will continue to get updates from the Chrome Web Store and can continue to install IPFS Companion from chrome://settings.
The following new API have been exposed to IPFS Companion only:
// Checks if a feature flag is enabled
chrome.ipfs.getIPFSEnabled((enabled: boolean) => {})
// Obtains a string representation of the resolve method
// method is one of the following strings:
// "ask" uses a gateway but also prompts them to install a local node
// "gateway" uses a gateway but also prompts them to install a local node
// "local" uses a gateway but also prompts them to install a local node
// "disabled" uses a gateway but also prompts them to install a local node
// Throws an error if IPFS feature flag is not enabled
chrome.ipfs.getResolveMethodType((method: string) => {})
// Launches a daemon
// Throws an error if IPFS feature flag is not enabled
// Throws an error if a local node is not installed
chrome.ipfs.launch((success: boolean) => {})
// Shutsdown a daemon
// Throws an error if IPFS feature flag is not enabled
// Throws an error if a local node is not installed
chrome.ipfs.shutdown((success: boolean) => {})
// Obtains the config contents of the local IPFS node
// Throws an error if IPFS feature flag is not enabled
// Throws an error if a local node is not installed
chrome.ipfs.getConfig((success: boolean, config: string) => {})
// Checks if the local node is installed
chrome.ipfs.getExecutableAvailable((available: boolean) => {})
Landing and shipping code behind a flag
In order to land code and not wait for the entire feature to be implemented, we’ll start by adding a flag in chrome://flags which will allow you to toggle on the experimental IPFS support.
Privacy-Preserving Product Analytics (P3A)
Pending review approval by the privacy team:
- IPFS companion installed? i) No, ii) Yes
- How many lifetime times are IPFS detection prompts shown without installing? i) 0 times, ii) 1, iii) 2-5 times, iv) 5+ times or more?
- IPFS Gateway setting - Ask (0), Gateway (1), Local Node (2), Disabled (3)
- How long did the IPFS daemon run? i) 0-5min, i) 5-60min, ii) 1h-24h, iii) 24h+?
Guest windows, Tor windows, and incognito windows
To avoid leaking IPs and other potential privacy issues, we've disable this feature in Tor windows.
Guest windows and Incognito windows also should not load IPFS and IPNS URIs on the local node since it would store content that is accessed.
Cache clearing (1.20.x and beyond)
When clearing cache in Brave, we automatically run ipfs repo gc.
Since this does not support time based deletions, we will clear the cache no matter which time interval is used.
Pinned items will be retained.
QA Plan
Settings:
- Make sure each IPFS setting functions immediately on the next page load as the changes are made.
- Check settings for how to resolve IPFS URIs
- Setting it to Gateway before installing should stop showing the infobar
- Setting it to disabled should not allow ipfs:// URIs to be loaded at all (being tracked in IPFS resolve type of disabled should disable #13472)
- After go-ipfs is installed, make sure Gatweay and local resolve correctly
- When a local node is configured, it should keep ipfs:// in the URL bar. When a gateway IPFS node is used, it should show the gateway address even if you try to load from an ipfs:// URI.
- When clicking a link from an ipfs:// page, it should still show ipfs:// in the URL bar if it was linking to a relative
Basic checks:
- Make sure navigating to an ipfs:// URL prompts for installing
- Make sure navigating to an ipns:// URL prompts for installing
- Make sure navigating to an https:// URL with /ipfs/[CID] prompts for installing only for the dweb.link IPFS gateway
- Make sure navigating to an https:// URL with /ipns/[CID] prompts for installing only for the dweb.link IPFS gateway
- Make sure Guest windows and Incognito windows cannot load IPFS URLs (Open blocking issue here IPFS and IPNS should not resolve in guest and incognito contexts #13463)
- Make sure Tor windows do not load IPFS URLs
- Make sure loading an ipfs:// URL works correctly after IPFS is installed
- Make sure session restore works correctly with a tab left open with ipfs://
- Should be able to use ipfs:// local node from every channel (Release, Beta, Nightly) at the same time. (They all use different ports)
WebUI page:
- You can use chrome://ipfs to see the status of things
- You can click to start and stop the node.
- This page is only meant for support / debugging though. Things like not automatically updating without a reload and making the page more usable doesn't need to be posted.
Lazy loading:
- Restarting the browser should not load the IPFS process in task manager
- But a navigation to an IPFS resource should start it.
Subresource testing:
- You can test these by following these steps: subresources.txt
- ipfs page should allow other ipfs subresources to be loaded within it
- http page should not allow ipfs subresources within it
- fetch should work on an IPFS page
IPFS Companion:
- Protocol Labs has a newer version that detects Brave and shows a panel about how to use the Brave node. I'm not sure if it is release yet. For our QA we should just make sure at least that having it installed doesn't cause any problems.
P3A:
- I verified all of the P3A issues already, no need to test that.
API:
- I verified the exposed APIs and Protocol Labs is using them already in IPFS Companion, no need to test that.
There are 3 issues that landed in past milestones, but they were originally marked as QA/No because the feature was disabled.
These should be tested according to the following test plans:
- Admin policy test plan:
- When there are no peers connected, should an interstitial page.
- Different origins shouldn't be able to set cookies
Here are some example gateway URLs:
- An HTTPS default IPFS gateway URL
- An HTTPS non-default IPFS gateway
- An IPFS scheme URI
ipfs://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq/wiki/Vincent_van_Gogh.html(not a link because github markdown doesn't allow ipfs://) - An IPNS URI:
ipns://Qmc9xb2cgWKgz8rCBihag6fmvvEu7B7nbmYPxD6RcFXe1Z
Talking to the IPFS node:
Create a test.txt file with "hello" inside of it.
tamarin:1.0.2 bbondy$ echo hello > test.txt
Add that file to your IPFS node
Note: Check which API port in chrome://ipfs and adjust accordingly.
tamarin:1.0.2 bbondy$ ./go-ipfs_v0.6.0_darwin-amd64 --api=/ip4/127.0.0.1/tcp/45004 add test.txt
added QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN test.txt
6 B / 6 B > [=============================================================================================================================================] > 100.00%
You can now load this file in the browser, or from anywhere on IPFS by loading:
ipfs://QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN
It will be slower if you are loading from a different node.
You can find this binary inside your profile directory. The profile dir and component ID folder name differs depending on which platform you are on.
tamarin:1.0.2 bbondy$ pwd
/Users/bbondy/Library/Application Support/BraveSoftware/Brave-Browser-Beta/nljcddpbnaianmglkpkneakjaapinabi/1.0.2
You can publish an IPFS CID to IPNS using a command like this:
tamarin:1.0.2 bbondy$ ./go-ipfs_v0.6.0_darwin-amd64 --api=/ip4/127.0.0.1/tcp/45002 name publish /ipfs/QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN
Published to Qmc9xb2cgWKgz8rCBihag6fmvvEu7B7nbmYPxD6RcFXe1Z: /ipfs/QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN
Then you can load it with ipns://Qmc9xb2cgWKgz8rCBihag6fmvvEu7B7nbmYPxD6RcFXe1Z
More go-ipfs command line info can be learnt here: https://docs.ipfs.io/reference/cli/