New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPFS Integration - Roadmap and discussion #819

Open
bsclifton opened this Issue Aug 23, 2018 · 15 comments

Comments

Projects
None yet
7 participants
@bsclifton
Member

bsclifton commented Aug 23, 2018

Carried over from brave/browser-laptop#9556

@diasdavid wrote:

Hi there brave team! I'm David, from the IPFS project.

I'm looking into the IPFS integration into the Brave browser and I'm hoping you could clarify some design details, just so that I'm sure I understand how everything is pieced together.

Right now, the most straightforward way is to mimic what WebTorrent did and create an extension. I've noticed that it gets its own WebPage, located at https://github.com/brave/browser-laptop/tree/master/js/webtorrent and that this page gets bundled in with WebPack -- https://github.com/brave/browser-laptop/blob/master/webpack.config.js#L153-L169 --. I can create something similar, no problem.

  • Where do I capture ipfs://, dweb:// and even http(s)://ipfs.io/ipfs/ urls so that they get resolved through the IPFS Extension?
  • How do I create a control panel like the one I see Dashlane/1Password
    image
  • Currently, we have full web applications that are hosted and served through IPFS. We can start by just adding IPFS support for download of files but ideally, we would like to see pages be rendered. Any advice on how to hint Brave the content type? Drop it into the dom like it's 🔥?

Thanks in advance! Super excited to do this :)

@drbh

This comment has been minimized.

drbh commented Aug 24, 2018

What are the next steps to shipping Brave with IPFS?

My understanding is that Brave is going to use a custom build of Chromium because it does not support custom protocols without the web+ prefix.

Quoted from arewedistributedyet/issues/23

Brave will use custom build of Chromium: they could override whitelisting rules as a part of the build, but it makes more sense to try to solve it upstream in Chromium first.

However he road block is upstream; getting Chromium to support DWeb protocols - which seemingly has been officially started on Aug 17 2018

Quoted by asanka@chromium.org from /chromium/issues/detail?id=651311

I'll start an intent-to-implement.

Assuming the protocols get supported and are included in a near future build of Chromium. What will the general integration process look like?

Will the navigator.registerProtocolHandle still be used or will this not be needed anymore?

If it is used, is the sample code below seem close to the general concept? Redirecting ipfs:// traffic to a ipfs gateway?

navigator.registerProtocolHandler("ipfs",
                                  "https://gateway.ipfs.io/ipfs/%s",
                                  "IPFS");
@lidel

This comment has been minimized.

lidel commented Aug 24, 2018

On HTTP-based Protocol Handler

When requirement for web+ prefix is removed, we can add code to register redirect-based handler on first use.

The problem with having navigator.registerProtocolHandler without webextension support via something like manifest.json/protocol_handlers is that it comes with severe UX limitations described in arewedistributedyet/arewedistributedyet#23 (comment) (previous comments are also a good read).

Still.. better than nothing :)

On Native Protocol Handler

As noted in brave/browser-laptop#9556 (comment) API at navigator.registerProtocolHandler is just a redirect-based kludge that makes URLs work and open, but in the longer term, we'll need a real, programmable protocol handler.

An experimental, native protocol handler API is being designed as WebExtension API at mozilla/libdweb/#protocol-api and ipfs:// is WIP at ipfs-shipyard/ipfs-companion#533. My hope is that when libdweb experiment matures to the point it is safe to include it in upstream Firefox at some point, we could ask Chromium to implement it as a part of WebExtension APIs as well. We could also think about at implementing it in Brave without waiting for Chromium, of course.

It is in a relatively a far future, and we could plan for delivering integration without native handler (see below).

Integration Milestones

Assuming the protocols get supported and are included in a near future build of Chromium. What will the general integration process look like?

Short Term: Choosing HTTP backend for IPFS Companion

Without native protocol handler API we can't fetch data from IPFS directly and need to use HTTP gateway so URLs still work and can be opened by HTTP-based handler registered with navigator.registerProtocolHandler.

HTTP Gateway can be provided by either locally running go-ipfs or js-ipfs running in Node (js-ipfs running in browser context can't open TCP port to serve HTTP requests). We could also just use public HTTP Gateway (go-ipfs) at https://ipfs.io/ipfs/* but that is a centralized fallback without doing actual p2p.

In the short term I would go with go-ipfs due to reasons listed in ipfs-shipyard/ipfs-companion#312 (comment).

I see three ways to deliver integration:

  1. The go-ipfs is shipped and orchestrated by Brave: would be something similar to Tor Tabs (very good UX, IPFS works out of the box)
  2. Ask user to install ipfs-desktop on first use of IPFS addresses (bad UX, surface for errors).
  3. Use public HTTP gateway for reads coupled with embedded js-ipfs for writes (+ user has an option to install go-ipfs manually and switch to local node in preferences)

If (1) is not possible then go with (2) to minimize maintenance burden and user confusion.

I would love to see (1) – what are the next steps to move in that direction?

Long Term: Adding Missing APIs for Native Handler, Discovery and Transport

To get all benefits of IPFS (and other p2p protocols) we need at least:

@drbh

This comment has been minimized.

drbh commented Aug 24, 2018

The go-ipfs is shipped and orchestrated by Brave: would be something similar to Tor Tabs (very good UX, IPFS works out of the box)

I really like shipping go-ipfs with Brave! 👍

Not only is it great UX, but it add a competitive advantage to Brave from a product growth perspective. Similar to how Tor Tabs incentives users to move from the commercial browsers to Brave (because of its added feature set)

How do we go about adding a program the way Tor Tabs did.

Looking through the project in brave-laptop I find

browser-laptop/blob/master/app/tor.js

/**
 * Tor daemon management.
 *
 * This doesn't actually manage the tor daemon: the parts that did are
 * commented out.  Rather it just watches the tor daemon's control
 * port file for activity and connects to its control socket.
 *
 * @module tor
 */

and
browser-laptop/blob/master/tools/package_tor.js

which seems to pull the binaries from S3 (interesting)

const torS3Prefix = 'https://s3.us-east-2.amazonaws.com/demo-tor-binaries/'
var torPath = process.argv.slice(2)[0] // npm run package-tor torPath
if (torPath === undefined) {
  torPath = path.join('app', 'extensions', 'bin')
}

Does this suggest we need to host the IPFS go binaries somewhere and these are pulled in and installed with Brave?

Is there someone who worked on TorTabs and has insight to the process of shipping a binary with Brave?

@bsclifton

This comment has been minimized.

Member

bsclifton commented Aug 24, 2018

@drbh

This comment has been minimized.

drbh commented Aug 25, 2018

@bsclifton thanks for suggesting help from @riastradh-brave and @darkdh.

Looking at their commits pertaining to Tor build packaging and the build_tor_scripts I've pieced together these basic next steps:

  • Build go-ipfs from source for all supported platforms linux, darwin and win32 like brave/tor_build_scripts builds Tor
  • update the tools/buildPackage.js file to include ipfs binaries
  • add new file called tools/package_ipfs.js that fetches and verifies the signature a precompiled ipfs binary

i've added a repo at /drbh/ipfs_build_scripts as a starting point for the build scripts.

also i've created a fork of brave/browser-laptop and added a branch called ipfsPackageBuild at /drbh/browser-laptop with what seems like the necessary files for brave to fetch the binaries.

The signature need to be updated in package_ipfs after the binaries are correctly built, and some adjustments to ipfs version are needed.

I'm most confused about the build process for the ipfs_build_scripts, because the tor_build_scripts seem to add dependencies in a specific way like openssl. Since go-ipfs's only dependency seems to be Golang, can we build the binaries straight from a golang:1.10-stretch base image?

If anyone has insight on the build process this would be greatly appreciated. I hope my boilerplate code is helpful.

@lidel

This comment has been minimized.

lidel commented Aug 25, 2018

@drbh thank you, the insight is really appreciated!

Yes, our official Docker image for go-ipfs is built on top of golang:1.10-stretch, there should not be any additional moving pieces apart from Dockerfile itself.

We also have https://github.com/ipfs/distributions which is responsible for fetching and publishing pre-built artifacts at https://dist.ipfs.io/#go-ipfs website. I can reach out to people and see how much of already existing process can be reused.

Before we dive into details, I have a question about ipfs_build_scripts: is build from source mandatory? Are you building Tor for Linux with a specific goal, eg. to solve dynamic linking issues present in the original build or something like that, or is it just a security/transparency policy?

I ask because we already have pre-built go-ipfs binaries for platforms you target with Brave and we could download them from there just like it is done with Tor on win32 here. It would simplify entire setup by removing a lot of moving pieces.

@drbh

This comment has been minimized.

drbh commented Aug 25, 2018

@lidel your a genuis! Pointing out how Tor win32 is built was super helpful.

Before we dive into details, I have a question about ipfs_build_scripts: is build from source mandatory?

Retrospectively it seems like overkill to build from source. I was doing so only because I was following the path that the Tor binaries needed.

I understand now that they only built them from source for dynamic linking purposes - it seems that the Windows binary doesn't need the same dyn linking that the darwin and linux bins do

I ask because we already have pre-built go-ipfs binaries for platforms you target with Brave

Based on this we have two paths forward - including some sample code to get close to the intended goals

  1. Directly downloading the ipfs precompiled binary - unzipping and storing.
    Edits would take place in /tools/package_ipfs.js
const request = require('request');
const os = require('os');
const fs = require('fs')

const tmpPath = `${os.tmpdir()}/ipfs-binary`;
var binary;
var opts = {
	"uri":"https://dist.ipfs.io/go-ipfs/v0.4.17/go-ipfs_v0.4.17_linux-amd64.tar.gz", 
	gzip: true
}

console.log(tmpPath)
request(opts, function (err, res, body) { 
	fs.writeFile(tmpPath, body, 'binary', console.log);
})

  1. Creating a bash shell build env - similar to how TorTabs did, to build rename and push binaries to S3
    Edits would take place in https://github.com/drbh/ipfs_build_scripts
#!/bin/sh
set -eu
rm -rf go-ipfs/

export IPFS_WIN32_VERSION="0.4.17"
export BRAVE_TOR_VERSION="5"

curl -fsSl "https://dist.ipfs.io/go-ipfs/v0.4.17/go-ipfs_v${IPFS_WIN32_VERSION}_windows-amd64.zip" -o ipfs-win32-$IPFS_WIN32_VERSION.zip

unzip ipfs-win32-$IPFS_WIN32_VERSION.zip

cp go-ipfs/ipfs.exe .
mv ipfs.exe ipfs-$IPFS_WIN32_VERSION-darwin-brave-$BRAVE_TOR_VERSION.exe

It seems like the first way of fetching the binaries within Brave is the easiest path forward. However it is noted that Tor binaries have been compiled and pushed to a S3 location. Maybe there is value in this data flow, however it seems unneeded for IPFS because it doesn't depend on dynamic linking.

I can take on the /tools/package_ipfs.js file and downloading IPFS directly if this seems like the best way forward

@lidel

This comment has been minimized.

lidel commented Aug 25, 2018

@drbh To keep momentum going let's go with direct download for now (1) and see if that works well enough. If it turns out we need custom build for some platforms or Brave decides they want to host files at S3 to control user experience end-to-end, we can always add step (2) at a later stage.

FYI dist.ipfs.io has a valid DNSLink, which means fetching from https://ipfs.io/ipns/dist.ipfs.io/go-ipfs/v0.4.17/ (or any other public gateway) will also work.

If we have that, next step would be making sure ipfs daemon is initialized and configured for Brave.
Some things from the top of my head:

  • pass IPFS_PATH to control where to store configuration and data
  • initialize repo and configuration at IPFS_PATH
  • customize configuration (go-ipfs/docs/config.md)
    • enable local discovery (so two users having Brave+IPFS can talk directly to each other on LAN)
    • run in dhtclient mode (laptops go offline which hurts DHT)
    • enable gc and set quota for IPFS data/cache
    • limit number of connections (consumer routers may not appreciate high number of connections)
    • configure ports for HTTP Gateway and API (we should detect if port is already used and pick other one, then have a way of passing that info to companion)
@darkdh

This comment has been minimized.

Member

darkdh commented Aug 25, 2018

for packaging binary, I suggested taking a look at brave/brave-core#316, we use different method in brave-browser.

@drbh

This comment has been minimized.

drbh commented Aug 25, 2018

@darkdh thanks for the suggestion. I am looking at /brave-core/pull/316 now and I don't fully understand where the binary is pulled and packaged.

Looking in /brave-core/tree/master/browser/extensions I see a few files that reference Tor but they seem to only reference a Tor client update.

Questions

  1. Where is TorTab's Tor packaged?
  2. What is the difference of packaging in Brave Browser vs Brave Core?
  3. Are parts of Brave Browser's Tor client related to the Brave Core Tor client?
  4. Is packaging IPFS in Brave Browser less reliable in some way then Brave Core?

Any clarification is greatly appreciated! Thanks

@drbh

This comment has been minimized.

drbh commented Aug 30, 2018

Updates!

  • Build go-ipfs from source for all supported platforms linux, darwin and win32 like brave/tor_build_scripts builds Tor
  • update the tools/buildPackage.js file to include ipfs binaries
  • add new file called tools/package_ipfs.js that fetches and verifies the signature of a precompiled ipfs binary

In addition - the biniary is pulled via dist.ipfs.io

FYI dist.ipfs.io has a valid DNSLink, which means fetching from https://ipfs.io/ipns/dist.ipfs.io/go-ipfs/v0.4.17/ (or any other public gateway) will also work.

Issues

  • ipfsd-ctl not configured for storage
  • ports are not working when attempting to specific on ipfsd-ctl server and daemon start
  • ipfs daemon fails to start with npm start likely due to app/filter.js not correctly setup

ipfs daemon starts when the ipfs.js script is loaded into a node command line.

Error

iipfs-brave-fail

Above code that fails:

mystart () {
    console.log('My Start')

    server.start((err) => {
      if (err) { throw err }
      console.log('Server Start')
      console.log(f)

// FAILS HERE - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
      f.spawn(options, (err, ipfsd) => {
        if (err) { throw err }
        console.log('Deamon Start')
        // this._process = ipfsd

        console.log('http://' + ipfsd.api.apiHost + ':' + ipfsd.api.apiPort + '/webui')

        ipfsd.api.id(function (err, id) {
          if (err) { throw err }

          // console.log(id.api.apiPort)
          // ipfsd.stop(server.stop)
        })
      })
    })

    console.log('Server Has Been Started')
  }

I don't exactly understand why the build fails when using npm start the install and packaging process succeeds with npm install

IPFS packaging and barebones daemon management - ipfsPackageBuild branch

@lidel

This comment has been minimized.

lidel commented Sep 5, 2018

@drbh apologies for not getting to it earlier, slipped thru my notifications.

The main use for ipfsd-ctl is test suite of IPFS itself,
so it may be rough around edges specific to our use case.

Are you still blocked by the above error?
I see you work on a fork of browser-laptop, which is the old Muon-based codebase.
Shouldn't we aim for brave-browser/brave-core instead? (following these steps)


Update: I think a safe path would be to create a PR to brave-core-crx-packager that creates CRX bundle with IPFS binary similar to one already present for Tor, and then get some feedback on what are the next steps for the Chromium version.

@drbh

This comment has been minimized.

drbh commented Sep 5, 2018

Just submitted a PR brave/brave-core-crx-packager#21 following the Tor CRX bundling

@da2x

This comment has been minimized.

da2x commented Oct 2, 2018

Will creators be able to earn Brave Rewards for content distributed over IPFS?

Creators could claim their IPNS by publishing the brave-payments-verification.txt under their IPNS. Brave could verify this using a public HTTP gateway such as https://cloudflare-ipfs.com/ipns/<id>/.well-known/brave-payments-verification.txt.

The Brave client would have to recognize and attribute time spent on http://127.0.0.1:8080/ipns/<id>/*, https://*/ipns/<id>/* (public gateways), ipns://<id>/*, and dweb://ipns/*<id> to a normalized address like ipns://<id> and handle this in the rewards system.

@mathiasrw

This comment has been minimized.

mathiasrw commented Oct 8, 2018

@da2x This is such a good idea. Lets implement IPFS first and then integrate it into the attention economy!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment