Add local gateway to allow browsers access to the network.

[RangerMauve]

The IPFS-companion extension has been doing amazing things for allowing browsers to integrate with the network.

They allow stuff like rerouting ipfs:// URLs to the local gateway and providing a global window.ipfs API that allows web pages to read and upload stuff to IPFS.

These features work by using an HTTP gateway for the local IPFS node installed on the user's machine which the extension will reroute requests to and proxy the global IPFS object API calls to.

Similar features in dat-land can only be found by installing the Beaker browser, which although it's really cool, it probably won't be the default choice for most casual users of the web.

Beaker supports the dat:// protocol and a DatArchive global which could both be provided through a similar extension as IPFS-companion.

In order to replicate what IPFS-companion did for a dat-specific extension, the extension would need some way of accessing dat since we can't just run node in a browser extension.

If dat-desktop allowed users to access all the dat APIs from an HTTP gateway, all that people will need to access data from dat and use distributed web apps is to install dat-desktop and a browser extension for whatever browser they're already using.

Would PRs be welcome for this functionality?

[martinheidegger]

Having a browser plugin / integration that works out of the box is certainly a nice thing!

By skimming over the implementations of ipfs it didn't become clear to me what API's a companion extension would need. Do you have better insight? If not it might be very good to collect the data necessary.

Aside from that: It is my goal to disconnect dat-desktop from a dat-daemon that stores/manages the local dats. And it seems to me like it should be rather this deamon's job to provide the http API's.

[RangerMauve]

@martinheidegger The docs for the IPFS HTTP API go over everything that's available, but it's essentially everything that the JS or Go APIs.

For the dat HTTP API, it should provide everything that the JS API provides. It should also have access control headers to only allow local connections to access it so that malicious web pages can't try to access it directly.

Would a PR against dat-daemon to add an HTTP API be welcome?

[martinheidegger]

dat-daemon is a few days old. It should provide all API's - at least the very basic ones (more complex stuff is still hidden - reading code right now to know more.

[okdistribute]

@RangerMauve yes!

[RangerMauve]

Looking at dat-daemon, I don't know if that's the right place to shove this.

I love the project, but it looks like it'd be better geared to managing archives rather than an RPC API that facilitates browser extensions.

I think that a good way forward would be to

• take node-dat-archive which already implements the DatArchive API from beaker
• wrap it in an RPC API (using websockets, but have it work over arbitrary duplex streams)
• embed the RPC into dat-desktop so that people can enable it with a flag.
• Having it just be standalone isn't good UX for casual users.
• Have an extension that will load dat URLs using the RPC API
• Have the extension provide a DatArchive API which uses an RPC to talk to the extension worker which then does access restrictions and relays RPC to dat-desktop

[sammacbeth]

I am currently working on a prototype firefox extension to do just this, using dat-gateway to bridge http to dat. There is also another project doing the same for chrome.

In the extension's current state, it is possible to load dat sites, but I have not yet started on implementing the DatArchive API.

[RangerMauve]

@sammacbeth Any ideas so far for the DatArchive API?

How does your extension deal with setting the origin to be different between dats?

[sammacbeth]

@RangerMauve For the DatArchive API I'm currently testing Native Messaging for a private communication channel between the extension and node process. Injecting the API in the document can be done in the same way as the ipfs-companion extension.

Regarding setting origins correctly, one can use a PAC file (via the extension proxy API) to implement this by proxying dat-like hostnames to the gateway, which then receives GET http://{dat key}/path, and can then return the correct document via dat.

[RangerMauve]

@sammacbeth I think I missed something, what does the origin get set to when I navigate to dat://mydaturlhere?

[sammacbeth]

@RangerMauve dat://datproject.org/ is redirected to http://datproject.org/ by the protocol handler, then proxied to the dat gateway by the PAC file.

[RangerMauve]

@sammacbeth Sorry, I think I didn't explain myself well enough.
There were problems with gateways for stuff like IPFS in that all files served from the gateways were getting the same origins and therefore were able to access each other's cookies and localstorage.

With your approach, will different dat:// URLs be served under different origins, or will it just redirect to the gateway and in turn use the gateway's origin?

[sammacbeth]

@RangerMauve Yes, that is exactly the problem the extension solves vs. simply using the gateway server.

[RangerMauve]

That's awesome! Thank you for clarifying! :D

[RangerMauve]

Just gonna bump this since I've got the dat-polyfill to a state where it can be used to run Fritter using a gateway.

Also, if there's a gateway set up in dat-desktop, you can use registerProtocolHandler to have dat-desktop intercept dat:// urls, and have it change the url to point to the local gateway and open in the user's default browser.

That would enable people to just install the desktop and not worry about browser extension compatibility.

[sammacbeth]

I've put together a test suite for DatArchive implementations, with the aim of helping the emerging implementations to converge and work with applications build for the Beaker browser reference implementation. It helped be find quite a few issues with the dat-fox implementation.

The source is available here and I'm happy to take contributions to improve the coverage.

[RangerMauve]

@sammacbeth That's awesome! I'll make sure to try to run it with dat-polyfill when I've stabilized it!

What's funny is that Beaker itself is failing on the read as base64 and read as hex tests. :P

[soyuka]

This is good news! Adding the gateway to the desktop app would indeed be nice!
If you want to work on this please base your work on the react branch :).

[RangerMauve]

Awesome, I've still got some stuff to do for dat-polyfill, but after I'm done with that I'll make this my next priority. :D

[sammacbeth]

@RangerMauve yes there was something I couldn't figure out with the b64 and hex encoding. In the hex one it seems the endianness of the non-ascii characters is different. I'm not sure if this is a mistake in the way the test converts the data, or that the data is actually stored on disk this way. Note the writing tests don't have this problem (what is read matches what was written).

[martinheidegger]

I read into dat-polyfill and other solutions and my opinion is that we need a "service"/"manager"/"daemon" that allows to bridge the data towards a client.

@soyuka built https://github.com/soyuka/dat-daemon which is the closest to what I think would be a good idea. But it does have a few issues that make it hard to use with dat-desktop. That being said, I certainly would love to see a PR against #516 and #458 respectively.

[RangerMauve]

@martinheidegger I was thinking that having dat-desktop running would be the daemon.

My main reasoning is that even though developers are fine with setting up daemons and talking to them via the CLI, casual users aren't going to bother with all that and would get a lot more utility out of an all-in-one solution.

As well, I don't think a demon should be accessible through browsers since it lack security. A dat-gateway is simple in that it doesn't really persist anything. If a user wants persistance, they can use the UI in dat-desktop to acheive it. (Maybe in the near future we could have a way to persist a dat that was created in the browser by having an API that takes the public and private key to ensure it wasn't created willy-nilly.

[martinheidegger]

@RangerMauve My thinking is that dat-desktop should come with a daemon and start it if no other daemon is present but can fallback to another, locally running daemon. Maybe this could be the same with dat-cli?

The most straightforward way I can think is that the daemon communicates through a tcp port. As such it could be accessible through an extension (even though it might not be available from a website). This could be used to add a dat from the browser directly show the continuous download progress of a dat (clicked in the browser).

[RangerMauve]

With regards to extensions, I think that this libweb work will replace the need for a gateway down the line, actually.