New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dat / Beaker browser support #246

tswast opened this Issue May 27, 2017 · 9 comments


None yet
6 participants

tswast commented May 27, 2017

Related: IPFS support #156

Dat is a similar protocol to IPFS. The P2P Beaker Browser chose to use Dat instead of IPFS because it has more polished support for mutable data (with versioned history).

Seems like it might be a better fit for neocities, too.


This comment has been minimized.


kyledrake commented May 29, 2017

Thanks for the inquiry!

I generally like Dat, but the messaging I've gotten is that they're focusing on scientific datasets, whereas IPFS is focusing on the distributed web. A distributed web could conceivably be built on top of Dat, but IMHO, replacing HTTP isn't an "oh, by the way, it could maybe do that too" kind of idea. Building a replacement to HTTP is a big deal, billions of people will be using it. It will require a lot of focused design choices that aren't necessarily going to be in the same problem domain as Dat's core use case. Without strong signaling that the distributed web is going to be a top priority, I'm not willing to drop our IPFS code, knowledge, and great access to the IPFS team (whom have gone out of their way many times to improve IPFS for our use case) to try something else that isn't trying to build precisely, exactly, publicly and passionately a distributed web implementation.

That said, a few specific technical issues also come to mind:

As it was described to me, a Dat mutable pubkeyhash starts by referencing an object which contains links to all historical versions of the archive. Some of our sites have tens of millions of versions. In order for Dat to work well for us, we would need a way to quickly get the latest version of a site, even if it's historically had a lot of updates. Does Dat provide a quick way to get just the latest version of an archive, or does the user need to retrieve that full version history object to get the latest version? Maybe this isn't an issue, but I've never gotten a solid answer on this.

Second, how does tooling work for a lot of keys? We host ~140k sites and our long term goal is to set up a mutable key for each one. The design of the Dat tools I've looked at seems to be focused around hosting "several" archives, not hundreds of thousands. It's possible the libraries allow this but I haven't seen any published software or example code to this effect.

At both of these things, IPFS appears to be "more polished" right now for our needs. The go-ipfs daemon provides the ability to create and announce as many keys (IPNS is the mutable key structure for IPFS) as I want, and the DHT quickly distributes the new version of the pubkeys. There is no constraint on the number of updates that can be made to sites. Things like version history can be built on top of IPFS with IPLD, but in a way that we can choose different implementations should one implementation not work well for our needs. IMHO that allows for more of a protocol democracy than an implementation that has a "single party" opinionated design.

RE Beaker: I do think Beaker is the right approach to the browser problem, at least in the short term. Long-term, the promise of an incentivized, distributed last-mile CDN combined with Net Neutrality being gutted I think will be too hard for the mainstream browsers to ignore (It hurts my brain to think about how much money Google will save on serving sites like Youtube using something like IPFS). But as of today, the Web Extension spec doesn't allow an extension to control the address bar for registered custom protocols, preventing things like proper origin security (but it, of course, has 0 problems with letting an extension hijack attempts to access the settings page, like the malware was doing that I just had to rip out of my friend's Chrome).

Our recent plan was to launch IPFS support with DNSLINK for Neocities sites and point users to Beaker as a browser they could use to try it out. Unfortunately Beaker appears to have made a decision to remove support, so that plan is now cancelled.

For the moment, we're kindof stuck in the mud on distributed web support until someone puts together a browser that can support IPFS (and no, a plugin that uses won't cut it, we need root origin support with DNSLINK). I will re-evaluate in the future to see if the situation has improved.


This comment has been minimized.

zeke commented May 31, 2017

Does Dat provide a quick way to get just the latest version of an archive, or does the user need to retrieve that full version history object to get the latest version?

cc @Kriesse @juliangruber @mafintosh


This comment has been minimized.

mafintosh commented May 31, 2017

Yes Dat supports getting the latest version quickly without having to download all of the history. This is a core feature.


This comment has been minimized.

pfrazee commented Jun 7, 2017

@kyledrake quick response --

You're right about Dat's project goal- it's not about the distributed Web, but Beaker is. We work with the Dat team, and if they disappeared tomorrow, we'd pick up the spec and keep it going. I don't think the tech's origin is going to matter as much in the long run.

We passed over IPFS because of the NURIs spec and their handling of the feedback. NURIs are unnecessarily problematic for us and the other browser vendors. When it came time for a decision on which tech to use, we felt better about how the Dat team worked with us, and figured that was more important than the lack of polish in the software (which you're not imagining). We probably would have moved faster by using IPFS, but I've been happier with Dat as a partner.

About the performance with histories, I don't get any indications that Dat's design is going to be slow. Mafintosh has been doing great work improving performance and adding features. I expect both IPFS and Dat will have scaling challenges but perform fine in the long run.

I can see how IPFS' tooling would be better for Neocities. At this point you have to build out Dat tools using nodejs and npm modules, and you have to be pretty in tune with their protocol work. Again, it's worked well for us, but it isn't generally convenient yet.

We'll see how things shake out.


This comment has been minimized.


kyledrake commented Jun 8, 2017

@pfrazee Thanks for the explanation. Your reasoning is quite understandable, though I do hope in the future you will consider implementing js-ipfs. I personally don't want to see a dweb browser war break out this early, I know from deep experience that it's not going to be good for anybody (the way these sorts of things usually turn out: deepest pockets win). On a completely personal and candid level, I'd rather have someone principled like you and your colleagues running the dweb browser implementation, with support for the protocols people want to use, than have 3-5 smaller competing implementations for each one that end up getting rolled over by The Big Scary One that's coming (example: Microsoft in the 90s when they used IE to try to, and almost succeed at, taking over the web). Just establishing legitimacy of the dweb concept looks like it's going to be a big enough problem as-is

For now, what I can do RE Neocities while we wait for a complete dweb stack (and/or humanity to "get ready" for something that's completely obvious and inevitable) is be able to attach pubkey hashes to DNS records. This is going to be a pretty massive undertaking in-and-of-itself, because it means we'll have to run our own DNS servers which we currently pawn off to DNSimple, which probably isn't going to like me throwing 140k records into their database.

This repo already has too many unactionable tickets open, so I'm going to close this one for now since there really isn't a path forward here. Thanks everyone for chiming in.

@kyledrake kyledrake closed this Jun 8, 2017


This comment has been minimized.

pfrazee commented Jun 8, 2017


A historic anecdote that's on my mind because I just listened to it: The French revolution of 1830 was co-opted at the last minute by a few inside players, turning it from a second Republican revolution to a monarchical regime change. The new monarch retold the revolution as having always been about the ascendance of the new king, frustrating a group of students who saw any monarchy as oppression. When they attempted a second revolution in 1832, they failed because the new king was relatively reasonable, and hadn't pissed anybody off lately. They all died in battle, but, on the bright side, their failed revolution of 1832 became the backdrop to the classic story, Les Miserables.

I believe a steamrolling is inevitable once a dweb browser gets the attention of existing vendors. What I hope is that we'll be able to define what the dweb is in the public mind before big players get involved. Fragmentation is upon us, whether we want it or not: at the browser level with Mist, Blockstack, and Brave, and at the tech level with Dat, SSB, IPFS, Zeronet, and the coins. I unfortunately don't think the solution will be an umbrella browser. Instead, it will be a well-designed stack, with clear goals, benefits, and underlying mechanics.

RE the public keys, I should point out, we opted to use a .well-known URL over TLS to distribute them in beaker/dat.


This comment has been minimized.

zeke commented Jun 9, 2017

I love hearing all this history and following these interesting hyperlinks. Keep going!


This comment has been minimized.


kyledrake commented Jun 11, 2017

Just a quick postscript for future travelers, incase it wasn't obvious:

I've decided to focus on IPFS for the implementation side of dweb for now, and will continue to collaborate with them on those efforts in the future. Thanks again for the input.


This comment has been minimized.

bnvk commented Jul 25, 2017

I personally don't want to see a dweb browser war break out this early

Amen to that, brother. However, as per @pfrazee point "fragmentation is upon us" rings true, and

a well-designed stack, with clear goals, benefits, and underlying mechanics.

Seems a sensible way to avoid conflict. Some of the Ethereum / Parity team is working to a create a neutral organization in this direction: Web 3 Foundation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment