Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

State of multiwriter? #230

Open
aral opened this issue Oct 9, 2018 · 11 comments

Comments

@aral
Copy link

@aral aral commented Oct 9, 2018

I was just wondering what the state of multiwriter support is in Hyperdrive currently. I see that the last work on multiwriter was around four months ago.

Is it still within the near-term roadmap to integrate multiwriter into mainline hyperdrive and DAT CLI?

@RangerMauve

This comment has been minimized.

Copy link
Contributor

@RangerMauve RangerMauve commented Oct 9, 2018

I think @jimpick has been working on it in his spare time. He's got a fork with the latest and greatest changes somewhere.

@pfrazee

This comment has been minimized.

Copy link
Collaborator

@pfrazee pfrazee commented Oct 9, 2018

It is still being worked on! @mafintosh has been steadily finalizing it while also working on the new discovery code. Jim has also been building out those test demos, as Ranger says.

@Frando

This comment has been minimized.

Copy link
Contributor

@Frando Frando commented Dec 29, 2018

It would be great if the changes by @jimpick could be pulled in. I didn't go through all of them, @jimpick would you do a PR to here? Or is there still experimental stuff in your branch?

It had some fixes that I needed so I started from there for some patches, which I could also try to rebase on @mafintosh's branch but wouldn't do that if not needed.

@jimpick

This comment has been minimized.

Copy link

@jimpick jimpick commented Jan 1, 2019

It's still pretty experimental... the last month has been very busy with travel and holidays, so apologies if it all seems stalled.

There are a lot of hacks in there to get it to work as much as it does, so it's not really production quality.

I've noticed that with multiwriter, when things don't work, it often results in deleted files, so it's probably not a good idea to just release the changes "into the wild" without a lot of review and discussion.

I gave a talk in Tokyo where I showed the Dat multiwriter command line interacting with a version of dat-shopping-list (with updated dependencies). I had to make a few quick fixes for that to work... the latest commit is here:

jimpick@a2648fb

Here are the slides from my talk:

https://tokyo-dat.jimpick.com/

If I can find some time in the next month, I really should write that up into a blog post.

I had previously talked with @joehand about possibly releasing the pre-release version using his dat-next npm package.

@aral

This comment has been minimized.

Copy link
Author

@aral aral commented Jan 1, 2019

@jimpick What are your thoughts on multiwriter vs multi-feed?

(I see them as complimentary: multiwriter for the use case of a person authorising additional devices for their master feed and multi-feed for “conversations” between different people.)

Would be great to perhaps get a small working group together to scope this out (e.g., lack of deauthorisation is an issue… one that could potentially be worked around in app implementations via a combination of a pasword-derived keypair and logout messages for specific devices.)

I’d love to be involved with this as it’s central to what I’m building.

CC @noffle

@jimpick

This comment has been minimized.

Copy link

@jimpick jimpick commented Jan 1, 2019

Multifeed is really useful when you want to replicate multiple feeds over a single replication stream. I actually built something very similar inside hypermerge (I was inspired by what hyperdb did internally). Hyperdb (and multiwriter) could probably even use it internally to manage the hypercores it replicates...

@okdistribute

This comment has been minimized.

Copy link
Collaborator

@okdistribute okdistribute commented Apr 10, 2019

Hey all, @mafintosh and @andrewosh just put out a new hyperdrive release candidate, 11 days ago:
https://github.com/mafintosh/hyperdrive/blob/3877eec49e6c0244b0df62d57cbc9e57b4d9224e/README.md

This is the first step towards multiwriter.

Time to try it out and give feedback and improve it! Really excited to see this roll out this year.

@aral

This comment has been minimized.

Copy link
Author

@aral aral commented Apr 10, 2019

This is so exciting; can’t wait to try it out (just working to finalise Indie Web Server, one of the core components of Hypha so the timing is perfect). Thank you for the heads up, @karissa :)

@joehand joehand pinned this issue Apr 10, 2019
@100ideas

This comment has been minimized.

Copy link

@100ideas 100ideas commented Jun 26, 2019

@aral last year @jimpick experimentally patched hyperdrive to support hyperdb + multiwriter storage. He published it as a package on npm, @jimpick/hyperdrive-next. I think the patched source code is here: https://github.com/jimpick/hyperdrive/tree/multiwriter-staging

I was curious about the potential to use dat + hyperdrive to create bidirectional filesystem synching between server & web app, so I made a little prototype: https://github.com/100ideas/hypercore-protocol-proxy-test

Its outdated by a year but it does show how to get duplex replication + synching set up between a dat archive on a server and a dat archive in the browser over a private websocket connection.

Also, while looking up some of these links, I came across @karissa's peerfs experiment, which was last updated may 2019 and is described as "multiwriter peer-to-peer filesystem, built on kappa-core and hyperdrive". It looks like it uses more recent (kappa) approaches to setting up multi-hyperdrive. https://github.com/karissa/peerfs.

Lastly these two tutorials show the way for setting up private key exchange for multiwriter situations and explain kappa architecture a bit better. (I haven't been paying attention to dat for the last year so I found them informative, apologies if they are well known already)

@aral

This comment has been minimized.

Copy link
Author

@aral aral commented Jul 8, 2019

Thanks @100ideas :) I’m still wondering if there’s a definitive timeline for multiwriter support in DAT. I don’t want to try and implement my own version only to find that it’s incompatible with DAT in the future. I know @mafintosh stated he was working on this in March. I’m waiting on this and working on other bits of my system in the meanwhile (mostly seamless deployment of the untrusted always-on web node).

@RangerMauve

This comment has been minimized.

Copy link
Contributor

@RangerMauve RangerMauve commented Jul 8, 2019

From what I understand, @mafintosh and hyperdivision got a contract to work on multiwriter by December. And it looks like their current path towards that is called "union mounts". Before that though, they'll be working on a system that enables you to share a secret key between devices with a third party service for coordinating writes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.