New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hyperlog-style semantics with hypercore/hyperdb? #58

Open
saranrapjs opened this Issue Mar 3, 2018 · 7 comments

Comments

Projects
None yet
4 participants
@saranrapjs
Copy link

saranrapjs commented Mar 3, 2018

Not sure if this is the appropriate forum for this type of question, but: is there a straightforward way of achieving append-only ("hyperlog"-style) multi-writer semantics with hyperdb?

I like the sparse replicating aspects of hypercore/hyperdb, and hyperdb seems to be the only hypercore implementation which supports multiple writers, but I'm basically just interested in appending to a log (e.g. I don't need the overhead of the key/value store hyperdb provides). I noted multifeed, which sort of seems to hint something in this direction?

At any rate, happy to be told that hyperdb is not the tool for this particular architecture, but would be curious to hear if there are plans/designs for something like what I'm describing! And if there's a better place to ask these kinds of questions :)

@noffle

This comment has been minimized.

Copy link
Collaborator

noffle commented Mar 3, 2018

I'd really like this too. Having a multifeed module would be a good base for making something that can implement hyperlog's API. Steps (in my mind):

  1. Finish multifeed (multicore?) (this could be just forking hyperdb and stripping the kv parts out)
  2. Write an implementation of the hyperlog API on top of that (corelog? hyperfeed?)
@pfrazee

This comment has been minimized.

Copy link
Collaborator

pfrazee commented Mar 3, 2018

Would it be sufficient to use hyperdb with a keyspace of monotonic integers? That is, you could just write to /1, /2, /3, etc.

@saranrapjs

This comment has been minimized.

Copy link
Author

saranrapjs commented Mar 4, 2018

(this could be just forking hyperdb and stripping the kv parts out)

I hadn't thought about this! Might give this a try, time permitting.

Would it be sufficient to use hyperdb with a keyspace of monotonic integers?

I thought about this too...I haven't closely inspected how the various hypercores in hyperdb get reconciled, but I'm assuming that the topographic ordering referred to in the README could lead to situations where a peer would inadvertently overwrite another peer's key.

Though, something that's actually just now occurring to me: it might be possible to use db.createHistoryStream as the append-only log, and have all peers just consistently write to the same key? You'd essentially use the topographic ordering to get at monotonicity...possibly at the cost of all of the overhead used to materialize the key/value parts of hyperdb?

@pfrazee

This comment has been minimized.

Copy link
Collaborator

pfrazee commented Mar 4, 2018

I'm assuming that the topographic ordering referred to in the README could lead to situations where a peer would inadvertently overwrite another peer's key.

Yeah that would happen. I guess the solution depends on how you want your log to behave. Internally hyperdb is a set of logs which get woven together. If you use the history stream by itself, I'd expect that you're going to get different orders for each device, but certainly no conflicts.

@noffle

This comment has been minimized.

Copy link
Collaborator

noffle commented Mar 4, 2018

You could also go under the covers and access db._writers directly, which is an array of Writers. Each Writer has a .feed field that's that writer's hypercore instance. Then you could just ignore the regular hyperdb kv API and manage the hypercore feeds directly. Though a proper multifeed module is a much more reusable way to go I think.

@mafintosh

This comment has been minimized.

Copy link
Owner

mafintosh commented Mar 13, 2018

noting that i have multifeed on npm in case anyone wants to take a stab at that. originally i wanted to build something on top of a multifeed module but in practice that is really tricky as you really need the trie internally to make this thing scale if you want random access replication

@mafintosh

This comment has been minimized.

Copy link
Owner

mafintosh commented Mar 13, 2018

what @pfrazee is suggesting is actually a really good solution btw as it gives you an implicit notion of time. i.e. if a fork happened a loooooooong time ago it's "seq" will be /{small-number} vs the latest changes being /{large-number}.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment