New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Anyone actively using Hyperdex? #233

Open
ddibiase opened this Issue Oct 29, 2016 · 19 comments

Comments

Projects
None yet
6 participants
@ddibiase

ddibiase commented Oct 29, 2016

Hi,

I was wondering if anyone is actively using Hyperdex? Love the project and everything about it, very sad that it's no longer maintained :(

Dave

@rescrv

This comment has been minimized.

Show comment
Hide comment
@rescrv

rescrv Oct 29, 2016

Owner

Hi Dave,

HyperDex is no longer maintained, but that's because circumstances have made me switch my attention to another project: Consus.

Consus is built using the same dependencies as HyperDex (which is why there is still some activity on them), and built using many lessons I learned working with people who were trying to use (and succeeding or failing to varying degrees) HyperDex.

Consus has two (three) differences from HyperDex that I believe will make it more appealing in the long run:

  1. Consus doesn't separate the implementations of transactions and the regular code. Once we have a paper public on the topic and I have clearance to release the code, all of the code will be available under the same license, with no commercial fork.
  2. Consus provides wide-area, geo-replicated transactions with much lower latency than is possible in any other system I know with the same consistency that Consus provides.
  3. I'm still actively working on Consus; I'm not actively working on HyperDex.

Of course, this doesn't help right now, but it does give you something to possibly follow in the future.

-Robert

Owner

rescrv commented Oct 29, 2016

Hi Dave,

HyperDex is no longer maintained, but that's because circumstances have made me switch my attention to another project: Consus.

Consus is built using the same dependencies as HyperDex (which is why there is still some activity on them), and built using many lessons I learned working with people who were trying to use (and succeeding or failing to varying degrees) HyperDex.

Consus has two (three) differences from HyperDex that I believe will make it more appealing in the long run:

  1. Consus doesn't separate the implementations of transactions and the regular code. Once we have a paper public on the topic and I have clearance to release the code, all of the code will be available under the same license, with no commercial fork.
  2. Consus provides wide-area, geo-replicated transactions with much lower latency than is possible in any other system I know with the same consistency that Consus provides.
  3. I'm still actively working on Consus; I'm not actively working on HyperDex.

Of course, this doesn't help right now, but it does give you something to possibly follow in the future.

-Robert

@unquietwiki

This comment has been minimized.

Show comment
Hide comment
@unquietwiki

unquietwiki Nov 22, 2016

@rescrv I came across this project via Wikipedia. Is http://www.garret.ru/consus.html the Consus you mention, or a different Consus?

unquietwiki commented Nov 22, 2016

@rescrv I came across this project via Wikipedia. Is http://www.garret.ru/consus.html the Consus you mention, or a different Consus?

@aXe1

This comment has been minimized.

Show comment
Hide comment
@aXe1

aXe1 commented Nov 22, 2016

@rescrv

This comment has been minimized.

Show comment
Hide comment
@rescrv

rescrv Nov 22, 2016

Owner

It is indeed http://consus.io

Owner

rescrv commented Nov 22, 2016

It is indeed http://consus.io

@unquietwiki

This comment has been minimized.

Show comment
Hide comment
@unquietwiki

unquietwiki Nov 22, 2016

@rescrv Thanks for that: I updated the wiki page with the new info. Question going forward: should folks use this project at all, or consider taking up maintenance of it? At the very least, a Debian/Jessie & Ubuntu/Xenial package for the server might be useful for existing projects. Also curious if this has any overlap with https://rethinkdb.com/ , which is another recently orphaned document store.

unquietwiki commented Nov 22, 2016

@rescrv Thanks for that: I updated the wiki page with the new info. Question going forward: should folks use this project at all, or consider taking up maintenance of it? At the very least, a Debian/Jessie & Ubuntu/Xenial package for the server might be useful for existing projects. Also curious if this has any overlap with https://rethinkdb.com/ , which is another recently orphaned document store.

@rescrv

This comment has been minimized.

Show comment
Hide comment
@rescrv

rescrv Nov 22, 2016

Owner

I've stopped developing the project for non-public reasons that aren't technically related to the project.

I'd be happy to hand off or publish the build scripts etc that I used to generate packages so that someone else can keep the project building, re-roll 1.8 with fixes I've applied since then, and package with the up-to-date dependencies that fix many bugs.

Owner

rescrv commented Nov 22, 2016

I've stopped developing the project for non-public reasons that aren't technically related to the project.

I'd be happy to hand off or publish the build scripts etc that I used to generate packages so that someone else can keep the project building, re-roll 1.8 with fixes I've applied since then, and package with the up-to-date dependencies that fix many bugs.

@unquietwiki

This comment has been minimized.

Show comment
Hide comment
@unquietwiki

unquietwiki Nov 22, 2016

@rescrv Publishing the build scripts + fixes would be helpful. I just found this project myself, looking up usable data stores for a project: I might be able to help out more if I can get it working on my end.

@ddibiase @aXe1 What about either of you? Or know anyone else helping in maintaining this? We could also try finding someone via a proper subreddit. https://www.reddit.com/r/programming/comments/a6qgz/proggit_im_trying_to_compile_all_the_known/ has a list of possible candidates.

unquietwiki commented Nov 22, 2016

@rescrv Publishing the build scripts + fixes would be helpful. I just found this project myself, looking up usable data stores for a project: I might be able to help out more if I can get it working on my end.

@ddibiase @aXe1 What about either of you? Or know anyone else helping in maintaining this? We could also try finding someone via a proper subreddit. https://www.reddit.com/r/programming/comments/a6qgz/proggit_im_trying_to_compile_all_the_known/ has a list of possible candidates.

@ddibiase

This comment has been minimized.

Show comment
Hide comment
@ddibiase

ddibiase Nov 22, 2016

@unquietwiki you have no idea how badly I'd like to help out, but I've got a stack on my plate right now. Also dbs aren't my forte but I'd be willing to learn/support other peoples efforts. Would love nothing more than to see a stable/maintained HyperDex codebase.

I'm wondering if maybe the author of LevelGraph might be interested. There's a LevelHyper adapter that I think he was involved in porting.

ddibiase commented Nov 22, 2016

@unquietwiki you have no idea how badly I'd like to help out, but I've got a stack on my plate right now. Also dbs aren't my forte but I'd be willing to learn/support other peoples efforts. Would love nothing more than to see a stable/maintained HyperDex codebase.

I'm wondering if maybe the author of LevelGraph might be interested. There's a LevelHyper adapter that I think he was involved in porting.

@unquietwiki

This comment has been minimized.

Show comment
Hide comment
@unquietwiki

unquietwiki Nov 22, 2016

@mcollina , @ddibiase and I were trying to help @rescrv figure out how to maintain this, since he's working on another related project now. I have some time and ability to help out, but I just discovered this project yesterday. Unless I understand incorrectly, it seems there is some convergence via https://github.com/rescrv/HyperLevelDB ?

unquietwiki commented Nov 22, 2016

@mcollina , @ddibiase and I were trying to help @rescrv figure out how to maintain this, since he's working on another related project now. I have some time and ability to help out, but I just discovered this project yesterday. Unless I understand incorrectly, it seems there is some convergence via https://github.com/rescrv/HyperLevelDB ?

@rescrv

This comment has been minimized.

Show comment
Hide comment
@rescrv

rescrv Nov 24, 2016

Owner

I'm actively maintaining the dependencies below HyperDex, so those dependencies will be easy to keep going. I've started pulling my build scripts for HyperDex from my backups. It may take past the holiday before I can make progress on this issue.

Owner

rescrv commented Nov 24, 2016

I'm actively maintaining the dependencies below HyperDex, so those dependencies will be easy to keep going. I've started pulling my build scripts for HyperDex from my backups. It may take past the holiday before I can make progress on this issue.

@rescrv

This comment has been minimized.

Show comment
Hide comment
@rescrv

rescrv Dec 1, 2016

Owner

I've pushed the HyperDex build process to GitHub. You'll need to clone the rescrv/minion repo and have Docker and go installed. In the minion repo, run "make" to build the "minion" and "minion-daemon" binaries. Add them to your path.

Next, setup the releng environment. I've uploaded a pre-bootstrapped build environment to S3 at s3://rescrv/hyperdex-releng.tar.gz. It is available to authenticated AWS users. Download that complete environment and untar it. The hyperdex-releng/build directory is a clone of the rescrv/hyperdex-releng repo.

Deploy minion by running minion-daemon in the hyperdex-releng directory. Pass this directory as the -d argument to any minion command. Run the commands in README in that repo to bootstrap your environment and build the tarballs and packages. I made sure it builds all the way through the tarballs and packages. I've disabled some broken pieces I didn't have time to fix.

You can copy the results of the build process with minion -d /path/to/hyperdex-releng get-artifacts dev latest output

Sorry for throwing everything at you quite quickly. I'm happy to answer questions and provide clarification. I just wanted to get this out to the interested folks as soon as possible.

Owner

rescrv commented Dec 1, 2016

I've pushed the HyperDex build process to GitHub. You'll need to clone the rescrv/minion repo and have Docker and go installed. In the minion repo, run "make" to build the "minion" and "minion-daemon" binaries. Add them to your path.

Next, setup the releng environment. I've uploaded a pre-bootstrapped build environment to S3 at s3://rescrv/hyperdex-releng.tar.gz. It is available to authenticated AWS users. Download that complete environment and untar it. The hyperdex-releng/build directory is a clone of the rescrv/hyperdex-releng repo.

Deploy minion by running minion-daemon in the hyperdex-releng directory. Pass this directory as the -d argument to any minion command. Run the commands in README in that repo to bootstrap your environment and build the tarballs and packages. I made sure it builds all the way through the tarballs and packages. I've disabled some broken pieces I didn't have time to fix.

You can copy the results of the build process with minion -d /path/to/hyperdex-releng get-artifacts dev latest output

Sorry for throwing everything at you quite quickly. I'm happy to answer questions and provide clarification. I just wanted to get this out to the interested folks as soon as possible.

@unquietwiki

This comment has been minimized.

Show comment
Hide comment
@unquietwiki

unquietwiki Dec 1, 2016

@rescrv Thank you so much for doing this. I plan on trying to make use of this soon, as time allows.

unquietwiki commented Dec 1, 2016

@rescrv Thank you so much for doing this. I plan on trying to make use of this soon, as time allows.

@rescrv

This comment has been minimized.

Show comment
Hide comment
@rescrv

rescrv Dec 15, 2016

Owner

An update to this issue. I have made Consus (http://hack.systems/2016/12/13/hello-consus/) open source and would strongly encourage anyone considering helping out with HyperDex to get involved in Consus instead. I can provide feedback and guidance for both, but am only going to be actively developing Consus going forward.

Owner

rescrv commented Dec 15, 2016

An update to this issue. I have made Consus (http://hack.systems/2016/12/13/hello-consus/) open source and would strongly encourage anyone considering helping out with HyperDex to get involved in Consus instead. I can provide feedback and guidance for both, but am only going to be actively developing Consus going forward.

@webmaven

This comment has been minimized.

Show comment
Hide comment
@webmaven

webmaven commented Dec 19, 2016

@rescrv, is this the public paper?: https://arxiv.org/abs/1612.03457

@rescrv

This comment has been minimized.

Show comment
Hide comment
@rescrv

rescrv Dec 19, 2016

Owner

Yes. It and the code are linked from the blog post I pointed to.

Owner

rescrv commented Dec 19, 2016

Yes. It and the code are linked from the blog post I pointed to.

@unquietwiki

This comment has been minimized.

Show comment
Hide comment
@unquietwiki

unquietwiki Dec 19, 2016

@rescrv Thank you for this. You should consider hitting up /r/programming on Reddit with a post about this too.

unquietwiki commented Dec 19, 2016

@rescrv Thank you for this. You should consider hitting up /r/programming on Reddit with a post about this too.

@rescrv

This comment has been minimized.

Show comment
Hide comment
@rescrv

rescrv Dec 19, 2016

Owner

You'll notice there is a doc/ directory. Part of rescrv/consus-releng builds this documentation. I need to write the HTML templates to generate the documentation (it currently generates jinja2 templates) and then post it.

After that I'll do wider promotion on social media. I had reasons to open source the code when I did, but I'm not ready to go on a promotional circuit of social media yet (it hit hacker news already).

Owner

rescrv commented Dec 19, 2016

You'll notice there is a doc/ directory. Part of rescrv/consus-releng builds this documentation. I need to write the HTML templates to generate the documentation (it currently generates jinja2 templates) and then post it.

After that I'll do wider promotion on social media. I had reasons to open source the code when I did, but I'm not ready to go on a promotional circuit of social media yet (it hit hacker news already).

@higaxman

This comment has been minimized.

Show comment
Hide comment
@higaxman

higaxman Jan 8, 2017

@rescrv Here at Forcive (www.forcive.com) we somewhat got interested at HyperDex and we are looking to it as a possible component part for our real time analytic platform. Understanding Consus is the continuator of HpyerDex we are even interesting to contribute to this project.

higaxman commented Jan 8, 2017

@rescrv Here at Forcive (www.forcive.com) we somewhat got interested at HyperDex and we are looking to it as a possible component part for our real time analytic platform. Understanding Consus is the continuator of HpyerDex we are even interesting to contribute to this project.

@rescrv

This comment has been minimized.

Show comment
Hide comment
@rescrv

rescrv Jan 8, 2017

Owner

@higaxman It's great to hear of your interest. If you're interested in HyperDex, I'd be happy to provide a little assistance getting you off the ground with the build scripts. If you're interested in Consus, I'd point you to its repo.

If you (or others reading this post) are deciding between the two, I'd suggest reading this: Why Consus? What happened to HyperDex?

Owner

rescrv commented Jan 8, 2017

@higaxman It's great to hear of your interest. If you're interested in HyperDex, I'd be happy to provide a little assistance getting you off the ground with the build scripts. If you're interested in Consus, I'd point you to its repo.

If you (or others reading this post) are deciding between the two, I'd suggest reading this: Why Consus? What happened to HyperDex?

Repository owner locked and limited conversation to collaborators Jan 12, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.