New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Product Roadmap #1

Open
manishrjain opened this Issue Nov 30, 2015 · 69 comments

Comments

Projects
None yet
@manishrjain
Member

manishrjain commented Nov 30, 2015

  • Low Latency
  • High Throughput
  • RDF Parsing
  • Rocks DB Badger DB for persistence
  • Commit Logs (Replaced by RAFT logs)
  • Query Language - GraphQL-like
    • Query
    • Root Arguments
    • Fields
    • Response in JSON
    • Field Arguments
    • Field Alias
    • Mutations #23 [v0.2]
    • Fragments #8
    • Variables
    • Type System
      • Scalar Types
      • Object Types
    • Mutation validation for scalar types
    • String matching / Name search
    • Sort by attribute
    • Limit number of results #9
    • Filter
      • anyof
      • allof
      • eq (equal)
      • inequality (>=, <=, >, <)
    • Aggregate Functions
      • count
      • sum
      • max
      • min
    • Geospatial Queries
      • Nearby
      • Within
      • Contains
      • Intersects
  • Official Clients
    • Javascript
    • Go [v0.3]
    • Java
  • Distributed Transactions
  • Distributed #14 [v0.2]
    • Distributed Loader [v0.2]
    • Distributed Server [v0.2]
  • Clustering
    • Node discovery and membership via Dgraph Zero
  • High Availability
    • Raft
    • Automatic Data Replication
    • Automatic Failover for reads
    • Read linearizability
  • Resilience
    • Shard moves to handle server failure
  • Export
  • Backup

After v1.0 / Proprietary Plugins

  • Multi-homing support
  • Cypher Support
  • Access Control Lists
  • Query Graphical User Interface
  • User authentication
  • Cluster Management
  • SPARQL [maybe]
  • Tinkerpop Support [maybe]
  • Distributed transactions [maybe]
@KyleAMathews

This comment has been minimized.

KyleAMathews commented Dec 3, 2015

It'd be cool if the DB could serve GraphiQL. This would be perfect for demos + internal spelunking through data.

@manishrjain

This comment has been minimized.

Member

manishrjain commented Dec 3, 2015

I'm running a DGraph server holding Freebase Film data, which you can query like so: curl dgraph.xyz/query -XPOST -d '{}'. An example of the query is in the README.

@manishrjain manishrjain modified the milestone: v.future Feb 17, 2016

@PhilAndrew

This comment has been minimized.

PhilAndrew commented Apr 6, 2016

Support tinkerpop? https://tinkerpop.apache.org

@manishrjain

This comment has been minimized.

Member

manishrjain commented Apr 6, 2016

Tinkerpop would happen with Gremlin support. We'll do that after v1.0 gets released. This roadmap is largely focused on v1.0.

@manishrjain manishrjain removed this from the v.future milestone Jun 3, 2016

@F21

This comment has been minimized.

Contributor

F21 commented Oct 19, 2016

For gossip discovery, I've found hashicorp/memberlist to work quite well.

@manishrjain

This comment has been minimized.

Member

manishrjain commented Nov 7, 2016

@F21: Instead of using Gossip protocol, we used the already implemented RAFT protocol for keeping track of membership. In terms of discovery, our cluster requires knowing at least one node in the cluster; any healthy node would do.

@jcebXD

This comment has been minimized.

jcebXD commented Dec 4, 2016

I suggest also support cypher for queries, to avoid the queries migration from neo4j users.

@manishrjain

This comment has been minimized.

Member

manishrjain commented Dec 5, 2016

@jcebXD -- Yeah, Cypher is something we'll fully support.

@pmualaba

This comment has been minimized.

pmualaba commented Dec 12, 2016

Cypher support will be great indeed for the application layer. I also see that Tinkerpop support is on the Roadmap after 1.0, but I would have prefered to see SPARQL instead. Since the data model for Dgraph is already Triple/Nquad based, it would be logic and convenient to expose a SPARQL 1.1 endpoint out of the box, in addition to the GraphQL endpoint. SPARQL support is very important to fully interact with the semantic web, without the need to take a full Dgraph (triples) backup which then again needs to be imported in a separate TripleStore such as BlazeGraph to be able to run SPARQL queries against the Dgraph data.

@gedw99

This comment has been minimized.

gedw99 commented Dec 30, 2016

Is there any interest in supporting mobile ?
Running a dgrpah DB on mobiles using gomobile is possible. Even with the CGO aspect of rocksdb.
I currently run boltDB on the servers and the clients with golang. It is nice to have that reuse.

For example i build apps for mobile, desktop using golang with this: https://github.com/therecipe/qt
It uses CGO heavily, and works well.

@manishrjain

This comment has been minimized.

Member

manishrjain commented Jan 2, 2017

@gedw99 -- We're not currently thinking about mobile. All our efforts are focused on production for large scale deployments.

@pmualaba

This comment has been minimized.

pmualaba commented Jan 30, 2017

Aggregations are not on the roadmap for 1.0, and neither after 1.0 ?
The ability to perform aggregations (COUNT, SUM, AVG, MIN, MAX, GROUP BY, DISTINCT,...) is clearly one of the main reasons you store data in a database in the first place and all acient and modern databases and database languages (SQL, CYPHER, SPARQL,...) support it. Why are they missing from the roadmap for DGraph?

Also for a new product anno 2017 it also make sense to have built in support for subscriptions (changefeeds ala RethinkDB). since many of the applications we built today needs support for real-time and push features...

DGraph is a great and promising database project, but without these features one would still be forced to move to other open-source alternatives such as ArangoDB which at this moment is the only serious open-source "competitor" for DGraph.

@ashwin95r

This comment has been minimized.

Contributor

ashwin95r commented Jan 30, 2017

@pmualaba: We're working on aggregate functions already! #532 It was missing on the roadmap. Thanks for pointing it out. We've added it now.

@pmualaba

This comment has been minimized.

pmualaba commented Jan 30, 2017

You guys are lightning fast ;-) Great to hear that you're already working on this.

@pmualaba

This comment has been minimized.

pmualaba commented Feb 20, 2017

While playing with DGraph, there is one thing that doesn't feel quite right to me :
DGraph returns JSON format (which is great!), GraphQL queries also uses JSON format (without values) which is also great! But GraphQL mutations only supports NQuad format?
I would like to see mutations also to be consistent in using JSON. As a matter of fact, there is already a W3C compliant RDF serialization format called JSON-LD: http://json-ld.org/

Accepting valid JSON-LD in mutations will also greatly improve developer experience using DGraph, since application logic these days all generate or serialize into JSON before persisting to a database. Manual JSON to NQuad conversion can then be eliminated from the application layer and moved to the database layer.

I would not suggest to remove support for NQuad mutations enterily (sometimes it can come in handy), But to add additional support for JSON-LD. I believe NQuad is still the way to go for massive batch imports, but JSON-LD is preferable for use in mutation queries, i guess.

@manishrjain

This comment has been minimized.

Member

manishrjain commented Feb 21, 2017

Thanks for the feedback, @pmualaba. JSON-LD seems like a nice format for mutations. We have @ashishnegi working on it.

@joeblew99

This comment has been minimized.

joeblew99 commented Mar 9, 2017

@manishrjain Are the proprietry plugins going to be closed source.
Sorry to aks this, but ACL's for example not being part of the open source offering ? Its pretty harsh.

@manishrjain

This comment has been minimized.

Member

manishrjain commented Mar 14, 2017

Hey @joeblew99 , others,

It is very important for us to figure out a revenue model, if we want to continue working on Dgraph three years down the lane. That's the only way we can keep supporting the salaries of people who are dedicated their lives on Dgraph. Without money, Dgraph would stop spinning (think RethinkDB).

Keeping that in mind, we have divided the project into what we consider open core and what we consider commercial. Anything that a startup needs to run their services and scale is part of the open core. This includes all of the current functionality and running Dgraph distributedly (which is not for e.g., allowed by Neo4j).

Anything that's an advanced feature would then be put into the commercial, closed-source version of Dgraph. This means: ACLs, cluster management, multi-tenancy, auto-scaling on Amazon, Google, etc.

This is a delicate balance between giving startups the same (I know it's better but I want to stay humble) level of technology built within the walls of big companies, like Google, Facebook, etc.; while also finding a way to monetization to make the company profitable and keep hiring the top engineers we can find.

Note that for small companies, which are pre-revenue, we would most likely make the enterprise version free of cost (free, but not open). But, if a company is making revenue and benefitting from Dgraph, we do expect them to pay. That's what is going to allow us to keep our engineers working for Dgraph, in a very competitive market where giants can pay up a huge sum to entice engineers. So, their salaries have to be competitive and they must feel that this company is not going to vaporize, and is going to be around 10 years from now.

@foobatman

This comment has been minimized.

foobatman commented Mar 14, 2017

@manishrjain

This comment has been minimized.

Member

manishrjain commented Mar 14, 2017

One thing that's clear to me: Dgraph can only survive if we are 10x better than Neo4j and others. If we genuinely didn't believe that, we'd be instead working for some big company, 9-5 and getting a nice fat check at the end of the year. So, making Dgraph the best graph database out there is the top priority. And for that, we need to hire and retain the best engineers.

I think Firefox is a bad example. They were thriving when Google was paying (donating?) them a big sum of money; until Google decided to stop that and focus on Chrome. Since then, Firefox revenue has steadily declined, while Chrome has taken over the world. At this point, Firefox can't afford to get the top talent working for them -- while Google (/Chrome) continues to recruit the best and brightest, which in turn makes Chrome even better.
http://howdoesitmakemoney.com/how-does-mozilla-firefox-make-money/

Consultancy is great, and we intend to do that, but consultancy alone is not going to make a company significantly profitable (more consultancy requires more human power). All top companies today are profitable due to selling software (Google, Microsoft, Facebook, etc.).

P.S. I don't intend this to become a thread of its own; so feel free to email me if anyone has concerns: manish, dgraph.io.

@joeblew99

This comment has been minimized.

joeblew99 commented Mar 14, 2017

@manishrjain i agree with you.
Many startups that were making new database systems failed due to lack of revenue. We dont want another rethinkdb, and the open core model makes sense.

But, everyone needs ACLS. I would think about if ACLS are in the open core of not.
The other stuff is for commercial for sure - so i agree with that.

But ACLS is so fundamental to any system you run always.
I mean any open source project that wants to build a forum or a chat system needs ACLS.

@flagello

This comment has been minimized.

flagello commented Apr 8, 2017

What are your thoughts on subscriptions? Do you plan on supporting libraries for languages other than Go (and Python and Java)?

@manishrjain

This comment has been minimized.

Member

manishrjain commented Apr 8, 2017

We have community run libraries for Python and Java. You can find their repos in dgraph-io org in Github.

Not sure what you mean about subscriptions.

@manishrjain

This comment has been minimized.

Member

manishrjain commented Oct 22, 2017

@Didericis @MichelDiz -- By JS client, I meant a Dgraph specific JS client which would talk to Dgraph over GRPC. We're adding support for transactions, and hence, all the mutable queries now need to happen via a client.

Apollo / GraphQL compatibility is something we intend to look into past v1.0, once our query language is stable, and we are clear that it fulfills most functionality required by our users. At that point, we can figure out a way to bridge the gap between GraphQL and our version of it.

@joeblew99 : There is an issue about something similar: #824

@sebastianmacias

This comment has been minimized.

sebastianmacias commented Oct 27, 2017

@manishrjain

Anything that's an advanced feature would then be put into the commercial, closed-source version of Dgraph. This means: ACLs, cluster management, multi-tenancy, auto-scaling on Amazon, Google, etc.

I think ACL is very important not to be considered part of the open source core.

graphcool offers ACL out of the box along with authentication/authorization, subscriptions and other features.

They offer a hosted service Graphcool Cloud and that is how the monetize; maybe a hosted service would allow you to monetize without sacrificing essential features all devs and small startups need.

Just a thought.

@manishrjain

This comment has been minimized.

Member

manishrjain commented Oct 30, 2017

@sebastianmacias : Noted. Once we're past v1.0, we'll reconsider this.

@sebastianmacias

This comment has been minimized.

sebastianmacias commented Oct 30, 2017

@manishrjain sounds good. Thanks

@manishrjain

This comment has been minimized.

Member

manishrjain commented Nov 15, 2017

@pmualaba : Dgraph v0.9 adds distributed transactions, and fully supports JSON as both input and output format. It makes interaction with Dgraph a lot easier.

@joeblew99

This comment has been minimized.

joeblew99 commented Nov 27, 2017

Can you guys write a demo example for distributed transactions.
Can be simulated with dockers.
Really curious how the linearizability performs. Lots of gremlins

@MichelDiz

This comment has been minimized.

Contributor

MichelDiz commented Nov 28, 2017

They will produce some examples soon. Follow the news in both the blog and the forum.

@gedw99

This comment has been minimized.

gedw99 commented Nov 28, 2017

@Senmumu

This comment has been minimized.

Senmumu commented Dec 11, 2017

will you guys consider the implementation of ALTER in web UI ?

@pawanrawal

This comment has been minimized.

Contributor

pawanrawal commented Dec 11, 2017

Hey @Senmumu

Yes, Alter and Mutate operations would be possible from the Dgraph Browser from the next release. It has already been implemented and merged to master.

@jimanvlad

This comment has been minimized.

jimanvlad commented Dec 13, 2017

Has the timeline for Cypher support changed?

@manishrjain

This comment has been minimized.

Member

manishrjain commented Dec 13, 2017

@jimanvlad : We'll start working on Cypher right after v1.0.

@Senmumu : In addition to what @pawanrawal said, just wanted to mention that we're investing heavily in the web UI. It would be broken out of the Dgraph binary, and be a standalone server, capable of doing a lot more things than what it does today. More on that in the next few months.

@darshanDevrai

This comment has been minimized.

darshanDevrai commented Mar 2, 2018

Any updates on Cypher support and Hosted service?

@manishrjain

This comment has been minimized.

Member

manishrjain commented Mar 3, 2018

We're working towards the hosted service quickly. In fact, starting to onboard a few early clients -- so feel free to reach out, if someone wants to go into private beta.

Cypher and other languages support are still slated for this year -- but we're a bit bogged down by all the activity around Dgraph since v1.0 release; doubling the team size to surface back and continue shipping features.

@smkhalsa

This comment has been minimized.

smkhalsa commented Apr 2, 2018

@manishrjain any update on timeline for hosted service? I'd be interested in private beta.

@elliottstarin

This comment has been minimized.

elliottstarin commented Apr 4, 2018

Are there any plans for supporting streaming/continuous query type operations?

@gedw99

This comment has been minimized.

gedw99 commented Apr 13, 2018

At the list at the top you have backup, but no restore. Does the community version support restore ? I need to test all DR aspects before recommendation to our team

@lazharichir

This comment has been minimized.

lazharichir commented May 3, 2018

How is the initial pricing thought to be for the hosted service in comparison to Amazon Neptune?

@manishrjain

This comment has been minimized.

Member

manishrjain commented May 3, 2018

We're not doing hosted service for now. Backup and restore would be one feature. There's no point of backup, without any way to restore :-). But, there's already a way to export the data in NQuad format and import it back.

@gedw99

This comment has been minimized.

gedw99 commented May 3, 2018

@MichelDiz

This comment has been minimized.

Contributor

MichelDiz commented May 8, 2018

@gedw99 The backup feature still not exist. So, for that there's no restore. But you have "Export" and ou can re-import this exported file into a new instance. See more details here: https://docs.dgraph.io/deploy#export-database

take a look on:
https://docs.dgraph.io/deploy#bulk-loader
https://docs.dgraph.io/deploy#live-loader

@gedw99

This comment has been minimized.

gedw99 commented May 8, 2018

@manishrjain

This comment has been minimized.

Member

manishrjain commented May 8, 2018

All Dgraph code base is open. We currently don't have any enterprise features.

@lynic

This comment has been minimized.

lynic commented Jul 12, 2018

I really like this project, but there're two missing features block me from using it at our development environment, one is user authentication, the other is Tinkerpop/gremlin support. If you could prioritize these two features, that would be very helpful!

@manishrjain

This comment has been minimized.

Member

manishrjain commented Jul 12, 2018

Gremlin is on the roadmap. We have ideas around user auth, but for now, you could use the password type to store user passwords and use that to authenticate them.

@zzgt

This comment has been minimized.

zzgt commented Jul 13, 2018

I'm learning Dgraph. A smarter "Query Graphical User Interface" would be very helpful when exploring the graph database, like Grakn.ai has. And moreover, if there is a "How to do reasoning" section in Help Doc, that would be wonderful. One more question: does dgraph support hypergraphs? Thanks!

@pierreis

This comment has been minimized.

pierreis commented Oct 12, 2018

Two questions:

  1. Multi-homing support: Does this relate to datacenter / rack awareness for geo-distributed setups?
  2. Any update on a hosted Dgraph service? Q1 2018 still is the period I'm hearing about.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment