Real-time / streaming / live query / RethinkDB support

[gadicc]

Hey, I know this isn't on the near term radar, but I thought it would be helpful to have a single issue that we could always refer back to on this. Feel free to edit this first post as useful. I guess the main things are:

• Ability to maintain an open websocket
  • Would be cool to (optionally) only open the websocket if active live queries are running, and disconnect after a configurable amount of time after all live queries have stopped.
• Server API to invalidate queries / push new results
• Client/server protocol for live query style add/update/change

Rough ETA: 2017 (expectation management :))

Related issues:

• Support for deferred/partial responses #18 - Support for deferred/partial responses
• [meta] Support real-time updates via event subscriptions facebook/relay#541 - Support real-time updates via event subscriptions

[gadicc]

I'd also like to relate this back to apollographql/apollo-client#257 (using apollo with rethinkdb horizon).

I'm still new to all of this, but can't we look at this another way? Instead of horizon implementing a graphQL server that can be used with the apollo client (and then, needing two distinct apolloClient instances), doesn't it make more sense to make this work with apollo server? And potentially having RethinkDB officially recommend Apollo?

I was thinking something like:

export const resolvers = {
  users(parent, args, context, liveQuery) {
    const query = r.table('users');
    rethinkChangeFeedAdaptor(conn, query, liveQuery);
  }
}

function rethinkChangeFeedAdaptor(conn, query, liveQuery) {
  // Tell the server not to use/expect an immediate return result
  // and specify the primary key.
  liveQuery.init('id');

  query.changes({ includeStates: true }).run(conn, (err, cursor) => {
    cursor.each(row => {
      if (['add','remove','change'].includes(row.type))
        liveQuery[row.type](row.new_val);
    });
  }
}

Just some musings :) What are the advantages of rethinkDB implementing their own GraphQL server?

[gadicc]

Facebook have also been doing some work on this, see GraphQL Future at react-europe 2016 on Youtube. They have experiments for live queries, and mentioned that there are plans to make their server infrastructure reactive in the future.

(For now, however, they've done more work on event-based subscriptions (facebook/relay#541), powering FB comments & likes, and the live reactions on streaming videos, amongst others.)

The live query experiment (which hasn't seen production use yet) looks like this (from 17:19):

Query:

{
  feed {
    stories {
      author { name }
      message
      likeCount @live
    }
  }
}

Initial payload (like normal):

{
  "feed": {
    "stories": [
      {
        "author": { "name": "Lee Byron" },
        "message": "GraphQL est grand!",
        "likeCount": 5
      },
      {
        "author": { "name": "Dan Schafer" },
        "message": "Anyone for lunch?",
        "likeCount": 5
      },
      {
        "author": { "name": "Christopher Chedeau" },
        "message": "React Native est l'avenir",
        "likeCount": 50
      }
    ]
  }
}

Sample update delta:

{
  "path": [ "feed", "stories", 0, "likeCount" ],
  "data": 6
}

[stubailo]

Thanks for opening this issue! I think it would be very exciting to experiment with the proposed features from react Europe, and were in a unique position to do it since we are the only team who have both a client and server implementation.

I completely agree that putting a GraphQL endpoint in a database is not the right approach, since not all of your data will be in that one source. So once we figure out how data streaming from resolvers should work, we should see if that can be easily made to work with rethinkdb live queries (as well as meteor MongoDB live queries)

[helfer]

@gadicc We're rewriting the server right now, and one of the things we want to work on after the refactor is websocket connections, which should enable some of the things you propose.

[gadicc]

Thanks, guys! This sounds awesome and I'm really excited for it.

[peteflanagan]

Until the server is rewritten, is there a recommended method for creating a connector to RethinkDB such as thinky, rethinkdbdash or horizon?

[helfer]

@peteflanagan Without support for pushing data to the client, the only way to get some level of reactivity is to do polling. @amandajliu is working on GraphQL subscriptions, but those will only get you part of the way there.

Until support for the proposed live queries is implemented in GraphQL, I think it will be difficult to get the full benefit of RethinkDB in GraphQL.

[willmkt]

You may want to take a look at this project, but it does not usr apolloserver

https://github.com/bhoriuchi/vue-graphql-rethink

[frankdugan3]

Now that we have graphql-subscriptions, subscriptions-transport-ws and https://github.com/davidyaha/graphql-redis-subscriptions... I'm wondering if it would be possible to come up with a similar integration for RethinkDB?

Not quite the same as live queries, but it could be a significant improvement while waiting for the live query spec to mature.

[rnenjoy]

I'm not qualified enough for making this myself. But wouldn't it be possible to just use a rethinkdb changefeed query inside a resolvers function, and keep that function returning results to the clients when changes occur? There is no really need for a pub/sub system?

[DxCx]

hi @gadicc
I'm also interested in full websocket transport, and already have a working PoC implementation for that.
however i don't have the bandwidth to work on that further at the moment.
let's discuss because i think that we can meet and sync up, i don't see any reason to work in parallel.

[gadicc]

Hey, @DxCx. Since the subscription support was added I never actually had a chance to look into this again. Realistically and unfortunately, I don't think I'll be able to dedicate any time to this until March at the earliest. Maybe if you can put your PoC online somewhere it might serve as a good basis for discussion and maybe others will be able to continue the work.

[thomasmodeneis]

Hi,
I'm very new to this topic, but it crossed my mind that whatever streaming/realtime/live query you may want to do will stumb upon rethinkdb/rethinkdb#3471 and rethinkdb/rethinkdb#4133
How are you guys working around this issues, if you are at all ?
Cheers.

[paralin]

I would say this shouldn't be tied to RethinkDB at all, we just need an interface to return a stream of entries, and for Apollo client to understand this.

Furthermore Apollo supports pluggable network interfaces. The approach I would use to implement this is to add another interface (like SubscriptionNetworkInterface) that supports this, and then worry about the actual transport later.

[ivosabev]

Its 2017 any status updates?

[paralin]

http://github.com/rgraphql

[helfer]

@ivosabev we're working on websocket integration in #272, which is the first step in this direction.

[rnenjoy]

Sorry for the dumb question. But why do we need a websocket integration? Why cant the transport method that already exists as a standalone package be used?

[helfer]

@rnenjoy we will probably only have one transport in the end that supports subscriptions and queries, but we're doing the work here to figure out what that should look like without complicating the subscriptions transport implementation.

[helfer]

Closing this for now, but we can keep the discussion going.