add support for defer and stream directives

[lilianammmatos]

The implementation in this pull request is a reference implementation of the specification proposal outlined in graphql/graphql-spec#742. It is no longer compatible with with the @defer and @stream implementation in Relay.

---

At 1stdibs, our GraphQL server is built on top of Apollo Server. We were first introduced to the concept of the @defer directive through an Apollo blog post detailing an experimental implementation. We attempted to resurrect this initiative by creating an up-to-date pull request (in hopes of that getting merged); however, it became clear to us over time that support for a @defer directive would occur only if it would make its way into the official GraphQL spec. There has been discussion on this issue to add @defer to the spec, but it seems it's still in its infancy!

Since then, Relay has added support for @defer and @stream, but there are no open source GraphQL servers that support its implementation.

This pull request is a proposal for a Relay compatible @defer and @stream directive. The main distinction between the Relay implementation of @defer and that of Apollo's is that the @defer directive is supported on a fragment spread and inline fragment instead of on an individual field.

Furthermore, this pull request is implemented according to our understanding of Relay’s support for @defer and @stream. We outlined a few examples here: https://gist.github.com/robrichard/f563fd272f65bdbe8742764f1a149b2b

This is a feature we are very passionate about at 1stdibs and would be interested in working with contributors in any way possible!

Sketch RFC: https://github.com/graphql/graphql-spec/blob/master/rfcs/DeferStream.md

[lilianammmatos]

The result of execute (and also the main graphql function) has been modified to return a patches field, which is an AsyncIterable. We chose not to change the result of execute to something like AsyncIterator<AsyncExecutionResult> | Promise<ExecutionResult> as that would cause type errors for existing consumers of these functions, even if they do not enable these features in their schema.

[lilianammmatos]

If anyone wants to try this out locally, I've published these changes to npm under the package graphql-experimental: https://www.npmjs.com/package/graphql-experimental.

[lilianammmatos]

If you already have graphql installed in a project, but would like to use graphql-experimental without updating your import/require statements, you can use:

yarn add graphql@npm:graphql-experimental

to alias graphql-experimental under graphql.

Reference

[Keweiqu]

At Facebook, we actually invalidate a query that looks like this. In practice, this is more often a mistake developers make rather than an intentional decision.

To allow developers to create multiple streams on the same field, or to both stream and not stream, we take advantage of the "alias" feature in GraphQL and do something like this:

friendsWithName: friends @stream(label: "namedLabel", initial_count: 1)

[robrichard]

@Keweiqu I updated the PR to add a validation rule that forbids streaming on the same field unless the label and initial_count are the same. Does this more closely match the implementation at Facebook?

c29e1e3#diff-a49ecb51ef73494dfb5db4effdf536a6R601

[Keweiqu]

@robrichard Thanks for adding the validation rule. I do think this validation is universally beneficial no matter which infrastructure developers are working with.

[Keweiqu]

@lilianammmatos thank you so much for driving this effort of adding reference implementations of @defer and @stream for graphql-js! I worked on @defer and @stream Facebook internal implementations, offering some suggestions here based on our experience with this feature.

[Keweiqu]

Relay actually relies on the AsyncIterator to inform whether the current payload is the last payload or not to change its state to "loading_incremental" to "loading_final". If the payload is the last one, the payload will include is_final: true in the extensions field.

Because we only insert "is_final" when the payload is the last one globally, there are several edge cases we should consider and test out to make sure that the dispatcher returns correctly. If a query contains multiple @defers and @streams, we don't add "is_final" per stream/defer label. If a query contains a nested @defer inside a @stream

friends @stream(label: "friendList", initial_count: 1) {
  nodes {
     ... on Friend @defer(label: "friendNode") {
       name
     }
  }
}

We insert "is_final" for the correct last payload.

[robrichard]

@Keweiqu thanks for the feedback! This is very helpful information. Do you think is_final in the extensions field is something that needs to be part of the spec for this feature, or is that more of a network implementation detail? In our testing, the patches were sent over a streaming HTTP connection. We were able to signal to Relay that there are no more patches by detecting the HTTP connection has been closed.

I will add some more tests to make sure the AsyncIterator is closed correctly for queries that contain multiple @streams and @defers.

[Keweiqu]

@robrichard good question. For the purpose of this PR, my inclination is to aim for only the absolutely essential features, because it's easier to add than to remove.

If we can signal to Relay through detecting a closed HTTP connection without relying on the "extension" field, and @stream/@defer proves to work fine. Let's start with it as the initial step.

We plan to initiate the discussion of evaluating whether to formalize @defer and @stream as a spec and how to best go about it in the next graphql working group meeting. From there on, we will have more information on whether is_final will be beneficial to the open source community at large. It will also help inform whether it's a good idea to include it in the reference implementation of graphql-js.

[IvanGoncharov]

@lilianammmatos Thanks for pushing this functionality forward 👍

however, it became clear to us over time that support for a @defer directive would occur only if it would make it's way into the official GraphQL spec.

Correct, here is document describing how new features are accepted into spec:
https://github.com/graphql/graphql-spec/blob/master/CONTRIBUTING.md#rfc-contribution-stages

There has been discussion on this issue to add @defer to the spec, but it seems it's still in it's infancy!

So the biggest blocker here is the absence of champion for this feature:

An RFC at the proposal stage is a solution to a problem with enough fidelity to be discussed in detail. It must be backed by a willing champion.

Looking at the amount of work done in this PR would be great to have you as a champion for this feature.
Do you have time for it?

If so, good next step would be to write a sketch of RFC document similar to the one that we had for Subscriptions: https://github.com/graphql/graphql-spec/blob/master/rfcs/Subscriptions.md
You can write the first iteration of this document (Introduction and Background should be enough) and submit it as PR against rfcs folder:
https://github.com/graphql/graphql-spec/blob/master/rfcs/Subscriptions.md

And we can discuss it on the Working Group meeting if you would be able to join the call on Jan 9th 5 PM UTC:
https://github.com/graphql/graphql-wg/blob/master/agendas/2020-01-09.md
You can add yourself and agenda item in PR similar to this one:
https://github.com/graphql/graphql-wg/pull/311/files

@Keweiqu Big thanks for sharing your expertise and reviewing this PR.
It would be great to have your input on the RFC document and have you at the Working Group call.

[morrys]

@lilianammmatos @robrichard @IvanGoncharov
The most important difference between this PR and my PR proposed here:
#2318
this is the mode in which "patches" are queued.

this mode of patch resolution causes a performance deterioration in the execution of the queries because if the same field is requested in 'n' fragment it is resolved 'n' times.

You can easily test it using the test case I created:
Directives-perf-test.js

This problem is also present in the @stream directive, and it is for this reason that I have not proposed its integration.

[lilianammmatos]

@IvanGoncharov I do have time to champion this! I'll start working on a sketch of an RFC document and create a pull request against rfcs folder ahead of the Working Group meeting on Jan 9th, 5 PM UTC.

I'll add myself and my colleague @robrichard to the agenda.

@Keweiqu thank you very much for your input!

[coco98]

Would it be worth talking about how we expect @defer and @stream to work with a websocket client? Over the https://github.com/apollographql/subscriptions-transport-ws spec might be a good place to start.

I couldn't find anything online based on a preliminary search. I can take a stab at putting some notes together if there isn't something already :)

[robrichard]

@coco98 while @defer and @stream can be used with a websocket in theory, it is more typically used with a standard HTTP connection with Transfer-Encoding: chunked. You can see how this works in this express-graphql PR: https://github.com/graphql/express-graphql/pull/583/files#diff-1fdf421c05c1140f6d71444ea2b27638R536

[coco98]

@robrichard Thanks!

I just meant that it would be nice to have support over websockets as well. I think this would be super convenient if one is using something like apollo client exclusively via the websocket transport. Considering that the current spec subscriptions-transport-ws supports queries/subscriptions/mutations it might be nice to add some information on what support for defer/stream would look like too. Also, this would kind of play well with the idea that GraphQL is a transport independent spec :)

[lilianammmatos]

Hi @coco98! Agreed that this should remain transport independent. As far as I know, there hasn’t been any discussion of how to adopt @defer and @stream responses over websockets, but I can't see why that wouldn't be possible.

[sibelius]

isFinal relay implementation facebook/relay@78d20cc

[mike-marcacci]

Out of curiosity, has there been any discussion around accepting an optional "priority" in @defer? I have a use-case with data requirements at more than two "tiers" of importance. Making a selection on a page requires:

1. some data for immediate display (priority=1);
2. supplemental "display-as-available" data (priority=2);
3. backgrounded data that is essentially "pre-loading" data for the most-common next actions (priority=3)

In our current scenario, I don't actually distinguish between priority 1 and 2 data, since doing so hasn't been ergonomic before @prefer, and priority 3 data is made as a separate request with heuristics that delay it until the higher-priority request is likely already in-flight. This has been working, but it feels like defer and stream provide almost all the machinery to solve this far more elegantly, except for the ability to distinguish between priority 2 and 3.

Has this kind of functionality been considered internally?

[robrichard]

Hi @mike-marcacci - graphql/graphql-spec#691 has been opened to discuss optimistic preloading.

[sibelius]

can we resync this with graphql v15 https://github.com/graphql/graphql-js/releases/tag/v15.0.0?

[acao]

so excited about this! let me know if there’s anything i need to do from the LSP/graphiql side. there is a bit of logic for existing built-in directives

[n1ru4l]

Out of curiosity why does the new experimental execute return an AsyncIterable and existing subscribe return an AsyncIterableIterator...

Which is "correct"?

You probably always want to use AsyncIterableIterator as it allows iterating over it using a async generator function (async * () { for await (const value of asyncIterableIterator) }). As as AsyncIterableIterator has a property under the Symbol.asyncIterator key.

There is currently an issue with TypeScript typings and Symbol polyfills (leebyron/iterall#49). Because of that people type their AsyncIterableIterator as AsyncIterable instead. (e.g. apollographql/graphql-subscriptions#232)

I am also very excited about this as returning AsyncIterableIterators from execute would make integrating live queries into servers easier 🤩 (https://github.com/n1ru4l/graphql-live-queries/blob/main/packages/graphql-live-query/src/executeTypes.ts).

[sibelius]

can we merge and release this as graphql@experimental

as react@experimental, relay@experimental ?

[robrichard]

@acao We have not had a chance to look at graphiql support for defer/stream yet. It would be great if you have some time to see what's needed get it working.

@sibelius we are working with @IvanGoncharov to publish this as an experimental tag on graphql-js after the next version of graphql-js is published.

[n1ru4l]

AsyncIterableIterator might be the better return type as it can easily be looped over with a for await of loop

[robrichard]

Updated to AsyncIterableIterator

[acao]

so I yarn linked this and the express-graphql PR up with graphiql monorepo real quick and it looks great so far, amazing work everyone! The only thing that doesn't work out of the box on the graphiql end of things is parsing and rendering the multipart responses.

what works transparently

• codemirror-graphql fragment and @defer/@stream completion are working
• inline codemirror-graphql fragment validation (placement, invalid types) and field argument validation are working as to spec, with the messages and all
• hover and completion documentation work as to spec

as most of us know, because we're using graphql@15, the introspection schema gives us plenty of information about directives and their arguments. previously, in graphql@14 era, the language service interface and parser had some baked-in spec directives logic because introspection didn't deal with fragments then

what doesn't work out of the box?

as expected, I'm getting raw multipart responses as per the relay spec, so we might need to implement fetch-multipart-graphql to parse the responses.

also, pinging @Urigo because we may have to consider this for the subscriptions fetcher, and supporting deferred schema

1. completion
   

2. validation from graphql-js
   
   

3. raw multipart response
   

graphiql PR forthcoming when I next get a chance, unless someone can beat me to it, haha!

[n1ru4l]

Is there any consensus or discussion ongoing about where to implement the initial data + patch merging on the GraphQL API consumer? Should the fetcher/network (e.g. apollo-link/relay-fetcher) layer always resolve with a ExecutionResult that is built from the initial result and the applied patches that is then consumed by the cache/store/layer or should the client/cache layer (e.g apollo-cache) be able to apply the sent patches to the cache directly?

Currently, I guess the network layer is the way to go; but doing it on the cache/store layer could make some optimizations possible.

[acao]

@n1ru4l looks like this implementation by the authors has a prescribed pattern for consumers as such, and it works with both apollo server and express-graphql (as per @robrichard 's notes in the express-graphql PR). Thanks @lilianammmatos and rob for building out such a helpful ecosystem and excellent PRs and documentation for this effort! Was very easy to get caught up on the latest details, and to get everything working.

I'm thinking that, as with subscriptions, we will make this an opt-in feature via the fetcher prop in 1.0.x. We are going to adopt @Urigo's wss subscriptions fetcher, and can reccomend an implementation with fetch-multipart-graphql so that folks can opt-in to either or both features.

if graphiql ends up introducing a higher-level service layer, where we generate the fetcher function (in primitive form, not template replacement) using user input, there can be an httpMultipartDeferStream boolean option that enables this for the generated fetcher.

that way, if people want to implement @defer or @stream using a non-http implementation, they are still free to handle that however using the fetcher prop

I created a discussion in GraphiQL about the GraphiQL implementation, if anyone is interested, would love to hear any feedback or to see folks propose any other/additional ideas they have. it's mostly just a discussion about abstractions around the fetcher prop that will enable easier out of the box support for @defer and @stream, @live and wss support if folks want that.

(to note, apollo studio and i'm guessing a few other graphql IDEs support this feature already)

[robrichard]

This branch is published to npm as version graphql@15.4.0-experimental-stream-defer.1 under the tag graphql@experimental-stream-defer. We encourage anyone interested in this feature to try out this version and report any feedback, issues, or potential improvements to the API. This feedback will help the working group gain confidence to move forward with standardization.

[n1ru4l]

I also created https://github.com/n1ru4l/graphql-bleeding-edge-playground for those that want to have a jump start without having to do all the boilerplate themselves

[AndreasHald]

Anyone have working examples of the experimental branch using apollo graphql server?

[n1ru4l]

@AndreasHald I think it is not possible without manual patching the apollo-server-core library (until it is properly implemented).

[AndreasHald]

@n1ru4l dang :( probably got a while to go then. thanks for the quick reply :)

[johnculviner]

Looking forward to this one! I'm seeing it on GQL libs for Java, Scala and Ruby now too.

This solves one of my biggest gripes with GQL - your response performance is limited to the performance characteristics of the slowest resolver dependency unless you issue multiple queries which really limits the ability of GQL to serve as a near perfect abstraction over legacy APIs etc.

[josephsavona]

Is there any consensus or discussion ongoing about where to implement the initial data + patch merging on the GraphQL API consumer? Should the fetcher/network (e.g. apollo-link/relay-fetcher) layer always resolve with a ExecutionResult that is built from the initial result and the applied patches that is then consumed by the cache/store/layer or should the client/cache layer (e.g apollo-cache) be able to apply the sent patches to the cache directly?

@n1ru4l Relay implements support for @defer and @stream directly in the core of the runtime. During the processing of an operation we maintain some state about which payloads have been received and which payloads are pending, and incrementally process each new payload unit. This approach allows us to avoid re-processing previously received data, as would happen if we were to apply patches at the network layer and push full, merged payloads into the cache each time a new chunk was received. I imagine other clients with a consistent cache will likely use a similar approach, while simpler clients without a cache can patch payloads and re-vend the entire updated response object.

[glasser]

(moving this comment to #2848)