New response format (June 2023)

[robrichard]

Over the past few months the incremental delivery working group has been working on the issues of data duplication, response amplification, and data consistency. The result of this is an updated response format for the @defer & @stream proposal, which contains some significant differences from the previous iterations. We would now like to get wider feedback on this proposal.

Features

• No overlapping branching.
• No duplicate delivery of any fields
• Reduces the risk of response amplification
• No complex WeakMap/similar caching shenanigans.
• No query rewriting.
• No look-ahead.
• Consistent delivery of fragments.
• Each deferred fragment can be delivered independently
• Labels are entirely optional - only exist for providing information to the client in case it wants to correlate pendings with the original directives.
• Labels do not affect the grouping of defers.
• Deferred fragment data will not be delivered if:
  • A null bubbled above the highest field in the deferred fragment
  • A null bubbles to a field that is shared with another deferred fragment

The incremental array is the actual data to be applied to the response, while the pending and completed arrays return "metadata" about the execution. They are used to inform clients that defers are being executed and when all fields for that defer have been delivered.

To ensure consistency, clients are expected to process all objects in the incremental array for a given payload before re-rendering the associated UIs.

Defer Examples

Example A

Overlapping fields from initial payload are not sent in subsequent payloads. Fragment consistency is preserved. Even if "MyFragment" is ready earlier, it is not sent until "j" is also ready.

query ExampleA {    
  f2 { a b c { d e f { h i } } }
  ... @defer {
    MyFragment: __typename
    f2 { a b c { d e f { h j } } }
  }
}

Example Result:

[
  {
    "data": {
      "f2": {
        "a": "a",
        "b": "b",
        "c": {
          "d": "d",
          "e": "e",
          "f": { "h": "h", "i": "i" }
        }
      }
    },
    // `pending`'s `path` field is always the location of `@defer` directive.
    // If `pending` is not sent, client should assume defer has been inlined by server
    "pending": [{ "path": [], "id": "0" }],
    "hasNext": true
  },
  {
    "incremental": [
      { "id": "0", "data": { "MyFragment": "Query" } }
      // `subPath` in `incremental` object is a sub-path of the `path` sent in the `pending` object.
      { "id": "0", "subPath": ["f2", "c", "f"], "data": { "j": "j" } }
    ],
    "completed": [{ "id": "0" }],
    "hasNext": false
  }
]

Example A2

Overlapping fields from parent defers are also not duplicated

query Example A2 {    
  f2 { a b c { d e f { h i } } }
  ... @defer(label: "D1") {
    f2 { a b c { d e f {
      h i j k
      ...@defer(label: "D2") {
        h i j k l m
      }
    } } }
  }
}

Example Result:

[
  {
    "data": {"f2": {"a": "A", "b": "B", "c": {
      "d": "D", "e": "E", "f": {
        "h": "H", "i": "I"
      }
    }}},
    "pending": [{"id": "0", "path": [], "label": "D1"}],
    "hasNext": true
  },
  {
    "incremental": [
      {"id": "0", "subPath": ["f2", "c", "f"] "data": {"j": "J", "k": "K"}}
    ],
    "pending": [{"id": "1", "path": ["f2", "c", "f"], "label": "D2"}],
    "completed": [
      {"id": "0"}
    ],
    "hasNext": true
  },
  {
    "incremental": [
      {"id": "1", "data": {"l": "L", "m": "M"}}
    ],
    "completed": [
      {"id": "1"}
    ],
    "hasNext": false
  }
]

Example B

Overlapping fields e & f are sent with whichever fragment completes first, and not sent in subsequent payloads.

{
  a {
    b {
      c {
        d
      }
      ... @defer(label: "Red") {
        e {
          f
        }
        potentiallySlowFieldA
      }
    }
  }
  ... @defer(label: "Blue") {
    a {
      b {
        e {
          f
        }
      }
    }
    g {
      h
    }
    potentiallySlowFieldB
  }
}

Example Result when potentiallySlowFieldA completes first:

[
  {
    "data": {
      "a": { "b": { "c": { "d": "d" } } }
    },
    "pending": [
      { "path": [], "id": "0", "label": "Blue" },
      { "path": ["a", "b"], "id": "1", "label": "Red" }
    ],
    "hasNext": true
  },
  {
    "incremental": [
      { "id": "1", "data": { "potentiallySlowFieldA": "potentiallySlowFieldA" } },
      // e is returned in a separate incremental data because there is overlap with other defer paths
      { "id": "1", "data": { "e": { "f": "f" } } }
    ],
    "completed": [{ "id": "1" }]
    "hasNext": true
  },
  {
    "incremental": [
      // `g` & `potentiallySlowFieldB` are returned in the same incremental data because they both belong to the same set of defers
      { "id": "0", "data": { "g": { "h": "h" }, "potentiallySlowFieldB": "potentiallySlowFieldB" } }
    ],
    "completed": [{ "id": "0" }]
    "hasNext": false
  }
]

Example Result when potentiallySlowFieldB completes first:

[
  {
    "data": {
      "a": { "b": { "c": { "d": "d" } } }
    },
    "pending": [
      { "path": [], "id": "0", "label": "Blue" },
      { "path": ["a", "b"], "id": "1", "label": "Red" }
    ],
    "hasNext": true
  },
  {
    "incremental": [
      { "id": "0", "data": { "g": { "h": "h" }, "potentiallySlowFieldB": "potentiallySlowFieldB" } },
      // e is returned in a separate incremental data because there is overlap with other defer paths
      // id with longer path is preferred
      { "id": "1", "data": { "e": { "f": "f" } } }
    ],
    "completed": [{ "id": "0" }]
    "hasNext": true
  },
  {
    "incremental": [
      { "id": "1", "data": { "potentiallySlowFieldA": "potentiallySlowFieldA" } }
    ],
   "completed": [{ "id": "1" }]
    "hasNext": false
  }
]

Example D

Non-overlapping fields in child selections of overlapping fields are sent in separate incremental objects.

query ExampleD {
  me {
    ... @defer {
      list {
        item {
          id
        }
      }
    }
  ...@defer {
    me {
      list {
        item {
          value
        }
      }
    }
  }
}

Example Result

[
  {
    "data": { "me": {} },
    "pending": [
      { "path": [], "id":" 0" },
      { "path": ["me"], "id": "1" }
    ],
    "hasNext": true
  },
  {
    "incremental": [
      {
        "id": "1",
        "data": { "list": [{ "item": {} }, { "item": {} }, { "item": {} }] }
      },
      // in this case "id" resolves before value
      { "id": "1", "subPath": ["list", "item", 0], "data": { "id": "1" } },
      { "id": "1", "subPath": ["list", "item", 1], "data": { "id": 2 } },
      { "id": "1", "subPath": ["list", "item", 2], "data": { "id": 3 } }
    ],
    "completed": [{ "id": "1" }],
    "hasNext": true
  },
  {
    "incremental": [
      { "id": "0", "subPath": ["me", "list", "item", 0], "data": { "value": "Foo" } },
      { "id": "0", "subPath": ["me", "list", "item", 1], "data": { "value": "Bar" } },
      { "id": "0", "subPath": ["me", "list", "item", 2], "data": { "value": "Baz" } }
    ],
    "completed": [{ "id":" 0" }],
    "hasNext": false
  }
]

Example F

Defer fragments with no fields are skipped entirely

query ExampleF {
  me {
    ...@defer(label: "A") {
      ...@defer(label: "B") {
        a b
      }
    }
  }
}

Example Result

[
  {
    "data": {
      "me": {}
    },
    "pending": [
      {"id": "0", "path": ["me"], "label": "B"}
    ],
    "hasNext": true
  },
  {
    "incremental": [
      {"id":0 , "data": {"a": "A", "b": "B"}}
    ],
    "completed": [
      {"id": "0"}
    ],
    "hasNext": false
  }
]

Example G

Deferred fragments at the same path are delivered independently

query ExampleG {
  me {
    ...Projects
    ...Billing @defer(label: "Billing")
  }
}

fragment Projects on User { 
  id
  avatarUrl
  projects {
    name
  }
}

fragment Billing on User {
  tier
  renewalDate
  latestInvoiceTotal
  ...PreviousInvoices @defer(label:"Prev")
}


fragment PreviousInvoices on User {
  previousInvoices { name }
}

Example Result

[
  {
    "data": {
      "me": {
        "id": 1,
        "avatarUrl": "http://…",
        "projects": [{ "name": "My Project" }]
      }
    },
    "pending": [
      { "id": 0, "path": ["me"], "label": "Billing" },
      { "id": 1, "path": ["me"], "label": "Prev" }]
    ],
    "hasNext": true
  },
  {
    "incremental": [
      {
        "id": 0,
        "data": {
          "tier": "BRONZE",
          "renewalDate": "2023-03-20",
          "latestInvoiceTotal": "$12.34"
        }
      }
    ],
    "completed": [{ "id": 0 }],
    "hasNext": true
  },
  {
    "incremental": [
      {
        "id": 1,
        "data": { "previousInvoices": [{ "name": "My Invoice" }] }
      }
    ],
    "completed": [{ "id": 1 }],
    "hasNext": false
  }
]

Example H

If a field in a subsequent defer nulls a previously sent field due to null bubbling, the entire fragment will not be delivered. Clients should treat this fragment similar to a fragment that is @skip(if: true).

Assume baz resolves before qux and qux is non-nullable and returns null.

query ExampleH {
  ... @defer(label: "A") {
    me {
      foo {
        bar {
          baz
        }
      }
    } 
  }
  me {
    ... @defer(label: "B") {
      anotherField
      foo {
        bar {
          qux
        }
      }   
    }
  }
}

Example Result

[
  {
    "data": {
      "me": {}
    },
    "pending": [
      {"id": "0", "path": [], "label": "A"},
      {"id": "1", "path": ["me"], "label": "B"}
    ],
    "hasNext": true
  },
  {
    "incremental": [
      {
        "id": "0", 
        "subPath": ["me"], 
        "data": {
          "foo": { 
            "bar": {
              "baz": "BAZ"
            }
          }
        }
      }
    ],
    "completed": [
      {"id": "0"}
    ],
    "hasNext": true
  },
    {
    // No "incremental" objects are sent. "completed" object contains the 
    // errors that triggered the conflict due to null bubbling. 
    "completed": [
      {
        "id": "1", 
        "errors": [
          {
            "message": "Cannot return null for non-nullable field Bar.qux.",
            "locations": [...],
            "path": ["foo", "bar", "qux"]
          }
        ]
      }
    ],
    "hasNext": false
  }
]

Stream Examples

Example I

query ExampleI {
  person(id: "1") {
    ...HomeWorldFragment @defer(label: "homeWorldDefer")
    name
    films @stream(initialCount: 2, label: "filmsStream") {
      title
    }
  }
}
fragment HomeWorldFragment on Person {
  homeworld {
    name
  }
}

[
  {
    "data": {
      "person": {
        "name": "Luke Skywalker",
        "films": [
          { "title": "A New Hope" },
          { "title": "The Empire Strikes Back" }
        ]
      }
    },
    "pending": [
      {"id": "0", "path": ["person"], "label": "homeWorldDefer"},
      // `pending` path is location of stream directive
      // one `pending` object is sent for the entire stream
      {"id": "1", "path": ["person", "films"], "label": "filmsStream"}
    ],
    "hasNext": true
  },
  {
    "incremental": [
      // No subpath or index, items must be returned in order. 
      // Multiple items can be returned in array
      { "id": "1", "items": [{"title": "Return of the Jedi"}] },
    ],    
    "hasNext": true
  },
  {
    // `completed` object may be sent after final list item, in cases where the server cannot know the list has ended synchronously 
    "completed": [{"id": "1"}],
    "hasNext": true
  }.
  {
    "incremental": [
      {"id": "0", "data": {"homeworld": {"name": "Tatooine"}}},
    ],
    "completed": [{"id": "0"}],
    "hasNext": false
  }
]

Example J

If after some list fields are streamed:

• either the underlying datasource errors
• or a null bubbles up to the list field

A "completed" object is sent for the stream with the errors and no more streamed results will be sent

query ExampleJ {
  person(id: "1") {
    films @stream(initialCount: 1, label: "filmsStream") {
      title
    }
  }
}

[
  {
    "data": {
      "person": {
        "films": [{ "title": "A New Hope" }]
      }
    },
    "pending": [
      {"id": "1", "path": ["person", "films"], "label": "filmsStream"}
    ],
    "hasNext": true
  },
  {
    "incremental": [
      // one streamed list item is delivered successfully
      { "id": "1", "items": [{"title": "The Empire Strikes Back"}] }
    ],    
    "hasNext": true
  },
    {
    // An error is encountered while loading the next list item
    "completed": [
      {
        "id": "1", 
        "errors": [
          {
            "message": "Cannot return null for non-nullable field Person.films.",
            "locations": [...],
            "path": ["person", "filmns"]
          }
        ]
      }
    ],    
    "hasNext": false
  }
]

Solution Criteria Evaluation

[A]	[B]	[C]	[D]	[E]	[F]	[G]	[H]	[I]	[J]	[K]	[L]	[M]	[N]	[O]	[P]	[Q]
✅️	✅️	✅️	✅️	✅	✅️	✅️	✅️	✅️	✅	✅️	✅️	✅	✅	❔	🚫	🚫

FAQ's

Why not done: true on incremental instead of completed?

It's possible for a single incremental object result to "complete" more than one deferred fragment. In this example assume baz takes longer to resolve than the other fields.

{
  me {
    ... @defer(label: "A") {
      foo
      bar
    }
    ... @defer(label: "B") {
      bar
      baz
    }
    ... @defer(label: "C") {
      foo
      baz
    }
  }
}

Example Response

{
  data: { me: {} },
  pending: [
    {id: 0, label: "A", path: ["me"]},
    {id: 1, label: "B", path: ["me"]},
    {id: 2, label: "C", path: ["me"]}
  ]
}
{
  incremental: [
    {id: 0, data: {foo: "foo"}},
    {id: 0, data: {bar: "bar"}}
  ],
  completed: [
    {id: 0}
  ]
}
{
  incremental: [
    {id: 1, data: {baz: "baz"}}
  ],
  completed: [
    {id: 1},
    {id: 2}
  ]
}

Should incremental[i].id be an array, since data is shared by multiple ids?

No, because then it would introduce confusion on which id is the prefix of the subpath. This single id does not signify that this is the only fragment being delivered as multiple fragments can be delivered together. We will prefer the id with the longest path in pending to minimize the size of the subpath.

Regarding example H, I'm not sure why "anotherField" need to suffer the casualty and not be delivered?

If we delivered the other fields in the fragments it could put clients into a bad state where they understand that all the fields from defer "B" have been delivered, but it has an object for "bar" and no result for "qux". Since the schema has a non-null constraint on "qux" this is a state that should be impossible.

We discussed options where bar is resent as null. Maybe there's some advanced clients that can simultaneously render both states if they occur in unrelated UI components, but if you are constructing the final reconciled object you end up either dropping the whole fragment, rendering the fragment in the above described invalid state, or remove data that has already been displayed to the user.

This should only happen when non-null fields are shared across sibling defers. We were thinking this situation could potentially be addressed in the future by the client controlled nullability proposal.

[jahfer]

👋 First off, thank you for exploring the topic of streaming response in GraphQL and pushing the proposal forward over the past several years. At Shopify, we’ve recently begun exploring @defer on some of the more performance-sensitive public APIs we offer. At the moment, we’ve implemented a client based on the original RFC, and have implemented a backend for both that previous spec and the one proposed in this discussion.

I see significant benefits to the new design, but also some space for possible refinements. I think the most important aspect of the new design is enforcing no duplicate delivery of any fields. This does add some complexity to the backend—in diffing the different branches and finding the overlapping leaves—but that is a cost that only needs to be paid once per query.

I know past discussions considered merging fragments together to a single deferred response, so I’m glad to see this proposal allows each deferred fragment to be delivered independently.

Where I would like to understand the proposal more is around the id, path, and subPath fields.

Cache key

In past discussions, it was suggested that the usage of id is intended to help clients cache pointers to subtrees, allowing the id to be used as the key. As well, the id is also used in any overlapping incremental results as a way of producing the shortest subpath:

We will prefer the id with the longest path in pending to minimize the size of the subpath.

It isn’t clear to me why id is more useful than using the path itself as a cache key (e.g. "a:b" instead of "0"). Introducing an id concept means we now must define semantics for that field (e.g. what are the rules for reuse of IDs?), and require client and server implementations to understand how to produce and consume that value safely. What value does the id carry that a path or label does not?

State management

In the proposed design, clients who receive an incremental payload must use the id reference to look up the associated path from a past response's pending object. This intertwining of id and path requires the client to maintain metadata state between messages, and rather than being a mechanism of caching convenience, it forces the client to track the mapping of id -> path so incremental payloads can be correctly applied. Clients cannot opt out of this state management as the data provided in the incremental payload is insufficient to apply the patch. This stateful coupling of past responses to future ones is my largest concern of the new design.

Improving completed

My instinct is that completed will be most useful for one-off client-side stateful actions so I want to make sure we don't lose the ability to leverage that context, as I think it's quite a nice design choice. For example, if pending is used to cache references to subtrees based on the path (so we don't need to traverse the data tree again when receiving an incremental result), the values in completed can be used to clear the cached references. Similarly, if clients are deferring the rendering of specific components based on pending subqueries, a result in the completed field will signal to the client that component can now be rendered.

If we add the label to the completed field, clients can opt-into using the label as their own cache key if they don't wish to construct one on the fly based on the path. For example, the completed list could look like [{ "path": ["a", "b"], "label": "Red" }], providing richer information about which branch of the query has been resolved, beyond the proposed [{ "id": "0" }].

Migrating clients

There is a significant difference between the proposed design in this issue and the one that appeared in the original RFC. I think it may be valuable to lessen this gap by making the incremental object mirror the patch format of the original design. This would allow any existing implementations to continue to use their patch application logic, and make adoption much simpler. As it stands, this proposed format will require a significant rearchitecting of any clients to track ids.

Original design

{
  "label": "homeWorldDefer",
  "path": ["person"],
  "data": {
    "homeworld": {
      "name": "Tatooine"
    }
  },
  "hasNext": true
}

Proposed design

{
  "incremental": [
    // No access to `path` or `label`
    // Must use new cross-message `id` state
    {
      "id": "0",
      "data": {
        "homeworld": {
          "name": "Tatooine"
        }
      }
    }
  ],
  "completed": [{ "id": "0" }],
  "hasNext": true
}

Suggested alternative design

{
  "incremental": [
    // Effectively, taking the top-level structure of the
    // old design, and moving it inside of `incremental`
    {
      "label": "homeWorldDefer",
      "path": ["person"],
      "data": {
        "homeworld": {
          "name": "Tatooine"
        }
      }
    }
  ],
  // Continue to provide metadata about branch completions
  "completed": [{ "label": "homeWorldDefer", "path": ["person"] }],
  "hasNext": true
}

Alternative proposal examples

Below I have replayed some of the above examples with a proposed alternative design. I don't intend for this to be a perfect iteration by any means, but I do believe it offers some improvements on the design with minimal drawbacks.

Example A

query ExampleA {
  f2 {
    a
    b
    c {
      d
      e
      f {
        h
        i
      }
    }
  }
  ... @defer {
    MyFragment: __typename
    f2 {
      a
      b
      c {
        d
        e
        f {
          h
          j
        }
      }
    }
  }
}

Example Result:

  [
    {
      "data": {
        "f2": {
          "a": "a",
          "b": "b",
          "c": {
            "d": "d",
            "e": "e",
            "f": { "h": "h", "i": "i" }
          }
        }
      },
-     "pending": [{ "path": [], "id": "0" }],
+     "pending": [{ "path": [] }],
      "hasNext": true
    },
    {
      "incremental": [
-       { "id": "0", "data": { "MyFragment": "Query" } },
+       { "path": [], "data": { "MyFragment": "Query" } },
-       { "id": "0", "subPath": ["f2", "c", "f"], "data": { "j": "j" } }
+       { "path": ["f2", "c", "f"], "data": { "j": "j" } }
      ],
-     "completed": [{ "id": "0" }],
+     "completed": [{ "path": [] }],
      "hasNext": false
    }
  ]

In this example, id was replaced with path in all cases. In the initial response, we continue to hint to the client which subtree paths should be cached for more efficient retrieval.

In the original proposed design, subPath is used when there are nested calls to @defer or if a deferred fragment overlaps some non-deferred data. In this proposed alternative, the subPath concept is dropped and instead the path is provided, which always refers to the key-path from the root. This keeps the design consistent with the original RFC and simplifies clients. As a consequence, this does mean caches performed early based on the hints in pending may not be useable. That can be mitigated by specifying a label:

query ExampleA {
  f2 {
    a
    b
    c {
      d
      e
      f {
        h
        i
      }
    }
  }
  ... @defer(label: "Red") {
    MyFragment: __typename
    f2 {
      a
      b
      c {
        d
        e
        f {
          h
          j
        }
      }
    }
  }
}

[
  {
    "data": {
      "f2": {
        "a": "a",
        "b": "b",
        "c": {
          "d": "d",
          "e": "e",
          "f": { "h": "h", "i": "i" }
        }
      }
    },
    "pending": [{ "path": [], "label": "Red" }],
    "hasNext": true
  },
  {
    "incremental": [
      { "label": "Red", "path": [], "data": { "MyFragment": "Query" } },
      { "label": "Red", "path": ["f2", "c", "f"], "data": { "j": "j" } }
    ],
    "completed": [{ "path": [] }],
    "hasNext": false
  }
]

By using a label, clients can identify which pending path is associated to the incremental payload, and follow only the parts of the path that were not present in the original pending dataset. That allows cached subtrees to be used for every incremental payload sent by the server.

In an alternative design, we could specify both path and subpath (e.g. "path": [], "subPath": ["f2", "c", "f"]. In that design, path will always match the one defined in pending and also refer to the site of the @defer directive on the query. Clients would need to concatenate the path and subpath to produce the full path to apply the patch to. I don't mind this, but having both fields is somewhat awkward, and doesn't offer a significant leap in performance compared with the simpler proposal.

Example B

{
  a {
    b {
      c {
        d
      }
      ... @defer(label: "Red") {
        e {
          f
        }
        potentiallySlowFieldA
      }
    }
  }
  ... @defer(label: "Blue") {
    a {
      b {
        e {
          f
        }
      }
    }
    g {
      h
    }
    potentiallySlowFieldB
  }
}

Example Result when potentiallySlowFieldA completes first:

[
  {
    "data": {
      "a": { "b": { "c": { "d": "d" } } }
    },
    "pending": [
-     { "path": [], "id": "0", "label": "Blue" },
+     { "path": [], "label": "Blue" },
-     { "path": ["a", "b"], "id": "1", "label": "Red" }
+     { "path": ["a", "b"], "label": "Red" }
    ],
    "hasNext": true
  },
  {
    "incremental": [
-     { "id": "1", "data": { "potentiallySlowFieldA": "potentiallySlowFieldA" } },
+     { "path": ["a", "b"], "label": "Red", "data": { "potentiallySlowFieldA": "potentiallySlowFieldA" } },
-     { "id": "1", "data": { "e": { "f": "f" } } }
+     { "path": ["a", "b"], "label": "Red", "data": { "e": { "f": "f" } } }
    ],
-   "completed": [{ "id": "1" }]
+   "completed": [{ "path": ["a", "b"], "label": "Red" }],
    "hasNext": true
  },
  {
    "incremental": [
-     { "id": "0", "data": { "g": { "h": "h" }, "potentiallySlowFieldB": "potentiallySlowFieldB" } }
+     { "path": [], "label": "Blue", "data": { "g": { "h": "h" }, "potentiallySlowFieldB": "potentiallySlowFieldB" } }
    ],
-   "completed": [{ "id": "0" }]
+   "completed": [{ "path": [], "label": "Blue" }],
    "hasNext": false
  }
]

Example Result when potentiallySlowFieldB completes first:

[
  {
    "data": {
      "a": { "b": { "c": { "d": "d" } } }
    },
    "pending": [
-     { "path": [], "id": "0", "label": "Blue" },
+     { "path": [], "label": "Blue" },
-     { "path": ["a", "b"], "id": "1", "label": "Red" }
+     { "path": ["a", "b"], "label": "Red" }
    ],
    "hasNext": true
  },
  {
    "incremental": [
-     { "id": "0", "data": { "g": { "h": "h" }, "potentiallySlowFieldB": "potentiallySlowFieldB" } },
+     { "path": [], "label": "Blue", "data": { "g": { "h": "h" }, "potentiallySlowFieldB": "potentiallySlowFieldB" } },
-     { "id": "1", "data": { "e": { "f": "f" } } }
+     { "path": ["a", "b"], "label": "Red", "data": { "e": { "f": "f" } } }
    ],
-   "completed": [{ "id": "0" }]
+   "completed": [{ "path": [], "label": "Blue" }]
    "hasNext": true
  },
  {
    "incremental": [
-     { "id": "1", "data": { "potentiallySlowFieldA": "potentiallySlowFieldA" } }
+     { "path": ["a", "b"], "label": "Red", "data": { "potentiallySlowFieldA": "potentiallySlowFieldA" } }
    ],
-  "completed": [{ "id": "1" }]
+  "completed": [{ "path": ["a", "b"], "label": "Red" }]
    "hasNext": false
  }
]

Example K(?)

In this new example we can see how using label provides a useful, but optional, way to disambiguate sibling @defer calls, :

query ExampleSibling {
  a {
    b {
      ... @defer {
        c {
          d
        }
      }
      ... @defer {
        e {
          f
        }
      }
    }
  }
}

Example Result:

[
  {
    "data": {
      "a": { "b": {} }
    },
    "pending": [{ "path": ["a", "b"] }],
    "hasNext": true
  },
  {
    "incremental": [{ "path": ["a", "b"], "data": { "e": { "f": "f" } } }],
    // empty since another deferred fragment shares this path
    "completed": [],
    "hasNext": true
  },
  {
    "incremental": [{ "path": ["a", "b"], "data": { "c": { "d": "d" } } }],
    // now we can identify this path as fully resolved
    "completed": [{ "path": ["a", "b"] }],
    "hasNext": false
  }
]

If the client wishes to be notified of each completion of a deferred fragment, they can use a label:

query ExampleSibling2 {
  a {
    b {
      ... @defer(label: "Red") {
        c {
          d
        }
      }
      ... @defer(label: "Blue") {
        e {
          f
        }
      }
    }
  }
}

Example Result:

[
  {
    "data": {
      "a": { "b": {} }
    },
    "pending": [
      { "path": ["a", "b"], "label": "Red" }
      { "path": ["a", "b"], "label": "Blue" }
    ],
    "hasNext": true
  },
  {
    "incremental": [
      { "path": ["a", "b"], "data": { "e": { "f": "f" } } }
    ],
    "completed": [{ "path": ["a", "b"], "label": "Blue" }],
    "hasNext": true
  },
  {
    "incremental": [
      { "path": ["a", "b"], "data": { "c": { "d": "d" } } }
    ],
    "completed": [{ "path": ["a", "b"], "label": "Red" }],
    "hasNext": false
  }
]

Design Summary

In the proposed alternative design, the client is still able to:

• Cache pointers to subtrees based on the objects in pending
• Identify which subtrees have resolved unless there are sibling @defer calls without using label
  • The "unless" here is where this design is inferior to the id-based form, but is signficantly less complex for both the client and server, as we reduce the amount of state management on both sides
  • This is easily mitigated by the client opting into the label field

Where this design improves:

• Minimal effort to support new design
  • Existing clients can re-use patch-application logic from original RFC design
• Clients can ignore all metadata if they so choose, while still constructing the correct end-state
  • Clients can still determine which subtrees have resolved via completed
  • Clients can directly apply the values in the incremental payloads since the path is provided

Where the protocol changes (I think):

• If there are multiple sibling @defer sites that have not resolved, the path must not appear in the completed list until the last sibling has resolved, unless it can be disambiguated via a label

Please let me know if I've made any incorrect assumptions about the use cases for the proposed designs, or anywhere this alternative approach might fare worse in real-world scenarios. I am eager to continue this conversation :)

[benjie]

Awesome, happy to hear that you're using this and you have feedback!

---

It isn’t clear to me why id is more useful than using the path itself as a cache key (e.g. "a:b" instead of "0").

Consider this query:

{
  a {
  ... @defer {
    b
    ... @defer {
      c
    }
    ... @defer {
      d
    }
  }
}

Here we have three different defers that each share the path a. They need to be uniquely identified in the response so that the client can determine which incremental results are finished and which are not.

Introducing an id concept means we now must define semantics for that field (e.g. what are the rules for reuse of IDs?)

id is not reused, it's essentially an opaque counter. Personally I'd be happy with it being an integer, and that is indeed how I have specified it in my version of the spec edits. Rob thinks a string is more flexible for server implementations. Either way, the client shouldn't care (so long as we dictate string or int when we're done writing up the spec).

and require client and server implementations to understand how to produce and consume that value safely

Client should not attempt to interpret the id value - it is an opaque value like a cursor or a node ID, not intended to be digested and understood by the client.

The server can produce it by following the spec; e.g. in my spec proposal it's just an auto-increment value: graphql/graphql-spec@benjie/incremental-common...benjie/incremental4#diff-0f02d73330

What value does the id carry that a path or label does not?

Labels are no longer significant, they only have meaning to the client and not the server. Include or omit them at will. If provided, they will be included with the pending entries, but other than this small piece of extra data they have no effect.

Path/label combinations are therefore not guaranteed nor required to be unique.

and rather than being a mechanism of caching convenience

Please note that id is not a caching convenience feature, it is a feature that allows tracking of which @defers are currently pending/active/completed and where to store the results without having to repeat the full path each time. Essentially it gives you the ability to put a loading spinner in the right place in your UI, since the fact that data is still loading is an "out of band" piece of information - it's not inherent to the data itself, but instead is indicated in a separate channel (the pending/completed fields).

Clients cannot opt out of this state management as the data provided in the incremental payload is insufficient to apply the patch. This stateful coupling of past responses to future ones is my largest concern of the new design.

The data only needs to be maintained whilst an id is still pending, once the entry is completed it can be forgotten about. Even for a client that does not "care" about tracking the loading status of the various @defer/@stream/etc fields, the id still confers benefits in that it significantly reduces the amount of data that must be represented in memory. Particularly with @stream, the same path may be written to hundreds of times - and it's much more concise to do that with an integer representation than a full operation path plus label (if provided).

If we add the label to the completed field, clients can opt-into using the label as their own cache key if they don't wish to construct one on the fly based on the path. For example, the completed list could look like [{ "path": ["a", "b"], "label": "Red" }], providing richer information about which branch of the query has been resolved, beyond the proposed [{ "id": "0" }].

The label is included in the pending, so the completed should not need to repeat this information.

It was determined that many users will not want to write their own labels (preferring to just use @defer directly without arguments), and that ensuring labels are unique across your whole application could be too much effort - essentially requiring labels to be provided and unique would mean that you need "smart" clients to use it well, and we'd prefer for regular people to be able to use it. Label was therefore relegated to metadata, which had the side benefit of significantly reducing the complexity of what was being built since the algorithm did not need to factor labels into its logic.

I think it may be valuable to lessen this gap by making the incremental object mirror the patch format of the original design. This would allow any existing implementations to continue to use their patch application logic, and make adoption much simpler. As it stands, this proposed format will require a significant rearchitecting of any clients to track ids.

We've discussed an alternative to this to help clients migrate by implementing a "client network middleware" that can translate the new data format to the old format for older clients. One major advantage of this approach is that the final solution that we come up with will not have to be swayed by earlier non-optimal forms of the solution.

---

I hope this helps!

[yaacovCR]

Want to echo the thanks for the feedback, I think it's very helpful.

The utility of the id

The id is completely opaque. It does not help clients identify the pending-active-completed state of the requested fragments.

Consider this query:

{
  a {
  ... @defer {
    b
    ... @defer {
      c
    }
    ... @defer {
      d
    }
  }
}

Here we have three different defers that each share the path a. They need to be uniquely identified in the response so that the client can determine which incremental results are finished and which are not.

Although the id does uniquely identify each defer, the client cannot use the id to differentiate them. They could be numbered sequentially, but they could be numbered in any order, or they could be not numbers at all. If clients are interested in knowing when a particular deferred fragment has completed, they must:

(1) use unique labels so that they can interpret the id mapping,
(2) include some hint in the actual returned response, a la including a__fragmentFulfilled: typename metafield, or
(3) use just the path information to deduce when all of the defers at a particular path have been delivered, which is less granular but perhaps better than nothing.

The use of id brings distinct advantages over the path-label combination:

(A) it allows shortening paths to subpaths, which could be very helpful in streams as @benjie notes, and
(B) it reduces slightly the metadata required in the response for completed entries.

Those advantages are real, and are even more compelling when a label is not used such that the path is not even unique! But the id story as it stands now is a payload optimization story rather than a new feature in terms of server-client signaling. @benjie let me know what I'm missing.

Are there any disadvantages to id?

(i) In terms of changes to required client-side state, there would be the trivial addition of the mapping of the path-label combination to that id, but the overall order of magnitude of preserved state would be unchanged.
(ii) The addition of id makes the logic for (3) above a tiny but more involved than it would be otherwise.¹.

For what it's worth....

The PR at graphql-js with an implementation does not yet use id, nor does the associated spec PR, although this will probably change very soon, as the advantages seem to outweigh the disadvantages.

Call me a very reluctant member of team id.

One last thought

We could require the label to be unique AND either (X) make the label mandatory and set the id equal to the path+label or (Y) have the label remain optional and set the id to the path+label only when it is provided.² To me, it seems desirable to not require developers to always use labels. I prefer not to have an id field with two different formats, one for when label is not provided, and one for when provided.

Translation to the old format

We've discussed an alternative to this to help clients migrate by implementing a "client network middleware" that can translate the new data format to the old format for older clients.

We had more involved candidate response formats that could potentially include among other things the ability to perform a translation to the older format. We rejected those formats as not very human-readable. The June 2023 format we have above, however, does not directly allow a translation to the old format because there is information lost during deduplication.³

Consider Example B above, the defer labelled Blue also includes the incremental entry { "id": "1", "data": { "e": { "f": "f" } } } as that data was already sent and is available in the presumed cached "final reconcilable object." However, if we read from the "final reconcilable object" at Blue's path [], we will get the entire "final reconcilable object," all fields within the parent and within the sibling defer, not the old format at all.⁴

My take: the original format was based on a stage 2 proposal released under experimental flags and can be safely deprecated.

From https://github.com/graphql/graphql-spec/blob/main/CONTRIBUTING.md

Drafts may continue to evolve and change, occasionally dramatically, and are not guaranteed to be accepted. Therefore, it is unwise to rely on a draft in a production GraphQL Service. GraphQL libraries should implement drafts to provide valuable feedback, though are encouraged not to enable the draft feature without explicit opt-in when possible.

Footnotes

1. Rather than storing the total number of defers at a particular path, and then simply counting down as the path is returned, an additional step is required, doing a lookup of the path for each id as it is returned. Performance-wise, this would not seem to be a huge deal, but it might seem counterintuitive, as one might think that something called id is helping with the problem of client-side identification, but it's just not, so we have to get back to the initial path. ↩

2. Even if the label is unique across an operation, because a fragment can be deferred within a list field, the path is required to make it truly unique within the response. ↩

3. From what I recall when rejecting the more involved format, we were satisfied that it would be possible to straightforwardly reconstruct an immutable copy of the data for each defer. I do not recall that we specifically were discussing whether we could map to the old format. ↩

4. Could we map from the new format to the old format if we are armed both with the response and the original operation? This should be possible IF the operation included all the necessary __typename fields such that the "final reconcilable object" is completely type annotated. This would cover where spreads with type conditions are used in deferred fragments, but without type conditions in the parent. ↩

[jahfer]

Thank you both for your thoughtful replies!

and require client and server implementations to understand how to produce and consume that value safely

Client should not attempt to interpret the id value - it is an opaque value like a cursor or a node ID, not intended to be digested and understood by the client.

I believe I phrased this poorly. I understand that the ID is opaque, but consumers still need to learn about its semantics (it does not repeat…but is that true over the course of the entire response? Can they repeat over subsequent responses? Clients will need to understand this to know if they can leverage a global ID cache or need a cache per request/response stream).

But the id story as it stands now is a payload optimization story rather than a new feature in terms of server-client signaling.

I appreciate the clarity of this statement as its what I was struggling with at the beginning: why does id exist at all? I admit that I hadn't considered the potential volume of repetition that would be commonly seen in a @stream response until @benjie mentioned it, so now I understand the healthy dose of paranoia around payload size of each response.

I do wonder if we are sacrificing ergonomics for the sake of what might amount to a couple of kilobytes spread out over multiple messages though. Is this a discussion that has happened for other parts of the GraphQL spec? How do you evaluate the priority between terseness and usability?

---

(i) In terms of changes to required client-side state, there would be the trivial addition of the mapping of the path-label combination to that id, but the overall order of magnitude of preserved state would be unchanged.

My concern is less about the volume of state that needs to be recorded, but the fact that any state is required in order to interpret the subsequent responses. It would be particularly elegant to be able to hand off streamed responses to a pipeline that can parse, process, and apply each incremental patch without having to maintain some connection to earlier requests beyond the raw data. As well, payloads beyond the initial response can themselves include new pending responses, making any processing pipeline itself state-creating.

(ii) The addition of id makes the logic for (3) above a tiny but more involved than it would be otherwise.

Indeed 😄 Without labels, the client would need to group all matching paths listed in pending, collect their IDs to a list, tracking when each one has completed. In the proposal I provided, the server would be responsible for this work, outputting the relevant path in the completed array when it has resolved all deferred elements.

---

We could require the label to be unique AND either (X) make the label mandatory and set the id equal to the path+label or (Y) have the label remain optional and set the id to the path+label only when it is provided. To me, it seems desirable to not require developers to always use labels. I prefer not to have an id field with two different formats, one for when label is not provided, and one for when provided.

If id is included in the spec, I think it makes sense to keep it opaque. Assigning its value to the path + label isn't something the client should take advantage of in any meaningful way.

I agree that we should not require labels for clients to use the protocol. This was a sentiment that @benjie echoed above as well:

essentially requiring labels to be provided and unique would mean that you need "smart" clients to use it well, and we'd prefer for regular people to be able to use it.

I would like the protocol to be something that is straightforward to adopt for simple clients, but with enough information to allow more advanced clients to optimize as they wish. The current requirement of id forces a certain shape of client that feels very opinionated to me, and also pushes a minimum level of complexity on client implementations above many other parts of GraphQL.

This is presumably a rare occasion in GraphQL where the client doesn't have the freedom to specify the shape of the response they want since the format is forced for all responses. How radical would it be to have clients opt into the payload-size-optimizing format as an argument to the directive itself?

For example:

{
  a {
  ... @defer(pathIds: true) { }
}

I can see that getting a little bit confusing if you start to nest @defer calls and we don't force inheritance of the behaviour, but that's always something that could be enforced (a child @defer must inherit the format of its parent).

[yaacovCR]

I believe I phrased this poorly. I understand that the ID is opaque, but consumers still need to learn about its semantics (it does not repeat…but is that true over the course of the entire response? Can they repeat over subsequent responses?

It is my understanding that the id is meant to be represent and be unique to an individual path and label combination for an individual response.

Clients will need to understand this to know if they can leverage a global ID cache or need a cache per request/response stream).

So I think they clients will need a cache per request/response stream.

My concern is less about the volume of state that needs to be recorded, but the fact that any state is required in order to interpret the subsequent responses. It would be particularly elegant to be able to hand off streamed responses to a pipeline that can parse, process, and apply each incremental patch without having to maintain some connection to earlier requests beyond the raw data. As well, payloads beyond the initial response can themselves include new pending responses, making any processing pipeline itself state-creating.

The introduction of deduplication requires maintenance of state for the duration of a response within the incrementally built "final reconcilable object." This can be done solely by processing the incremental entries if they contain complete paths rather than id + subPath. The introduction of id means that to build the "final reconcilable object," clients also have to process "pending" entries to later understand the mapping between the id and path, even if the client is not interested in tracking the pending/completed state of individual payloads. This mapping will have to live alongside the final reconcilable object. More accurately, we expect that the mapping will not be between the id and the path, but between the id and the node within the final reconcilable object corresponding to the path so that subsequent insertions can be O(1). This tradeoff seems worth it?

(ii) The addition of id makes the logic for (3) above a tiny but more involved than it would be otherwise.

Indeed 😄 Without labels, the client would need to group all matching paths listed in pending, collect their IDs to a list, tracking when each one has completed. In the proposal I provided, the server would be responsible for this work, outputting the relevant path in the completed array when it has resolved all deferred elements.

We do not merge defers with common paths, but rather send each defer as it completes; it could be that operations include __fulfilled: typename metafields that perform there own unique signalling.

This is presumably a rare occasion in GraphQL where the client doesn't have the freedom to specify the shape of the response they want since the format is forced for all responses. How radical would it be to have clients opt into the payload-size-optimizing format as an argument to the directive itself?

For example:

{
  a {
  ... @defer(pathIds: true) { }
}

I can see that getting a little bit confusing if you start to nest @defer calls and we don't force inheritance of the behaviour, but that's always something that could be enforced (a child @defer must inherit the format of its parent).

Argument-controlled differences in payload format could definitely work, but I imagine it would be on a per client level rather than a per defer?

[robrichard]

@jahfer thanks again for spending time with the defer/stream proposal and providing feedback. I think @benjie and @yaacovCR covered most of my sentiments, but I just wanted to add a few additional notes.

id

As you noted, we think one of the major benefits of the id field is that it allows caching pointers to subtrees. While it does reduce response size, I think that's mostly irrelevant when gzip is used. Our intention was to allow clients to use these ids to store pointers to efficiently insert the incremental data in the correct place, either by inserting into a normalized store or building a final object. This does come with the tradeoff that clients needs to manage this state between payloads, but we think it's worth it to avoid repeatedly processing potentially long path arrays.

label

In your examples, you recommend using label in place of id. Keep in mind that label cannot be used to uniquely identify a @defer, even with unique labels are provided to all @defers. Consider this example:

query {
  friends {
    pet {
      @defer(label: "petNameLabel") {
        name
      }
    }
  }
}

Assuming the friends field is a list type, there will be a deferred response for each element in the friends list. The pending objects will look something like this:

[
  { "id": "0", "path": ["friends", 0, "pet"], "label": "petNameLabel" },
  { "id": "0", "path": ["friends", 1, "pet"], "label": "petNameLabel" },
  { "id": "0", "path": ["friends", 2, "pet"], "label": "petNameLabel" },
]

In this case the same label can refer to multiple different base paths. This is why we chose to use an id to reference the base path in the incremental objects.

We hope to release an alpha version of graphql-js with this new response format soon. We appreciate this feedback, please let us know if you have any more concerns.

[Keweiqu]

Thank you so much Rob!! What a milestone to celebrate!

Can we revisit the decision of throwing away the entire defer fragment if it sets a field that's delivered previously? An important principle we should consider always follow is that a query executed with @defer/@stream should always deliver the same amount of data as the same query without @defer/@stream. AKA the amount of the information should be the same, despite the format difference.

In your example below, if we executed the query without @defer, we would have delivered "anotherField" when an error occurs on "qux". However, when we apply @defer to the query, we no longer have "anotherField".

If a field in a subsequent defer nulls a previously sent field due to null bubbling, the entire fragment will not be delivered. Clients should treat this fragment similar to a fragment that is @Skip(if: true).

Assume baz resolves before qux and qux is non-nullable and returns null.

query ExampleH {
  ... @defer(label: "A") {
    me {
      foo {
        bar {
          baz
        }
      }
    } 
  }
  me {
    ... @defer(label: "B") {
      anotherField
      foo {
        bar {
          qux
        }
      }   
    }
  }
}

Example Result

[
  {
    "data": {
      "me": {}
    },
    "pending": [
      {"id": "0", "path": [], "label": "A"},
      {"id": "1", "path": ["me"], "label": "B"}
    ],
    "hasNext": true
  },
  {
    "incremental": [
      {
        "id": "0", 
        "subPath": ["me"], 
        "data": {
          "foo": { 
            "bar": {
              "baz": "BAZ"
            }
          }
        }
      }
    ],
    "completed": [
      {"id": "0"}
    ],
    "hasNext": true
  },
    {
    // No "incremental" objects are sent. "completed" object contains the 
    // errors that triggered the conflict due to null bubbling. 
    "completed": [
      {
        "id": "1", 
        "errors": [
          {
            "message": "Cannot return null for non-nullable field Bar.qux.",
            "locations": [...],
            "path": ["foo", "bar", "qux"]
          }
        ]
      }
    ],
    "hasNext": false
  }
]

[robrichard]

@Keweiqu happy to revisit this, I agree it's not ideal. We made this decision because we wanted to ensure that

1. We don't break the guarantees provided by Non-null fields
2. We don't return multiple values for the same field in a single response

I also posted this in FAQs in the parent post:

If we delivered the other fields in the fragments it could put clients into a bad state where they understand that all the fields from defer "B" have been delivered, but it has an object for "bar" and no result for "qux". Since the schema has a non-null constraint on "qux" this is a state that should be impossible.

We discussed options where bar is resent as null. Maybe there's some advanced clients that can simultaneously render both states if they occur in unrelated UI components, but if you are constructing the final reconciled object you end up either dropping the whole fragment, rendering the fragment in the above described invalid state, or remove data that has already been displayed to the user.

I am open to alternate ways to handle this case, do you have a recommendation?