Incremental delivery: RuntimeError "Invalid state while adding deferred fragment node" when a @defer'd field errors

[ben-xo]

Hi there,

I've been working to resolve a production error we've logged a few thousand times in the last 48 hours - below is the diagnosis and minimal reproduction case that I homed in on with the help of Claude code.

I raised a fix PR - #273 .

• Ben (Mixcloud)

---

Summary

experimental_execute_incrementally raises an internal RuntimeError: Invalid state while adding deferred fragment node. on a spec-valid query when two sibling top-level @defer fragments — each containing a field that errors and its own nested @defer — are executed with async resolvers. The crash is in the # pragma: no cover branch of IncrementalGraph._add_deferred_fragment_node.

We hit this in production (Strawberry over ASGI, streaming @defer as multipart/mixed): ~5,700 occurrences / 48h across ~4,100 users, firing mid-stream after the HTTP 200 status line (so clients get a truncated incremental response). It reproduces on 3.3.0a12 through 3.3.0rc0 (current tip).

Environment

• graphql-core 3.3.0rc0 (also 3.3.0a12)
• Python 3.14 (also seen on 3.13)

Query (spec-valid — passes graphql.validate)

query {
  obj {
    ... @defer(label: "a") { boom   child { ... @defer(label: "b") { fast } } }
    ... @defer(label: "c") { boomNN child { ... @defer(label: "d") { slow } } }
  }
}

boom / boomNN are async resolvers that raise; fast / slow / child are ordinary async resolvers. Small random delays perturb execution-group completion ordering (the bug is a race, so the repro loops).

Stacktrace

File ".../graphql/execution/incremental_publisher.py", line 248, in _handle_completed_execution_group
    self._incremental_graph.add_completed_successful_execution_group(completed_execution_group)
File ".../graphql/execution/incremental_graph.py", line 86, in add_completed_successful_execution_group
    self._add_incremental_data_records(incremental_records, deferred_records)
File ".../graphql/execution/incremental_graph.py", line 192, in _add_incremental_data_records
    self._add_deferred_fragment_node(deferred_fragment_record, initial_result_children)
File ".../graphql/execution/incremental_graph.py", line 266, in _add_deferred_fragment_node
    self._add_deferred_fragment_node(parent, initial_result_children)   # recurse to parent
File ".../graphql/execution/incremental_graph.py", line 262, in _add_deferred_fragment_node
    raise RuntimeError(msg)
RuntimeError: Invalid state while adding deferred fragment node.

Minimal trigger (each part is required)

Variant	Reproduces?
Two sibling top-level @defer, each with an erroring field and a distinct nested @defer	yes (~1 in 4 runs)
Only one sibling has a nested @defer	no
Single deferred fragment (not two siblings)	no
The two nested @defers share a label (dedupe to one record)	no
No erroring field	no

Mechanism

When a deferred fragment's field errors, that top-level fragment's execution group fails and the fragment is removed from _root_nodes. A sibling group completing afterwards (add_completed_successful_execution_group → _add_incremental_data_records) discovers its nested @defer and, via _add_deferred_fragment_node, recurses up to the now-removed parentless ancestor. With parent is None and initial_result_children is None, it hits the # pragma: no cover branch and raises:

def _add_deferred_fragment_node(self, deferred_fragment_record, initial_result_children=None):
    if deferred_fragment_record in self._root_nodes:
        return
    parent = deferred_fragment_record.parent
    if parent is None:
        if initial_result_children is None:  # pragma: no cover
            msg = "Invalid state while adding deferred fragment node."
            raise RuntimeError(msg)          # <-- reached via the resolver-error race
        ...

Suggested direction

Reaching a parentless record that is no longer in _root_nodes here looks tolerable rather than fatal — the errored parent's subtree isn't being delivered anyway — so returning/no-op instead of raising would mirror the future.cancelled() guard recently added to _enqueue ("defensive guard against a race with a stopping consumer"). Happy to open a PR if that direction sounds right.

Self-contained reproduction (graphql-core only)

import asyncio
import random

from graphql import (
    GraphQLDeferDirective,
    GraphQLField,
    GraphQLNonNull,
    GraphQLObjectType,
    GraphQLSchema,
    GraphQLString,
    parse,
    specified_directives,
    validate,
)
from graphql.execution import (
    ExperimentalIncrementalExecutionResults,
    experimental_execute_incrementally,
)
from graphql.pyutils import is_awaitable

MAX_ITERS = 500


async def value(v: str) -> str:
    await asyncio.sleep(random.uniform(0, 0.002))  # perturb completion ordering
    return v


async def boom(_source, _info) -> str:
    await asyncio.sleep(random.uniform(0, 0.002))
    raise RuntimeError("resolver error")


async def obj(_source, _info) -> dict:
    await asyncio.sleep(random.uniform(0, 0.001))
    return {}


OBJ = GraphQLObjectType(
    "Obj",
    lambda: {
        "fast": GraphQLField(GraphQLString, resolve=lambda *a: value("fast")),
        "slow": GraphQLField(GraphQLString, resolve=lambda *a: value("slow")),
        "boom": GraphQLField(GraphQLString, resolve=boom),
        "boomNN": GraphQLField(GraphQLNonNull(GraphQLString), resolve=boom),
        "child": GraphQLField(OBJ, resolve=obj),
    },
)
schema = GraphQLSchema(
    GraphQLObjectType("Query", {"obj": GraphQLField(OBJ, resolve=obj)}),
    directives=[*specified_directives, GraphQLDeferDirective],
)

QUERY = """
query {
  obj {
    ... @defer(label: "a") { boom   child { ... @defer(label: "b") { fast } } }
    ... @defer(label: "c") { boomNN child { ... @defer(label: "d") { slow } } }
  }
}
"""


async def run_once() -> None:
    result = experimental_execute_incrementally(schema, parse(QUERY), root_value={})
    if is_awaitable(result):
        result = await result
    if isinstance(result, ExperimentalIncrementalExecutionResults):
        async for _patch in result.subsequent_results:
            pass


async def main() -> None:
    import graphql

    print("graphql-core:", graphql.__version__)
    assert not validate(schema, parse(QUERY)), "query should be valid"
    for i in range(1, MAX_ITERS + 1):
        try:
            await run_once()
        except RuntimeError as exc:
            if "Invalid state while adding deferred fragment node" in str(exc):
                print(f"Reproduced on iteration {i}")
                raise
            raise
    print(f"Did not reproduce in {MAX_ITERS} iterations (try re-running).")


asyncio.run(main())

---

This crash, and its minimal trigger, were discovered and confirmed by fuzzing @defer/@stream query shapes with erroring async resolvers.