JSONL streaming responses miss the anti-buffering headers (Cache-Control, X-Accel-Buffering) that SSE sets

[torrresagus]

First Check

• I added a very descriptive title here.
• I used the GitHub search to find a similar question and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from collections.abc import AsyncIterable

from fastapi import FastAPI
from fastapi.testclient import TestClient
from pydantic import BaseModel


class Item(BaseModel):
    name: str


app = FastAPI()


@app.get("/sse-items")
async def sse_items() -> AsyncIterable[Item]:
    # text/event-stream response (SSE)
    yield Item(name="a")


@app.get("/jsonl-items")
async def jsonl_items() -> AsyncIterable[Item]:
    # application/jsonl response (JSON Lines streaming)
    yield Item(name="a")


client = TestClient(app)

print("SSE  ->", dict(client.get("/sse-items").headers))
# ... includes 'cache-control': 'no-cache' and 'x-accel-buffering': 'no'

print("JSONL->", dict(client.get("/jsonl-items").headers))
# ... has neither 'cache-control' nor 'x-accel-buffering'

Description

This is a small bug report with a proposed fix (happy to open the PR).

FastAPI's Server-Sent Events responses set two headers so streaming survives a proxy (fastapi/routing.py, the is_sse_stream branch, added in #15030):

response.headers["Cache-Control"] = "no-cache"
# For Nginx proxies to not buffer server sent events
response.headers["X-Accel-Buffering"] = "no"

The JSON Lines streaming responses (the yield-based application/jsonl streaming added in #15022, the is_json_stream branch) stream items incrementally in the same way, but do not set those headers.

• Expected: both incremental streaming responses tell proxies not to buffer, so clients receive items as they are produced.
• Actual: behind a proxy with buffering enabled by default (e.g. Nginx with proxy_buffering on), the JSONL lines are buffered and flushed together, which defeats the streaming. SSE is unaffected because it already sends X-Accel-Buffering: no.

The two features landed in separate PRs (SSE #15030, JSONL #15022), which is likely why only the SSE path got the headers.

Proposed fix: set the same Cache-Control: no-cache and X-Accel-Buffering: no on the JSONL StreamingResponse, mirroring the SSE branch (a 3-line change), plus a test asserting the headers. I have this implemented locally and the existing SSE + streaming test suites still pass.

Would this be welcome as a PR? One open question: include Cache-Control: no-cache too (for full consistency with SSE), or only X-Accel-Buffering: no?

Operating System

Linux

Operating System Details

WSL2 (Linux 6.6)

FastAPI Version

0.137.2

Pydantic Version

2.13.4

Python Version

3.13.7

Additional Context

No response

[fournarin]

Confirmed on 0.137.x — the asymmetry is real.

In fastapi/routing.py, the SSE branch sets both headers right before returning:

response.headers["Cache-Control"] = "no-cache"
response.headers["X-Accel-Buffering"] = "no"

The JSONL branch (is_json_stream) builds a StreamingResponse with media_type="application/jsonl" and only extends response.headers.raw — no Cache-Control or X-Accel-Buffering.

We hit this class of issue in production with SSE behind Nginx: without X-Accel-Buffering: no, tokens/events arrive in bursts instead of incrementally. JSONL streaming has the same failure mode — the client sees a 200 immediately but lines don't flush until the proxy buffer fills.

On your open question: I'd include both headers, same as SSE. X-Accel-Buffering is the one that actually disables Nginx buffering; Cache-Control: no-cache keeps intermediaries from caching a long-lived stream. Matching SSE avoids "works in dev, bursty in prod" surprises when someone switches from text/event-stream to application/jsonl.

A 3-line mirror + a TestClient header assertion (like the existing SSE streaming tests) seems like the right fix. 👍 on the PR.

[torrresagus]

Thanks @fournarin, that's really useful — especially the production detail about the 200 landing immediately while the lines sit in the proxy buffer until it fills. That's the exact failure mode I was worried about, good to have it confirmed outside of local dev.

Agreed on including both headers to match SSE. The silent regression when someone switches text/event-stream → application/jsonl is the trap worth closing, and keeping the two branches symmetric is easier to reason about than explaining why only one of them sets Cache-Control.

I've got the fix ready locally: the 3-line mirror in the is_json_stream branch, plus a TestClient test asserting x-accel-buffering: no and cache-control: no-cache across the JSONL streaming variants (parametrized like the existing SSE streaming tests). The SSE + streaming suites still pass.

Happy to open a PR whenever a maintainer is good with the approach. 🙏

[fournarin]

Glad it helped! If my comment above confirmed your approach, would you mind Marking as answer on that reply? Thanks — and good luck with the PR! 🙏

[torrresagus]

Appreciate it @fournarin — and genuinely, thanks for taking the time to reproduce it on 0.137.x and to bring in the production angle. The detail about the 200 arriving immediately while the lines pile up in the proxy buffer is exactly the part that's easy to miss in local dev, where there's no Nginx in front to expose it. That's the kind of context that makes this worth fixing properly rather than hand-waving away as "works on my machine".

On the "mark as answer" request — I'm going to hold off on that for now, and I hope it doesn't come across the wrong way. The reason is just that this isn't really a Q&A thread where a comment is the resolution: it's a bug report with a proposed fix, and what it's actually waiting on is a maintainer being okay with the approach so the PR can go in. If I mark it answered now, it drops out of the unanswered queue that the team and the FastAPI Experts triage from, and I'd rather keep it visible until the fix is actually merged. Once the PR lands I'm happy to come back, link it here, and wrap the thread up properly.

None of that takes away from your comment being useful — it settled my open question (include both headers, matching SSE) and gave me a concrete real-world failure mode to point at, which is more than enough to move forward with confidence.

For what it's worth, the fix is small and ready on my end: the three-line mirror in the is_json_stream branch, plus a parametrized TestClient test asserting x-accel-buffering: no and cache-control: no-cache across the JSONL streaming variants, in the same style as the existing SSE streaming tests. The SSE and streaming suites stay green. So the moment a maintainer is good with it, it's a quick PR.

Thanks again for the careful look — and if a maintainer does swing by, your production confirmation here will probably help make the case. 🙏

[delcenjo]

Until the headers are added to the built-in JSONL branch, you can get the same proxy-safe behavior today by returning a StreamingResponse yourself and setting the headers explicitly:

from collections.abc import AsyncIterable

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel

app = FastAPI()


class Item(BaseModel):
    name: str


async def item_stream() -> AsyncIterable[str]:
    for item in (Item(name="a"), Item(name="b")):
        yield item.model_dump_json() + "\n"


@app.get("/jsonl-items")
async def jsonl_items():
    return StreamingResponse(
        item_stream(),
        media_type="application/x-ndjson",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no",  # stop Nginx (proxy_buffering on) from buffering
        },
    )

On the proposal itself: mirroring the SSE branch makes sense — both are incremental streams, so both should send X-Accel-Buffering: no, and adding Cache-Control: no-cache for parity with SSE is reasonable. A small PR with a test asserting the headers sounds welcome.

[torrresagus]

Thanks @delcenjo — that's a solid stopgap, and worth having in the thread for anyone hitting this before the fix lands. Dropping down to a manual StreamingResponse with the headers set explicitly is exactly what the built-in branch does internally, so it behaves the same behind a proxy. One small nuance for parity: the built-in JSONL branch uses application/jsonl as the media type rather than application/x-ndjson — either works for a hand-rolled endpoint, just worth noting.

And good to have another confirmation that mirroring the SSE branch is the right call. The fix is ready on my end (the 3-line mirror plus a header-asserting test), so it's a quick PR whenever a maintainer gives the go-ahead. 🙏

[torrresagus]

Opened the PR: #15813. It mirrors the SSE branch — sets Cache-Control: no-cache and X-Accel-Buffering: no on the JSONL StreamingResponse — plus a test asserting both headers across the JSONL streaming variants. Thanks @fournarin and @delcenjo for the confirmation and the production context. 🙏

[luzzodev]

Hello @torrresagus as another workaround you can use this approach for your example.

from collections.abc import AsyncIterable

from fastapi import FastAPI, Depends
from fastapi.sse import EventSourceResponse
from fastapi.responses import Response
from fastapi.testclient import TestClient
from pydantic import BaseModel


class Item(BaseModel):
    name: str


app = FastAPI()

async def live_jsonl_headers(response:Response) -> Response:
    response.headers["Cache-Control"] = "no-cache"
    response.headers["X-Accel-Buffering"] = "no"
    return response


@app.get("/sse-items", response_class=EventSourceResponse)
async def sse_items() -> AsyncIterable[Item]:
    # text/event-stream response (SSE)
    yield Item(name="a")


@app.get("/jsonl-items")
async def jsonl_items(response: Response = Depends(live_jsonl_headers)) -> AsyncIterable[Item]:
    # application/jsonl response (JSON Lines streaming)
    yield Item(name="a")


client = TestClient(app)

print("SSE  ->", dict(client.get("/sse-items").headers))
# SSE  -> {'content-type': 'text/event-stream; charset=utf-8', 'cache-control': 'no-cache', 'x-accel-buffering': 'no'}

print("JSONL->", dict(client.get("/jsonl-items").headers))
# JSONL-> {'content-type': 'application/jsonl', 'cache-control': 'no-cache', 'x-accel-buffering': 'no'

[torrresagus]

Quick update for anyone following this: after review feedback from @luzzodev on the PR (#15813), the approach changed slightly.

JSON Lines isn't only a live event stream like SSE — it's also used for bulk/streamed exports where caching can be legitimate. Since Cache-Control: no-cache forces revalidation rather than meaning "don't store", defaulting to it would quietly take away the max-age "serve from cache without hitting origin" option from those export endpoints. So that's a policy the framework shouldn't impose.

The PR now sets only X-Accel-Buffering: no (buffering defeats incremental streaming in every case), leaves caching headers up to the user, and documents the behavior in the JSON Lines tutorial.

Thanks @fournarin and @delcenjo for the earlier confirmation and production context, and @luzzodev for the review. 🙏