Large StreamingResponse locks up server

[alex-r-bigelow]

Description

I'm following the example @tiangolo suggested in #58 with StreamingResponse. In my case, I'm attempting to stream slices of an intervaltree in a chunked way, but the server locks up when the request is big.

Example
This kind of thing works just fine for small requests, even when db is large:

@app.get('/datasets/{label}/intervals')
def intervals(label: str, begin: float, end):
    async def intervalGenerator():
        yield '['
        firstItem = True
        for r in db[begin:end]:  # db is an IntervalTree
            if not firstItem:
                yield ','
            yield json.dumps(db[label]['intervals'][r.data])
            firstItem = False
        yield ']'
    return StreamingResponse(intervalGenerator(), media_type='application/json')

The resulting JSON is consumed on the client side by oboe.js—a thing built to handle chunked JSON on the fly that has some mechanisms to allow for rendering+discarding chunks in the browser in a way that we don't have to keep the whole response in memory.

All goes well for small-ish requests, but when larger slices are requested, the whole server locks up (Control-C doesn't work; I have to kill -9 to get the thing to stop).

Main question
I'm having trouble debugging where the problem is (IntervalTree, starlette, FastAPI, or maybe the way my generator is written), but I think the problem is that the server is trying to assemble all of the data in memory somewhere before it sends any chunked results—is FastAPI or starlette doing anything like that internally (if not, this might be an IntervalTree problem)? Any clues how to avoid it, or obvious things I'm doing wrong?

[tiangolo]

@alex-r-bigelow you are using json.dumps to yield inside an async function. json.dumps is blocking, but as your function is async, it is running in the main loop, so it might be blocking your app.

Don't use async, use a normal function that returns a normal iterator.

In the latest versions, StreamingResponse can take a normal generator (not only an async generator).

That might solve your problem.

[alex-r-bigelow]

Thanks! Updating to the latest version and using a normal generator seems to fix the main problem (at least the server locking part—for anyone reaching this via Google, it looks like intervaltree still has a memory issue in that it collects + returns all results instead of iterating over them, but the server doesn't lock up anymore).

[tiangolo]

Thanks for reporting back and closing the issue!