GZip Middleware does not compress Streaming Response

[maramizo]

First Check

• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from fastapi.middleware.gzip import GZipMiddleware
from starlette.middleware.cors import CORSMiddleware


app = FastAPI()
app.add_middleware(GZipMiddleware)


csv_file(path: str):
    with open(path) as f:
        yield f.readline()

@app.get("/receivings")
def receivings(start: str = start_date, end: str = now) -> StreamingResponse:
    return StreamingResponse(csv_file(path), media_type='text/csv')



app.add_middleware(
    CORSMiddleware,
    allow_origins=["https://www.website.com"],
    allow_methods=["POST"],
    allow_headers=["*"]
)


if __name__ == '__main__':
    uvicorn.run("middleserver:app", host="0.0.0.0", port=8080, log_level='trace')

Description

• Create a regular app with an endpoint that returns a StreamingResponse
• Have GZip as a middleware
• Result is not compressed even though it should be
• Try again with FileResponse
• Result is now compressed properly

Operating System

Windows

Operating System Details

No response

FastAPI Version

0.75

Python Version

3.10.3

Additional Context

No response

[jonatasoli]

You try add middleware after your receivings function?

[maramizo]

Yes, made no difference. I've abandoned this entire method of responses as a result.

[JarroVGIT]

I played around a bit with this, and found a couple of quirks.

First of, your generator that you feed as content in your StreamingResponse would be my implementation as well. However, when I test it out, it will only yield once and I get an incomplete response. That response is small and when that happens, the GZIP middleware doesn't perform compression (expected behaviour).

However, the Starlette documentation on StreamingResponse states:

Have in mind that file-like objects (like those created by open()) are normal iterators. So, you can return them directly in a StreamingResponse.

And, when I change your code to the following:

@app.get("/receivings")
def receivings() -> StreamingResponse:
    file = open("filepath/1mb.txt")
    return StreamingResponse(file, media_type='text/csv')

it does work (meaning, I get the proper headers and the response is gzipped). However, I never close the file explicitly and that worries me. When wrapping the response into a context manager like so:

with open(path) as f: 
    return Streamingresponse(f, media_type='text/csv')

raises a ValueError: ValueError: I/O operation on closed file.

Strange behaviour, and I cannot put my finger on it yet. I would expect we would need to close the file-handler ourselves, but trying to do so gives the ValueError.

[xingdongzhe]

Hey guys! I read the code that GZipMiddleware only works when client has set header:Accept-Encoding:gizp and length of response data greater than 500 by default

I would expect we would need to close the file-handler ourselves

@JarroVGIT That will be great. But only user pass below code can do it

def csv_file(path: str):
    with open(path) as f:
        for line in f:
            yield line

[JarroVGIT]

Of course, that is a neat solution!

[xalteropsx]

yah this streaming response doing no compress at split file either .... i tried brotli and gzip both fail

[Kludex]

There's nothing FastAPI can do about it. This is entirely Starlette.

Feel free to create a discussion over there.