Strategies for limiting upload file size

[abdusco]

Description

I'm trying to create an upload endpoint. I want to limit the maximum size that can be uploaded.

My endpoint looks like this:

@app.post('/upload', response_model=UploadedFileDTO)
async def upload_file(file: UploadFile = File(...), db: Session = Depends(get_db_session)):
    save_path = local.generate_path(file.filename)
    with file.file as f:
        local.save(stream=f, save_path=save_path)

    u = Upload(filename=file.filename,
               path=str(save_path))
    db.add(u)
    db.commit()

    return u

I checked out the source for fastapi.params.File, but it doesn't seem to add anything over fastapi.params.Form.

The only solution that came to my mind is to start saving the uploaded file in chunks, and when the read size exceeds the limit, raise an exception. But I'm wondering if there are any idiomatic ways of handling such scenarios?

[abdusco]

Ok, I've found an acceptable solution. But it relies on Content-Length header being present.

Edit: I've added a check to reject requests without Content-Length

from starlette import status
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.requests import Request
from starlette.responses import Response
from starlette.types import ASGIApp


class LimitUploadSize(BaseHTTPMiddleware):
    def __init__(self, app: ASGIApp, max_upload_size: int) -> None:
        super().__init__(app)
        self.max_upload_size = max_upload_size

    async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
        if request.method == 'POST':
            if 'content-length' not in request.headers:
                return Response(status_code=status.HTTP_411_LENGTH_REQUIRED)
            content_length = int(request.headers['content-length'])
            if content_length > self.max_upload_size:
                return Response(status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE)
        return await call_next(request)

using it is quite straightforward:

app = FastAPI()
app.add_middleware(LimitUploadSize, max_upload_size=50_000_000)  # ~50MB

The server sends HTTP 413 response when the upload size is too large, but I'm not sure how to handle if there's no Content-Length header.
Edit: Solution: Send 411 response

[sm-Fifteen]

The server sends HTTP 413 response when the upload size is too large, but I'm not sure how to handle if there's no Content-Length header.

You can reply HTTP 411 if Content-Length is absent.

[ac3d912]

@tiangolo This would be a great addition to the base package

[dmontagu]

For what it's worth, both nginx and traefik have lots of functionality related to request buffering and limiting maximum request size, so you shouldn't need to handle this via FastAPI in production, if that's the concern.

[erosennin]

You can use an ASGI middleware to limit the body size.

Example: https://github.com/steinnes/content-size-limit-asgi

[tiangolo]

Thanks everyone for the discussion here!

So, here's the thing, a file is not completely sent to the server and received by your FastAPI app before the code in the path operation starts to execute.

So, you don't really have an actual way of knowing the actual size of the file before reading it.

---

You could require the Content-Length header and check it and make sure that it's a valid value. E.g.

from fastapi import FastAPI, File, Header, Depends, UploadFile


async def valid_content_length(content_length: int = Header(..., lt=50_000_000)):
    return content_length


app = FastAPI()

@app.post('/upload', dependencies=[Depends(valid_content_length)])
async def upload_file(file: UploadFile = File(...)):
    # do something with file
    return {"ok": True}

And then you could re-use that valid_content_length dependency in other places if you need to.

⚠️ but it probably won't prevent an attacker from sending a valid Content-Length header and a body bigger than what your app can take ⚠️

---

Another option would be to, on top of the header, read the data in chunks. And once it's bigger than a certain size, throw an error.

E.g.

from typing import IO

from tempfile import NamedTemporaryFile
import shutil

from fastapi import FastAPI, File, Header, Depends, UploadFile, HTTPException
from starlette import status


async def valid_content_length(content_length: int = Header(..., lt=80_000)):
    return content_length


app = FastAPI()


@app.post("/upload")
def upload_file(
    file: UploadFile = File(...), file_size: int = Depends(valid_content_length)
):
    real_file_size = 0
    temp: IO = NamedTemporaryFile(delete=False)
    for chunk in file.file:
        real_file_size += len(chunk)
        if real_file_size > file_size:
            raise HTTPException(
                status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE, detail="Too large"
            )
        temp.write(chunk)
    temp.close()
    shutil.move(temp.name, "/tmp/some_final_destiny_file")
    return {"ok": True}

[github-actions]

Assuming the original issue was solved, it will be automatically closed now. But feel free to add more comments or create new issues.

[jd-solanki]

@tiangolo What is the equivalent code of your above code snippet using aiofiles package?

I noticed there is aiofiles.tempfile.TemporaryFile but I don't know how to use it.

Regards.

[engineervix]

@tiangolo What is the equivalent code of your above code snippet using aiofiles package?

I noticed there is aiofiles.tempfile.TemporaryFile but I don't know how to use it.

Regards.

Hey @jd-0001, this is how I did it ...

import aiofiles
from fastapi import FastAPI, File, Header, Depends, UploadFile, HTTPException
from starlette import status


async def valid_content_length(content_length: int = Header(..., lt=80_000)):
    return content_length


app = FastAPI()


@app.post("/upload")
async def upload_file(
    file: UploadFile = File(...), file_size: int = Depends(valid_content_length)
):
    output_file = f"/path/to/{file.filename}"
    real_file_size = 0

    try:
        async with aiofiles.open(f"{output_file}", "wb") as out_file:
            while content := await file.read(1024):  # async read chunk
                real_file_size += len(content)
                if real_file_size > file_size:
                    raise HTTPException(
                        status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE,
                        detail="Too large",
                    )
                await out_file.write(content)  # async write chunk
        msg = f"Successfuly uploaded {file.filename} for processing"
    except IOError:
        msg = "There was an error uploading your file"

    return {"message": msg}

[jd-solanki]

Thanks @engineervix I will try it for sure and will let you know.

[johnPractice]

for the check file size in bytes, you can use

size=file.file._file.getbuffer().nbytes

and then set condition for check size

[bravmi]

#362 (comment)
Great stuff, but somehow content-length shows up in swagger as a required param, is there any way to get rid of that? :)
Tested with python 3.10 and fastapi 0.82

[paulovitorweb]

Thanks for everyone's valuable tips here.

This is how I did it, trying to avoid NamedTemporaryFile as an intermediary.

file_destination = Path(destination)

try:
    # Writes file to destination while validating file size
    real_file_size = 0
    with file_destination.open("wb") as buffer:
        for chunk in upload_file.file:
            real_file_size += len(chunk)
            if real_file_size > max_fize_size:
                raise HTTPException(
                    status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE, detail="Too large"
                )
            buffer.write(chunk)
except HTTPException:
    os.unlink(file_destination)
    raise
finally:
    upload_file.file.close()

[em1208]

This is still an issue with the latest version of FastAPI.

from fastapi import FastAPI, status, UploadFile
from starlette.responses import Response
from starlette.types import ASGIApp
from starlette.requests import Request
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from typing import List
from enum import Enum


class AudioFileTypeName(str, Enum):
    mp3 = "audio/mp3"
    mpeg = "audio/mpeg"
    ogg = "audio/ogg"
    wave = "audio/wave"
    wav = "audio/wav"

    @classmethod
    def list(cls):
        return list(map(lambda c: c.value, cls))


class ValidateUploadFileMiddleware(BaseHTTPMiddleware):
    def __init__(
        self,
        app: ASGIApp, max_size: int = 1048576, # 1MB
        file_types: List[str] = AudioFileTypeName.list()
    ) -> None:
        super().__init__(app)
        self.max_size = max_size
        self.file_types = file_types

    async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
        if request.method == 'POST':
            form = await request.form()
            content_type = form[next(iter(form))].content_type
            if content_type not in self.file_types:
                return Response(status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE)
            if 'content-length' not in request.headers:
                return Response(status_code=status.HTTP_411_LENGTH_REQUIRED)
            content_length = int(request.headers['content-length'])
            if content_length > self.max_size:
                return Response(status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE)
        return await call_next(request)


app = FastAPI()

# File size upload middleware does not work because of a bug in starlette. This code is broken https://github.com/tiangolo/fastapi/issues/362
app.add_middleware(
    ValidateUploadFileMiddleware
)


@app.post("/upload")
def upload(file: UploadFile):
    return {'success': True}


@app.get("/health")
def health():
    return Response(status_code=status.HTTP_200_OK)

I believe this should be fixed since it is a very common requirement for APIs.

[dmontagu]

@em1208 As noted above, you need to make sure to check the size of the bytes you are receiving as you receive them. This is possible to do following the approach in either of these comments:

• Strategies for limiting upload file size #362 (comment)
• Strategies for limiting upload file size #362 (comment)

If you want this to be handled more automatically for you, I think it might be possible by subclassing starlette.datastructures.UploadFile and overriding the read method, and then making it possible to get your overridden type used by FastAPI (may require FastAPI changes, not sure). This might also be a reasonable feature request for starlette, you might ask (or search around; it's possible others have already requested this) there.

[em1208]

@dmontagu I think the middleware should be the best approach for most cases but using dependencies should also be supported in case of different limits for different views. Do you agree that the middleware approach should be supported and needs to be fixed?

[dmontagu]

I wouldn't use the word "fixed" (in "it needs to be fixed") because I think this is a feature request rather than a bug. But I do think it's a reasonable feature request.

That said, a quick google for "asgi middleware file size" found me this: https://github.com/steinnes/content-size-limit-asgi. I haven't tried it at all but it looks like it is an ASGI middleware (so I think should work with FastAPI/starlette) and appears to implement content length validation.

Edit: They specifically include an example using starlette in the README.md, so I'm pretty sure this should work for your use case.

[em1208]

@dmontagu thanks for pointing out this solution https://github.com/steinnes/content-size-limit-asgi, I verified that it works. Still I believe there is an issue in the documentation because the approach #362 (comment) should work but unfortunately it does not. I would be happy to work on this and open a PR once we agree on the approach to solve this problem / feature request.