Using UploadFile together with BackgroundTasks results in unexpected behavior because the file is closed

[msehnout]

First Check

• I added a very descriptive title here.
• I used the GitHub search to find a similar question and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import asyncio

from fastapi import FastAPI, UploadFile, BackgroundTasks

app = FastAPI()


async def do_sth_with_the_file(file: UploadFile):
    await asyncio.sleep(1)
    print(f"file is {'closed' if file.file.closed else 'opened'}")


@app.post("/upload-file")
async def create_upload_file(file: UploadFile, background_tasks: BackgroundTasks):
    background_tasks.add_task(do_sth_with_the_file, file=file)
    return {"filename": file.filename}

Description

As discussed in the issue #10857. The file used to be open when executing the do_sth_with_the_file function. But it gets closed in recent versions of FastAPI.

I created a workaround using copy.deepcopy:

import asyncio
import copy

from fastapi import FastAPI, UploadFile, BackgroundTasks

app = FastAPI()


async def do_sth_with_the_file(file: UploadFile):
    try:
        await asyncio.sleep(1)
        print(f"file is {'closed' if file.file.closed else 'opened'}")
    finally:
        file.file.close()


@app.post("/upload-file")
async def create_upload_file(file: UploadFile, background_tasks: BackgroundTasks):
    background_tasks.add_task(do_sth_with_the_file, file=copy.deepcopy(file))
    return {"filename": file.filename}

As a friend of mine pointed out, maybe it is not a good idea to process the file in background at all. But anyway, the result was a bit surprising to me. Maybe it is worth mentioning in the documentation?

I'd like to help anyone who would stumble upon the same issue. But I don't know if coping is the way to go, or doing something completely different.

Operating System

Linux

Operating System Details

No response

FastAPI Version

0.108.0

Pydantic Version

2.5.3

Python Version

Python 3.12.1

Additional Context

#10857

[alrooney]

Agree that it would be useful and sometimes necessary to be able to use an UploadFile in a background task. For me it is necessary as customers sometimes upload large files that take a long time to process so cannot hold up response waiting for that processing to finish. Also would like to avoid deep copying the file.

[martinkozle]

What I did eventually for my case was to not use UploadFile but instead read it's bytes and take it's file name and send that to the background task function, you could even wrap it back into a io.BinaryIO if a "file like" object is required.
This is definitely not optimal though if the file is large and you don't want to fully read it in memory but instead just send a pointer to the file on disk.

But what I am guessing is that FastAPI could only guarantee that this tempfile.SpooledTemporaryFile is still located on disk while the request connection is still open, after a response is returned it can't guarantee that the resources would still exist. Because if FastAPI wasn't responsible for clearing this temporary file then who should be? How can it know that someone else is still using that resource? It can lead to a sort of "storage leak" where the disk would just fill up.

I think FastAPI allowing this to happen pre v0.106.0 should be considered a bug?

Maybe there should be an explicit method on UploadFile to be able to take ownership of the underlying resource, and that would maybe mark the object that the resource is still in use and that FastAPI shouldn't delete the temporary file resource. Maybe even a sort of reference counter so that FastAPI is still eventually the one that deletes it.

I am no expert in this and I could very well be misunderstanding how FastAPI actually handles uploaded files. So sorry for any mistakes, but I hope it leads to more discussions.

[Scott-Simmons]

I'd like to help anyone who would stumble upon the same issue.

I have run into the same issue. Thank you for this workaround.

I think that the documentation on the behavior w.r.t. UploadFile and Background task interactions could be clearer. I would be happy to help, but I may also be misunderstanding the behavior here. Regardless if the behavior is good or not, I think this behavior should be documented somewhere.

In-memory file manipulation + background tasks seems like it would be a fairly common pattern?

As a friend of mine pointed out, maybe it is not a good idea to process the file in background at all.

What is the reason given for your friend saying this?