How to integrate FastAPI background from existing Django in order to alleviate large CSV download timeout?

[scheung38]

Original Django file to work with large excel files:

//
import tempfile
import zipfile

from django.http.response import StreamingHttpResponse
import openpyxl
import pyarrow
from rest_framework import permissions
from rest_framework import status
from rest_framework.decorators import api_view
from rest_framework.decorators import permission_classes
from rest_framework.response import Response
from wsgiref.util import FileWrapper

from main.tokens import MultiToken
from core import constants

logger = logging.getLogger(__name__)
SINGLE_OUTPUT_PER_FILE = False


@api_view(['GET'])
@permission_classes([permissions.AllowAny])
def work_file(request, key):

    // main business logic not shown but excel_file_filepath is generated

    with open(excel_file_filepath, 'rb') as f:
        wrapper = FileWrapper(io.BytesIO(f.read()))
    response = StreamingHttpResponse(wrapper, content_type='application/vnd.ms-excel', status=status.HTTP_200_OK)
    response['Content-Length'] = excel_file_filesize
    response['Content-Disposition'] = f'attachment; filename="{excel_file_filename}"'
    response['Cache-Control'] = 'no-cache'
    return response

/******************************************************/
// Would FastAPI background help in this case?

from fastapi import FastAPI, BackgroundTasks

app = FastAPI()

@app.get("/")
async def test(background_tasks: BackgroundTasks):
    background_tasks.add_task(work_file, request, key)
    return {'result': 'success'} 

If so how do we integrate FastAPI into existing Django project as I see there are a few config settings needed.

How do we tell Django not to call work_file directly but from FastAPI?

or simply rename work_file into fastapi_work_file and work_file so Django calls the same work_file function but delegates background with new function:

//
@api_view(['GET'])
@permission_classes([permissions.AllowAny])
def fastapi_work_file(request, key):
   etc


async def work_file(background_tasks: BackgroundTasks):
    background_tasks.add_task(fastapi_work_file, request, key)
    return {'result': 'success'} 

But this work_file changes to a different input arguments of background_tasks instead of the original request, key

or define BackgroundTasks locally inside FastAPI function?

//
async def work_file(request, key):
    from fastapi import BackgroundTasks
    background_tasks: BackgroundTasks
    background_tasks.add_task(fastapi_work_file, request, key)
    return {'result': 'success'} 

I imagine localhost:8000/docs to show endpoints from FastAPI on top of existing localhost:8000/swagger from Django without any conflict

Description

Environment

• OS: macOS Big Sur 11.2.3:
• FastAPI Version 0.65.1:

To know the FastAPI version use:

python -c "import fastapi; print(fastapi.__version__)"

• Python version:

To know the Python version use:

python --version

Python 3.9.5

Additional context

[Hultner]

I'm not familiar with Django but this seems like something you should be able to fix within Django.
Personally I wouldn't integrate a new framework just because this one problematic endpoint. What's causing the timeout to start with? You could probably increase that time affordance for this particular endpoint.

And I'm not really sure how you think this is going to work. You have an async FastAPI function calling a Django API route in the background after it's returned its response, where do you suppose the file should end up? And how should this resolve the problems you've had with Django?

There's a StreamingResponse in starlette.responses which I think might fill the same purpose as your original codes StreamingHttpResponse

[scheung38]

thanks @Hultner you mentioned to not use another framework, but StreamingHttpResponse is in starlette.responses which is not in Django, so just try and pip3 install starlette without fastapi?

[Hultner]

thanks @Hultner you mentioned to not use another framework, but StreamingHttpResponse is in starlette.responses which is not in Django, so just try and pip3 install starlette without fastapi?

I was referring to from django.http.response import StreamingHttpResponse which you use in your original code, I mentioned it since the Starlette version StreamingResponse is most likely what you would be looking for if you were to do this in FastAPI (or someone else reading this issue later). However I'd still say don't use two frameworks just because of this particular problem, especially if you have a somewhat significant codebase in Django already.

Now if you regardless would like to use them both in conjunction, for instance if you plan to migrate to FastAPI for newer things going forward then you could do that by mounting the WSGI-application in your FastAPI app using the WSGIMiddleware. You can read more about that under Including WSGI - Flask, Django, others in the documentation. Not sure if the other way around is possible, you'd have to consult with Django and/or your WSGI-server for that.

[scheung38]

Hi again @Hultner it is not just this, but within this particular Django project there are a number of timeout issues, not just this one, and also in Starlette there is also a FileResponse

from starlette.responses import FileResponse

as this is a download timeout, maybe slightly more appropriate than the StreamingResponse in this case? Yes I saw we need this WSGIMiddleware should both framework are needed.

Or simpler just to serve existing Django with ASGI server like uvicorn or Hypercorn and move existing functions into Django 3.1+ async functions?

[Hultner]

Hi again @Hultner it is not just this, but within this particular Django project there are a number of timeout issues, not just this one, and also in Starlette there is also a FileResponse

from starlette.responses import FileResponse

as this is a download timeout, maybe slightly more appropriate than the StreamingResponse in this case? Yes I saw we need this WSGIMiddleware should both framework are needed.

Or simpler just to serve existing Django with ASGI server like uvicorn or Hypercorn and move existing functions into Django 3.1+ async functions?

I’m not familiar enough with Django to answer that. About the timeouts, it must be possible to configure django, or the server you’re serving the django content with to allow for longer timeouts.

As for FileResponse that’s for serving static file directly, not for ones generated dynamically in runtime. If you are serving static files you should probably use a static file server like nginx or something like WhiteNoise if you want to keep it in Python.

[ricky-sb]

@scheung38 you have a few options here:

• You can keep your existing integration and just increase the Gunicorn timeout to an insane level
• Since you only want to use the Task Queue feature of FastAPI, you can use Celery, which is very common to use with Django for this use-case. @zander9648 has a great tutorial on Celery here.

In my opinion, FastAPI isn't the right choice for your use-case.