FastAPI performance hit?

[mavwolverine]

core running on docker

Created 2 hello world FastAPI apps, one vanilla, other with supertokens.

from fastapi import FastAPI
from starlette.middleware.cors import CORSMiddleware

from supertokens_python import (
    InputAppInfo,
    SupertokensConfig,
    init,
)
from supertokens_python.recipe import (
    dashboard,
    emailverification,
    multitenancy,
    session,
    usermetadata,
    userroles,
    emailpassword,
)
from supertokens_python import (
    get_all_cors_headers,
)
from supertokens_python.framework.fastapi import get_middleware


def create_app():
    app = FastAPI()
    app.add_middleware(
        CORSMiddleware,
        allow_origins=[
            "http://localhost:3000",
            "http://localhost:8000",
        ],
        allow_credentials=True,
        allow_methods=["GET", "PUT", "POST", "DELETE", "OPTIONS", "PATCH"],
    )

    @app.get("/")
    def read_root():
        return {"Hello": "World"}

    # TODO: start server

    return app


def create_app_supertokens():
    init(
        app_info=InputAppInfo(
            app_name="SuperTokens test",
            api_domain="http://localhost:8000/",
            website_domain="http://localhost:3000/",
            api_base_path="/auth",
            website_base_path="/auth",
        ),
        supertokens_config=SupertokensConfig(
            # https://try.supertokens.com is for demo purposes. Replace this with the
            # address of your core instance (sign up on supertokens.com), or self host
            # a core.
            # connection_uri="https://try.supertokens.com",
            connection_uri="http://localhost:3567",
            # api_key=<API_KEY(if configured)>
        ),
        framework="fastapi",
        recipe_list=[
            emailpassword.init(),
            session.init(),  # initializes session features
            dashboard.init(),
            emailverification.init(mode="REQUIRED"),
            userroles.init(),
            usermetadata.init(),
            multitenancy.init(),
        ],
        mode="asgi",  # use wsgi if you are running using gunicorn
    )

    app = FastAPI()
    app.add_middleware(get_middleware())

    app.add_middleware(
        CORSMiddleware,
        allow_origins=[
            "http://localhost:3000",
            "http://localhost:8000",
        ],
        allow_credentials=True,
        allow_methods=["GET", "PUT", "POST", "DELETE", "OPTIONS", "PATCH"],
        allow_headers=["Content-Type"] + get_all_cors_headers(),
    )

    @app.get("/")
    def read_root():
        return {"Hello": "World"}

    # TODO: start server

    return app

benchmark for vanilla app:

❯ wrk -t12 -c400 -d30s http://localhost:8008
Running 30s test @ http://localhost:8008
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    36.99ms    7.54ms 210.71ms   80.16%
    Req/Sec     0.90k    69.06     1.05k    67.42%
  321465 requests in 30.07s, 43.53MB read
  Socket errors: connect 0, read 1092, write 12, timeout 0
Requests/sec:  10690.42
Transfer/sec:      1.45MB

benchmark with supertokens middleware

❯ wrk -t12 -c400 -d30s http://localhost:8009
Running 30s test @ http://localhost:8009
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    86.59ms   20.83ms 499.97ms   94.85%
    Req/Sec   382.60     45.98   565.00     75.47%
  137462 requests in 30.08s, 18.62MB read
  Socket errors: connect 0, read 1251, write 1, timeout 0
Requests/sec:   4569.42
Transfer/sec:    633.70KB

Is this big difference expected?

Thanks

[mavwolverine]

It is a manifestation of the starlette BaseHTTPMiddleware.
https://github.com/supertokens/supertokens-python/blob/master/supertokens_python/framework/fastapi/fastapi_middleware.py#L28

See fastapi/fastapi#6985 (comment)

[mavwolverine]

Ran a simple test

from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.requests import Request

from starlette.types import ASGIApp, Receive, Scope, Send


class AsgiMiddleware:
    def __init__(self, app: ASGIApp) -> None:
        self.app = app

    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
        await self.app(scope, receive, send)


class StarletteHTTPMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next: RequestResponseEndpoint):
        return await call_next(request)

With app.add_middleware(AsgiMiddleware)

With app.add_middleware(StarletteHTTPMiddleware)

[mavwolverine]

Tried to create a asgi middleware replacement for supertokens fastapi middleware. (its very late for me, sorry if I messed up something.)

from typing import Any, Dict, Union

from starlette.requests import Request
from starlette.responses import Response
from starlette.types import ASGIApp, Message, Receive, Scope, Send
from supertokens_python import Supertokens
from supertokens_python.exceptions import SuperTokensError
from supertokens_python.framework import BaseRequest, BaseResponse
from supertokens_python.framework.fastapi.fastapi_request import (
    FastApiRequest,
)
from supertokens_python.framework.fastapi.fastapi_response import (
    FastApiResponse,
)
from supertokens_python.recipe.session import SessionContainer
from supertokens_python.supertokens import manage_session_post_response
from supertokens_python.utils import default_user_context


def get_middleware():
    class AsgiMiddleware:
        def __init__(self, app: ASGIApp) -> None:
            self.app = app

        async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
            if scope["type"] != "http":
                await self.app(scope, receive, send)
                return

            async def _manage_session_post_response(
                request: BaseRequest, result: BaseResponse, user_context: Dict[str, Any]
            ):
                if hasattr(request.state, "supertokens") and isinstance(
                    request.state.supertokens, SessionContainer
                ):
                    manage_session_post_response(
                        request.state.supertokens, result, user_context
                    )

            async def send_wrapper(message: Message) -> None:
                await send(message)

                if message["type"] == "http.response.start":
                    response.response.status_code = message["status"]
                    if "headers" in message:
                        response.response.init_headers(
                            {
                                k.decode("utf-8"): v.decode("utf-8")
                                for k, v in message["headers"]
                            }
                        )
                elif message["type"] == "http.response.body":
                    response_body += message["body"]
                    more_body = message.get("more_body", False)

                    # If the body is not complete yet, return to wait
                    if more_body:
                        return

                    response.response.body = response.response.render(
                        response_body.decode("utf-8")
                    )

                    await _manage_session_post_response(request, response, user_context)
                return

            st = Supertokens.get_instance()

            request = Request(scope)
            custom_request = FastApiRequest(request)
            user_context = default_user_context(custom_request)

            try:
                response = FastApiResponse(Response())
                result: Union[BaseResponse, None] = await st.middleware(
                    custom_request, response, user_context
                )
                if result is None:
                    response = FastApiResponse(Response())
                    response_body = b""
                    await self.app(scope, receive, send_wrapper)
                    return
                else:
                    await _manage_session_post_response(request, result, user_context)

                    if isinstance(result, FastApiResponse):
                        await result.response(scope, receive, send)
                        return

                return
            except SuperTokensError as e:
                response = FastApiResponse(Response())
                error_result: Union[BaseResponse, None] = await st.handle_supertokens_error(
                    FastApiRequest(request), e, response, user_context
                )
                if isinstance(error_result, FastApiResponse):
                    await error_result.response(scope, receive, send)
                    return

            raise Exception("Should never come here")

    return AsgiMiddleware

[rishabhpoddar]

Thanks for opening this issue @virajkanwade . We will have a look at this in a few week's time

[mavwolverine]

https://medium.com/@robbe.sneyders/redesigning-connexion-as-an-asgi-middleware-stack-a5dc17e81ff8

might help on some of the nuances of asgi middleware

[sattvikc]

hi @virajkanwade ,

From the stats you have mentioned:

Without Supertokens middleware, each request takes about 0.09 ms
and with Supertokens middleware, each request takes about 0.21 ms

This means that the supertokens adds an overhead of about 0.12 ms per request. Which means, 1.2 sec loss on 10000 requests. Does that make a huge difference?

How does your test fare if you had a db call in the API?

[mavwolverine]

@sattvikc you are looking at it from response time side.

I am looking at it from infrastructure side.

FastAPI by default can handle around 10,000 req/sec.

To support 100,000 req/sec, I need to run 10-11 FastAPI servers.

With the supertokens middleware (using Starlette BaseHTTPMiddleware), it goes to around 4,800 req/sec.
To support 100,000 req/sec, I would need to run 20-21 servers.

With the supertokens middleware reimplemented as ASGI middleware (#503 (comment)), it could support little over 8,200 req/sec.
To support 100,000 req/sec, I would need to run 13-14 servers.

Now scale that up to 200,000 req/sec, 500,000 req/sec.

Its about optimizing performance.

[mavwolverine]

@sattvikc Also do look at Kludex/starlette#2160 (comment)

I’m warming up to the idea of not removing it for 1.0. I still think we should discourage its use, but maybe we can keep it around until 2.0 to ease the burden for migrating to 1.0? Does something like this even make sense or if we’re going to discourage it should we just remove it?

[rishabhpoddar]

This has been implemented and released in python SDK version >= 0.20.2. Thanks @virajkanwade .

Feel free to add yourself to the supertokens-core repo readme as a contributor :)