Slowdown of cold start in connexion 3.0

[ThomasSchwengler]

Description

When switching to connexion 3.0 we noticed a considerable slowdown at startup.
Startup with connexion 3.x: 700ms-1s
Startup with connexion 2.x: 1ms-10ms

Expected behaviour

No slowdown at startup compared to connexion 2.x

Actual behaviour

Slowdown at startup.

Steps to reproduce

• Have a medium sized OpenApi specification in yaml format (700 lines)
• Have a simple app created with a factory method:

  # openapi_server/factory.py
  
  from connexion import App
  
  
  def create_app() -> App:
      connexion_app = App(__name__, specification_dir="./openapi/")
      connexion_app.add_api("openapi.yaml")
      return connexion_app

• Use pytest and have a simple dummy unit test:
  For connexion 3.x:

  # tests/test_dummy.py
  
  import pytest
  from openapi_server.factory import create_app
  
  @pytest.fixture()
  def app():
    return create_app().test_client()
  
  @pytest.mark.parametrize("arg", range(10))
  def test_startup(app, arg):
    app.get("/")

  For connexion 2.x:

  # tests/test_dummy.py
  
  import pytest
  import webtest
  
  from openapi_server.factory import create_app
  
  
  @pytest.fixture()
  def app():
      return webtest.TestApp(create_app())
  
  
  @pytest.mark.parametrize("arg", range(10))
  def test_startup(app, arg):
    app.get("/")

• Timings:
  • connexion 3.x: pytest tests/test_dummy.py 8,97s user 0,11s system 98% cpu 9,188 total
  • connexion 2.x: pytest tests/test_dummy.py 2,36s user 0,10s system 85% cpu 2,888 total

Additional info:

Output of the commands:

• python --version
  Python 3.9.18
• pip show connexion | grep "^Version\:"
  Version: 3.0.1

What I tried:

• Use AsyncApp (no speedup)
• Use different backends for starlette (no speedup)
• Use a different python version (some speedup)

Looking at the profiler it seems like the yaml parsing now takes more time?

[RobbeSneyders]

Thanks for the report @ThomasSchwengler

You mention

Startup with connexion 3.x: 700ms-1s
Startup with connexion 2.x: 1ms-10ms

But the difference seems a lot smaller in your tests

Timings:

• connexion 3.x: pytest tests/test_dummy.py 1,45s user 0,10s system 99% cpu 1,554 total
• connexion 2.x: pytest tests/test_dummy.py 1,01s user 0,11s system 97% cpu 1,141 total

Where do the top numbers come from?

---

Looking at the profiler it seems like the yaml parsing now takes more time?

Could you post some data from the profiler?

[ThomasSchwengler]

@RobbeSneyders thanks for the fast reply,

I updated the examples to run the tests multiple times to see the difference.
Before that I guess the startup of pytest interfered too much.

[ThomasSchwengler]

Regarding the profiler data:
For connexion 3.x: https://cloud.dilt.at/s/tSjNRBcgTbjHnXf
For connexion 2.x: https://cloud.dilt.at/s/KCdTxwkDbqLw8Q9

[ThomasSchwengler]

Where do the top numbers come from?

Those are the timings Pycharm tells me, which probably don't include the pytest framework starting up.

[RobbeSneyders]

I'm indeed able to reproduce a longer startup time in Connexion 3 compared to Connexion 2 using the following script. You can add it to the restyresolver example and install the connexion version you want to test.

import time
from multiprocessing import Process

import requests
from connexion import App
from connexion.resolver import RestyResolver


def start_app():
    app = App(__name__, specification_dir="spec")
    app.add_api(
        "openapi.yaml",
        arguments={"title": "RestyResolver Example"},
        resolver=RestyResolver("api"),
    )
    app.run(port=8080)


def wait_for_app():
    while True:
        try:
            r = requests.get("http://localhost:8080/openapi/pets")
        except Exception:
            pass
        else:
            if r.status_code == 200:
                break


if __name__ == '__main__':
    p = Process(target=start_app)
    start = time.time()
    p.start()
    wait_for_app()
    print(time.time() - start)
    p.kill()

Parsing the specification and passing it to the application has a bigger impact on Connexion 3 than Connexion 2, but can't completely explain the slowdown. I'll have a deeper look when I find the time. I can't really make sense of the v3 pstat file, since almost all time is spent on thread.lock and there's a lot of pytest stuff in there.

FYI, we did test the response time of Connexion 3 vs Connexion 2 before the release and didn't notice any meaningful difference between the two.

[ThomasSchwengler]

@RobbeSneyders thank you very much for looking into this, I will try to investigate myself in the meantime :)

Regarding response time I also didn't notice any slowdown.

[vladislavkoz]

I also, faced performance issues after the upgrade. Will share some profiling results tomorrow.
But what I can see is that my laptop is getting hot with Connexion 3 when I run tests. Tests execution time increased dramatically.

[vladislavkoz]

Here is an output from the profiler for the same file with tests.

Python 3.10.7
Connexion[flask, uvicorn]==3.0.1
Flask==3.0.0

Python 3.10.7
Connexion==2.14.1
Flask==2.2.5

[yevheniimykytiuk]

I have same issue

[RobbeSneyders]

Thanks for the reports everyone, I just submitted a fix. Will release it as a 3.0.3 once it's merged.

[vladislavkoz]

3.0.3

It's awesome. Thank you very much.

[xcelerit-team]

I'm glad to see this - can't wait for the release. Thank you very much. Currently working on an app with >300 API tests (with testclient), developing with TDD. The time for a full test run has become almost unbearable (>15min)...

[vladislavkoz]

I'm glad to see this - can't wait for the release. Thank you very much. Currently working on an app with >300 API tests (with testclient), developing with TDD. The time for a full test run has become almost unbearable (>15min)...

Same)

[vladislavkoz]

So 3.0.3 now released. It looks much better now but still not ideal in comparison to Connexion2.
Our tests time with Connexion2 - 2m
Connexion3.0.1 - 10m+++(Close to 15minutes)
Connexion3.0.3 - 4m 30s

[VojtaOrgon]

Hi not a good solution, but tip for TDD people. I have same problems where my tests execution jumped too high 2x - 3x even.

Ultimately I was able to make my final ASGI app stateless adding my own middleware to handle the initial state (using lifespan protocol). And that allowed me to create app just once for test session and reuse it for all tests. Cutting my test run time in half compared to connexion 2.x. It also forced my to much nicer solution for a server state than before.