Very slow client initialization times

[Samrose-Ahmed]

Describe the bug
Initializing a client in aiobotocore on my laptop takes on the order of ~200ms.

Initializing a client in botocore takes ~50ms.

Checklist

• I have reproduced in environment where pip check passes without errors
• I have provided pip freeze results
• I have provided sample code or detailed way to reproduce
• I have tried the same code in botocore to ensure this is an aiobotocore specific issue
• I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
• I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

Reproduction

import time, asyncio, botocore, aiobotocore
import botocore.session
import aiobotocore.session

event_loop = asyncio.new_event_loop()
asyncio.set_event_loop(event_loop)

async def aiobotocore_client():
    async with aiobotocore.session.get_session().create_client("s3"):
        pass

def main():
    print("===================== aiobotocore ===============")
    st = time.perf_counter()
    event_loop.run_until_complete(aiobotocore_client())
    elapsed = time.perf_counter() - st
    print(f"took: {elapsed} secs")

    print("===================== botocore ==================")
    st = time.perf_counter()
    botocore.session.get_session().create_client("s3")
    elapsed = time.perf_counter() - st
    print(f"took: {elapsed} secs")


if __name__ == "__main__":
    main()

pip freeze results

aiobotocore==2.4.0
aiohttp==3.8.3
aioitertools==0.11.0
aiosignal==1.2.0
async-timeout==4.0.2
attrs==22.1.0
botocore==1.27.59
charset-normalizer==2.1.1
frozenlist==1.3.1
idna==3.4
jmespath==1.0.1
multidict==6.0.2
python-dateutil==2.8.2
six==1.16.0
urllib3==1.26.12
wrapt==1.14.1
yarl==1.8.1

Environment:

• Python Version: Python 3.10.4
• OS name and version: Ubuntu 22

Additional context
A quick profile shows the bulk of time loading SSL certs multiple times?

[thehesiod]

this must be something with your machine, here's my test case:

import asyncio
import aiobotocore.session
import botocore.session
import time


async def _boto_client():
    client = botocore.session.get_session().create_client('s3')
    client.close()


async def _aioboto_client():
    async with aiobotocore.session.AioSession().create_client('s3'):
        pass


async def main():
    reps = 200

    # warm-up
    await _boto_client()
    await _aioboto_client()

    start = time.time()
    for _ in range(reps):
        await _boto_client()
    elapsed = time.time() - start

    print(f'boto elapsed: {elapsed} per call: {elapsed/reps}')

    start = time.time()
    for _ in range(reps):
        await _aioboto_client()
    elapsed = time.time() - start

    print(f'aioboto elapsed: {elapsed} per call: {elapsed/reps}')


if __name__ == '__main__':
    asyncio.run(main())

results

boto elapsed: 25.254255771636963 per call: 0.12627127885818482
aioboto elapsed: 28.21110963821411 per call: 0.14105554819107055

fundamentally the only thing different between the two is we use aiohttp and botocore uses urllib3

[thehesiod]

I looked at the perf graphs between the two didn't seem too different. btw fixed above code and results.

[ekeric13]

I am also getting similar times.

Tangentially related:
I do find it weird that the docs seem to push you to creating a new client every endpoint/function:
https://aiobotocore.readthedocs.io/en/latest/index.html

Is there a best practice on how to pass around just the same client for the duration of your server?

[thehesiod]

i don't see that from the docs, it does multiple calls with the same client. you should keep your client at the "app" level. i like keeping wrapped it in an async exit stack.

[thehesiod]

note aiobotocore is no more heavy weight than botocore (in terms of design, each client holds onto a connection pool, and creating a session has a heavy start-up cost as it reads the service jsons), so you'd have the same issues there.