osc-sdk-python initialization overhead

[pitouche]

Hello,

I noticed, when dealing with a huge number of account, that the initialization phase of the Gateway object is very slow.

When benchmarking the call to:

osc_api_client = Gateway(access_key=creddict["AccessId"], secret_key=creddict["SecretKey"], region=creddict["ComputeRegion"]

It takes arround 230ms to initialize.

Since i am dealing with a set of 3000 accounts, this overhead lead to a total of 750 Seconds only for the object initialization.

I have benched also some other libs (boto3.client) this init phase is arround 50ms which is somehow huge also, but 5 times less.

Why is this initialization time so huge ? Is there a way to reduce this overhead ?

Thanks.

[jerome-jutteau]

Hi @pitouche,

We can reproduce the issue with this snippet:

from osc_sdk_python import Gateway
from os import getenv
import timeit

ak=getenv("OSC_ACCESS_KEY")
sk=getenv("OSC_SECRET_KEY")
region=getenv("OSC_REGION")
number=100
t = timeit.repeat(lambda: Gateway(access_key=ak, secret_key=sk, region=region), number=number, repeat=1)
print(t[0] / float(number))

I have a mean time of ~200ms which is high.

Thanks for pointing out the issue, we will look at it.

[jerome-jutteau]

osc-sdk-python in its current version have to dynamically load at init time a quite large yaml describing the API (osc-api).

If you are using a single python instance, we can quickly add you a new function to change your credentials by using the same Gateway object over and over.

This is a tactical solution but note that we were already planning to switch to a static python sdk which will avoid this type of effort at init time and help editors to auto-complete.

[outscale-mgo]

Hello, I've juste make a PR(#94), that enable to use multiples account with the same Gateway.
This doesn't fix initialization overhead, but can serve as workaround for some use-case.
Would that fix your use-case @pitouche ?

[pitouche]

Hello,

As i understand, we would get the overhead only once, which is fair, and since jobs i am running is using a single python instance i will be able to "re-use" the Gateway object.

Just one question, will update_credential be suitable if we change "region" also ? Is the Gateway object generated still suitable if we change region ?

Thanks for your quick answer and proposal

[jerome-jutteau]

Hi,

update_credential() should also allow you to switch region. We will make a new release once this feature pushed to ease your fix.

[outscale-mgo]

tested with

>>> gw = Gateway(access_key="XXX", secret_key="YYYYYYY", region="eu-west-2") # replace with real AK/SK
>>> gw.ReadVms()['Vms'][1]['Tags']
[{'Value': 'Packer Builder', 'Key': 'Name'}]
>>> gw.update_credentials(access_key="XXX", secret_key="YYYYYYY", region="eu-west-3")  # replace with real AK/SK
-> error

So it work, as eu-west-3 doesn't exist, so the region was taken into acount
Note that ak/sk need to be resend, but as ak/sk will changes depending of the region, I guess it's not a problem to resend all Credetials arguments each times update_credentials is call ?
Could be a little annoying for password/login though

[outscale-mgo]

I guess update_credentials workaround is enough, as no one participate to this issue since January.
so if it's still an issue, thanks to reopen