Add Table performance improvements #3509
Conversation
superset/models/core.py
Outdated
|
|
||
| def get_dialect(self): | ||
| sqla_url = url.make_url(self.sqlalchemy_uri_decrypted) | ||
| entrypoint = sqla_url._get_entrypoint() |
There was a problem hiding this comment.
why not return self.get_sqla_engine().dialect ?
There was a problem hiding this comment.
There's a bit more overhead to it, plus creating an engine just to get the dialect and then ignoring the instance seems wasteful.
These few lines are what create_engine does to determine dialect anyway.
There was a problem hiding this comment.
I hate calling private functions in external libs though... there's no guarantees as to whether the api may change in minor version changes.
To mitigate the cost we could change the use of get_sqla_engine by using a cached sqla_engine @property. I think it's only a matter of adding a decorator.
There was a problem hiding this comment.
check out the @memoize decorator in superset.utils
There was a problem hiding this comment.
also for the sake of moving forward I don't mind if you just call get_sqla_engine here and save the sqla_engine memoized property for another PR
There was a problem hiding this comment.
Sounds good. I was looking for a way to cache the sqla_engine because that function does get called frequently.
There was a problem hiding this comment.
Oh, turns out we can just call sqla_url.get_dialect()
1 similar comment
* Improved performance of 'Add table' function * got rid of pvt function call * changes metric obj to key on metric_name
* Improved performance of 'Add table' function * got rid of pvt function call * changes metric obj to key on metric_name
mistercrunch
added
🏷️ bot
Reduces the number of database requests issued during "Add Table". Testing on a local environment with the example datasets gave about a 4.5x speed up.
session.commit()from the inside of loops