-
Notifications
You must be signed in to change notification settings - Fork 13.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Table performance improvements #3509
Conversation
superset/models/core.py
Outdated
|
||
def get_dialect(self): | ||
sqla_url = url.make_url(self.sqlalchemy_uri_decrypted) | ||
entrypoint = sqla_url._get_entrypoint() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not return self.get_sqla_engine().dialect
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a bit more overhead to it, plus creating an engine just to get the dialect and then ignoring the instance seems wasteful.
These few lines are what create_engine
does to determine dialect anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I hate calling private functions in external libs though... there's no guarantees as to whether the api may change in minor version changes.
To mitigate the cost we could change the use of get_sqla_engine
by using a cached sqla_engine
@property
. I think it's only a matter of adding a decorator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
check out the @memoize
decorator in superset.utils
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also for the sake of moving forward I don't mind if you just call get_sqla_engine
here and save the sqla_engine
memoized property for another PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good. I was looking for a way to cache the sqla_engine
because that function does get called frequently.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, turns out we can just call sqla_url.get_dialect()
1 similar comment
* Improved performance of 'Add table' function * got rid of pvt function call * changes metric obj to key on metric_name
* Improved performance of 'Add table' function * got rid of pvt function call * changes metric obj to key on metric_name
Reduces the number of database requests issued during "Add Table". Testing on a local environment with the example datasets gave about a 4.5x speed up.
session.commit()
from the inside of loops