Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DBMS-DataSources should use memoization for queries with :params #97

Closed
schuderer opened this issue Mar 31, 2020 · 0 comments · Fixed by #104
Closed

DBMS-DataSources should use memoization for queries with :params #97

schuderer opened this issue Mar 31, 2020 · 0 comments · Fixed by #104
Labels
enhancement New feature or request

Comments

@schuderer
Copy link
Owner

schuderer commented Mar 31, 2020

Description

Replace caching (independent from arguments) with memoization (dependent on arguments).

The expires config setting of OracleDataSource just blindly caches the result. This limits its use case.

It does not take into consideration that the optional arg_dict/params parameter can contain query parameters, and that successive calls to get_dataframes therefore should actually not return the same data.

It is probably the same for the examples' RecordsDataSource and ImpalaDatasource (cf #90).

Probably also a good opportunity to refactor and clean up the caching/memoization code in the base classes and existing DataSource implementations (e.g. a decorator wrapping lru_cache in a non-memory-leaking way, see e.g. https://stackoverflow.com/questions/33672412/python-functools-lru-cache-with-class-methods-release-object)

If #96 is ever tackled, this issue should be taken into consideration there, as well.

@schuderer schuderer added the enhancement New feature or request label Mar 31, 2020
@schuderer schuderer added this to To do in Release 1.0.0 tracker via automation Mar 31, 2020
Release 1.0.0 tracker automation moved this from To do to Done Apr 24, 2020
@schuderer schuderer mentioned this issue Jun 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

1 participant