fix: Remove double pickling for cached payloads #10222
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
john-bodley
force-pushed
the
john-bodley--pickle
branch
from
56565d8
to
0d0b4ff
Compare
john-bodley
force-pushed
the
john-bodley--pickle
branch
from
0d0b4ff
to
cf613d4
Compare
john-bodley
force-pushed
the
john-bodley--pickle
branch
from
cf613d4
to
78d9f7b
Compare
john-bodley
commented
Jul 1, 2020
| @@ -260,14 +258,8 @@ def get_df_payload( # pylint: disable=too-many-locals,too-many-statements | |||
| if is_loaded and cache_key and cache and status != utils.QueryStatus.FAILED: | |||
| try: | |||
| cache_value = dict(dttm=cached_dttm, df=df, query=query) | |||
| cache_binary = pkl.dumps(cache_value, protocol=pkl.HIGHEST_PROTOCOL) | |||
|
|
|||
| logger.info( | |||
There was a problem hiding this comment.
I annexed this as we no longer a priori know the size of the payload and it seems somewhat like an unnecessary comment.
mistercrunch
reviewed
Jul 2, 2020
| stats_logger.incr("set_cache_key") | ||
| cache.set(cache_key, cache_binary, timeout=self.cache_timeout) | ||
| cache.set(cache_key, cache_value, timeout=self.cache_timeout) |
There was a problem hiding this comment.
random thought [out of scope for this PR], we should time this operation eventually
mistercrunch
approved these changes
Jul 2, 2020
auxten
pushed a commit
to auxten/incubator-superset
that referenced
this pull request
Nov 20, 2020
Co-authored-by: John Bodley <john.bodley@airbnb.com>
mistercrunch
added
🏷️ bot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
It seems we pickle the visualization payloads et al. prior to caching via the Flask-Caching package, however for all backends as part of the set/get logic the object to cache is pickled and the unpicked respectively, i.e., for Redis the set and get methods call
dump_objectandload_objectrespectively which in turn callspickle.dumpsandpickle.loads, hence we were actually double pickling the payloads.BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TEST PLAN
CI.
ADDITIONAL INFORMATION