feat(key-value): add superset metastore cache #19232

villebro · 2022-03-17T08:12:06Z

SUMMARY

Add custom Flask-Caching backend that leverages the new key_value table that was introduced in #19078. This makes it possible to use the metadata database as a cache. Superset defaults to the new cache for required caches that have not been defined in the configs. This is to reintroduce support for multi-pod deployments that don't have a dedicated cache.

The cache implements all cache methods that are used in the application (add, set, get, has, delete), and can be used as a caching backend for any purpose. For instance, to use it to store chart data, the following can be added to superset_config.py:

DATA_CACHE_CONFIG = {
    "CACHE_TYPE": "SupersetMetastoreCache",
    "CACHE_KEY_PREFIX": "superset_results", 
    "CACHE_DEFAULT_TIMEOUT": 86400,
}

In addition, Explore form data/Dashboard filter state caches in the ephemeral environments are currently failing on master - after this gets merged they should now work as intended, as they'll use this cache.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

docs/docs/installation/cache.mdx

superset-frontend/cypress-base/cypress/integration/dashboard/key_value.test.ts

superset/key_value/cache.py

superset/utils/cache_manager.py

superset/key_value/cache.py

tests/integration_tests/key_value/cache_test.py

villebro · 2022-03-18T17:19:23Z

tests/integration_tests/key_value/cache_test.py

+    with freeze_time(dttm):
+        cache.set(FIRST_KEY, FIRST_KEY_INITIAL_VALUE, int(delta.total_seconds()))
+        assert cache.get(FIRST_KEY) == FIRST_KEY_INITIAL_VALUE
+    with freeze_time(dttm + delta - timedelta(seconds=1)):
+        assert cache.has(FIRST_KEY)
+        assert cache.get(FIRST_KEY) == FIRST_KEY_INITIAL_VALUE
+    with freeze_time(dttm + delta + timedelta(seconds=1)):
+        assert cache.has(FIRST_KEY) is False
+        assert cache.get(FIRST_KEY) is None


Btw I truly LOVE freeze_time! ❤️

codecov · 2022-03-18T17:36:11Z

Codecov Report

Merging #19232 (22f52f6) into master (d645579) will increase coverage by 0.05%.
The diff coverage is 90.19%.

@@            Coverage Diff             @@
##           master   #19232      +/-   ##
==========================================
+ Coverage   66.53%   66.59%   +0.05%     
==========================================
  Files        1667     1670       +3     
  Lines       64360    64501     +141     
  Branches     6493     6493              
==========================================
+ Hits        42824    42952     +128     
- Misses      19854    19867      +13     
  Partials     1682     1682

Flag	Coverage Δ
hive	`52.54% <9.45%> (-0.11%)`	⬇️
mysql	`81.53% <90.19%> (+0.05%)`	⬆️
postgres	`81.57% <90.19%> (+0.04%)`	⬆️
presto	`52.39% <9.45%> (-0.11%)`	⬇️
python	`81.99% <90.19%> (+0.03%)`	⬆️
sqlite	`81.35% <90.19%> (+0.05%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/dashboards/permalink/commands/create.py	`92.59% <ø> (ø)`
superset/dashboards/permalink/commands/get.py	`82.35% <ø> (ø)`
superset/explore/permalink/commands/create.py	`93.33% <ø> (ø)`
superset/explore/permalink/commands/get.py	`88.57% <ø> (ø)`
superset/key_value/commands/delete.py	`86.11% <50.00%> (-2.78%)`	⬇️
superset/key_value/commands/update.py	`89.58% <66.66%> (-1.91%)`	⬇️
superset/key_value/commands/delete_expired.py	`80.00% <80.00%> (ø)`
superset/key_value/commands/create.py	`93.87% <83.33%> (+4.40%)`	⬆️
superset/key_value/commands/upsert.py	`89.79% <89.79%> (ø)`
superset/extensions/metastore_cache.py	`98.18% <98.18%> (ø)`
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d645579...22f52f6. Read the comment docs.

villebro · 2022-03-18T18:19:05Z

/testenv up

github-actions · 2022-03-18T18:21:26Z

@villebro Ephemeral environment spinning up at http://35.87.132.23:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

villebro · 2022-03-18T19:07:33Z

I can confirm that the dashboard and explore ephemeral env now work as intended by defaulting to the new cache

ktmud

Nice work! I'm still a little hesitant on conflating caching and persistent storage (the semantics are just so different), but this does seem to be an improvement to status quo, so I'll vote yes on this change.

ktmud · 2022-03-18T21:04:07Z

superset/key_value/cache.py

+from superset.key_value.types import KeyType
+
+RESOURCE = "superset_cache"
+KEY_TYPE: KeyType = "uuid"


Linking my previous comment about KeyType in case you missed it: #19078 (comment)

Thanks - many good comments there - I'll be opening a follow-up PR to address those!

superset/key_value/cache.py

superset/utils/cache_manager.py

superset/key_value/cache.py

villebro · 2022-03-19T10:47:42Z

@ktmud I believe I've addressed all your comments

michael-s-molina

LGTM. Thanks for the improvement!

ktmud

LGTM. Thanks for the updates!

github-actions · 2022-03-21T17:47:24Z

Ephemeral environment shutdown and build artifacts deleted.

(cherry picked from commit 72b9a7f)

superset-github-bot bot added the preset-io label Mar 17, 2022

pull-request-size bot added the size/L label Mar 17, 2022

villebro added 5 commits March 18, 2022 13:04

feat(key-value): add superset cache

51280e5

rename method

c97c1d6

lint

6a410ac

fix command

eae089b

add tests + docs

a835e9c

villebro force-pushed the villebro/superset-cache branch from 0f76444 to a835e9c Compare March 18, 2022 12:26

villebro changed the title ~~[WIP] feat(key-value): add superset cache~~ feat(key-value): add superset cache Mar 18, 2022

lint

d4338b9

villebro force-pushed the villebro/superset-cache branch from 7c2d53d to d4338b9 Compare March 18, 2022 12:39

villebro mentioned this pull request Mar 18, 2022

fix: Explore long URL problem #18181

Merged

9 tasks

villebro requested review from michael-s-molina, ktmud, betodealmeida, etr2460 and nytai and removed request for betodealmeida March 18, 2022 12:48

michael-s-molina reviewed Mar 18, 2022

View reviewed changes

address comments + fix test

d2cb486

betodealmeida reviewed Mar 18, 2022

View reviewed changes

superset/key_value/cache.py Outdated Show resolved Hide resolved

tests/integration_tests/key_value/cache_test.py Outdated Show resolved Hide resolved

villebro added 2 commits March 18, 2022 19:07

more fixes

0e06830

make namespace truly mandatory

78dfe29

villebro commented Mar 18, 2022

View reviewed changes

redisable cypress test

ef1984e

ktmud reviewed Mar 18, 2022

View reviewed changes

Merge branch 'master' into villebro/superset-cache

90a659c

villebro changed the title ~~feat(key-value): add superset cache~~ feat(key-value): add superset metastore cache Mar 19, 2022

address review comments

58fd5ba

pull-request-size bot added size/XL and removed size/L labels Mar 19, 2022

lint

22f52f6

villebro requested a review from ktmud March 19, 2022 10:47

michael-s-molina approved these changes Mar 21, 2022

View reviewed changes

ktmud approved these changes Mar 21, 2022

View reviewed changes

villebro merged commit 72b9a7f into apache:master Mar 21, 2022

villebro deleted the villebro/superset-cache branch March 22, 2022 09:15

villebro mentioned this pull request Mar 22, 2022

fix(cache): only warn about fallback cache for non-debug mode #19305

Merged

9 tasks

michael-hoffman-26 pushed a commit to nielsen-oss/superset that referenced this pull request Mar 23, 2022

feat(key-value): add superset metastore cache (apache#19232)

d603128

villebro mentioned this pull request Mar 25, 2022

fix(caching): support metastore cache for non-required caches #19369

Merged

9 tasks

villebro added the lts-v1 label Mar 29, 2022

villebro added a commit that referenced this pull request Apr 3, 2022

feat(key-value): add superset metastore cache (#19232)

f317201

(cherry picked from commit 72b9a7f)

villebro mentioned this pull request Apr 4, 2022

chore(releasing): use node 16 for testing and remove redundant updating note #19505

Merged

9 tasks

michael-s-molina mentioned this pull request Jul 12, 2022

[SIP-85] OAuth2 for databases #20300

Closed

mistercrunch added 🍒 1.5.0 🍒 1.5.1 🍒 1.5.2 🍒 1.5.3 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 2.0.0 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(key-value): add superset metastore cache #19232

feat(key-value): add superset metastore cache #19232

villebro commented Mar 17, 2022 •

edited

villebro Mar 18, 2022

codecov bot commented Mar 18, 2022 •

edited

villebro commented Mar 18, 2022

github-actions bot commented Mar 18, 2022

villebro commented Mar 18, 2022

ktmud left a comment

ktmud Mar 18, 2022

villebro Mar 19, 2022

villebro commented Mar 19, 2022

michael-s-molina left a comment

ktmud left a comment

github-actions bot commented Mar 21, 2022

feat(key-value): add superset metastore cache #19232

feat(key-value): add superset metastore cache #19232

Conversation

villebro commented Mar 17, 2022 • edited

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

villebro Mar 18, 2022

Choose a reason for hiding this comment

codecov bot commented Mar 18, 2022 • edited

Codecov Report

villebro commented Mar 18, 2022

github-actions bot commented Mar 18, 2022

villebro commented Mar 18, 2022

ktmud left a comment

Choose a reason for hiding this comment

ktmud Mar 18, 2022

Choose a reason for hiding this comment

villebro Mar 19, 2022

Choose a reason for hiding this comment

villebro commented Mar 19, 2022

michael-s-molina left a comment

Choose a reason for hiding this comment

ktmud left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 21, 2022

villebro commented Mar 17, 2022 •

edited

codecov bot commented Mar 18, 2022 •

edited