[migration] metadata for dashboard filters #9109

graceguo-supercat · 2020-02-10T18:36:37Z

SUMMARY

Migrate filter_immune_slices and filter_immune_filter_fields since we enabled dashboard scoped filter metadata filter_scopes.

TEST PLAN

We will not copy old filter_immune_slices and filter_immune_filter_fields metadata. If upgrade failed, users will lose filter immune settings in their dashboards.

So before system admin do this upgrade, please backup dashboards table.

ADDITIONAL INFORMATION

REVIEWERS

@serenajiang @etr2460 @mistercrunch

codecov-io · 2020-02-10T18:55:05Z

Codecov Report

Merging #9109 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #9109      +/-   ##
=========================================
- Coverage    59.1%   59.1%   -0.01%     
=========================================
  Files         372     372              
  Lines       11920   11922       +2     
  Branches     2917    2919       +2     
=========================================
+ Hits         7045    7046       +1     
- Misses       4693    4694       +1     
  Partials      182     182

Impacted Files	Coverage Δ
superset-frontend/src/chart/chartAction.js	`43.33% <0%> (+0.09%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2913063...008ce3f. Read the comment docs.

serenajiang

Mostly 🐍 nits. How long does this take to run?

serenajiang · 2020-02-11T06:33:15Z

superset/migrations/versions/3325d4caccc8_dashboard_scoped_filters.py

+    viz_type = Column(String(250))
+
+
+dashboard_slices = Table(


Just for consistency, can you make this the same format as Slice and Dashboard?

class DashboardSlices(Base): __tablename__ = "dashboard_slices" id = Column(Integer, primary_key = True) ...and so on

I found a few places in our code base that use variable (instead of class) for relationship table.

serenajiang · 2020-02-11T06:47:45Z

superset/migrations/versions/3325d4caccc8_dashboard_scoped_filters.py

+            slice_ids = [slice.id for slice in dashboard.slices]
+            filters = (
+                session.query(Slice)
+                .filter(and_(Slice.id.in_(slice_ids), Slice.viz_type == "filter_box"))
+                .all()
+            )


I don't think the query for filters is necessary, since the slices shouuuld have all the info loaded, so you could just do something like:

filters = [slice for slice in dashboard.slices if slice.viz_type == "filter_box"]

it works! thank you.

serenajiang · 2020-02-11T06:48:40Z

superset/migrations/versions/3325d4caccc8_dashboard_scoped_filters.py

+            )
+
+            # if dashboard has filter_box
+            if len(filters):


I slightly prefer

if filters:

serenajiang · 2020-02-11T06:53:03Z

superset/migrations/versions/3325d4caccc8_dashboard_scoped_filters.py

+
+    dashboards = session.query(Dashboard).all()
+    for i, dashboard in enumerate(dashboards):
+        print("scanning dashboard ({}/{}) >>>>".format(i + 1, len(dashboards)))


Hmm, I don't have a strong preference for which, but we should probably either stick to print or logging.info and not do both.

in my test, scanning 11000 records takes ~4 minutes.
this print some progress in the console so that during deployment we can see how much work is done. Otherwise it will look like db upgrade is hanging there.
the logging.info is used to save a record (can't see it from console).

serenajiang · 2020-02-11T06:54:16Z

superset/migrations/versions/3325d4caccc8_dashboard_scoped_filters.py

+
+            session.merge(dashboard)
+        except Exception as e:
+            logging.exception(f"dashboard {dashboard.id} has error: {e}")


When you tested this, were there ever any errors? I'm curious what kind of errors might occur.

i mean to catch any user data caused null pointer exception. for example, user may have added invalid json_metada that I can't parse ? but should be very rare.

serenajiang · 2020-02-11T07:03:22Z

superset/utils/dashboard_filter_scopes_converter.py

+        slice_params = filter_slice.params or {}
+        configs = slice_params.get("filter_configs") or []
+
+        if slice_params.get("date_filter", False):


Suggested change

if slice_params.get("date_filter", False):

if slice_params.get("date_filter"):

serenajiang · 2020-02-11T21:36:36Z

superset/migrations/versions/3325d4caccc8_dashboard_scoped_filters.py

+            slice_ids = [slice.id for slice in dashboard.slices]
+            filters = [
+                slice
+                for slice in dashboard.slices
+                if slice.id.in_(slice_ids) and slice.viz_type == "filter_box"
+            ]


Don't need the extra slice_ids step - can just do.

filters = [slice for slice in dashboard.slices if slice.viz_type == "filter_box"]

john-bodley

@graceguo-superset could you add a line item in UPDATING.md with the warning you mentioned in the PR description.

john-bodley · 2020-02-12T01:55:34Z

superset/migrations/versions/3325d4caccc8_dashboard_scoped_filters.py

+
+            json_metadata.pop("filter_immune_slices", None)
+            json_metadata.pop("filter_immune_slice_fields", None)
+            if filter_scopes:


It seems this logic can be put within the if filters: block.

john-bodley · 2020-02-12T01:57:33Z

superset/migrations/versions/3325d4caccc8_dashboard_scoped_filters.py

+                logging.info(
+                    f"Adding filter_scopes for dashboard {dashboard.id}: {json.dumps(filter_scopes)}"
+                )
+            if json_metadata:


Don’t we also need to update the JSON metadata of the record if it’s falsey?

correct! I fixed it.
some old dashboards have no json_metadata. so after this migration i want to keep their json_metadata attribute be None instead of {}

john-bodley · 2020-02-12T01:58:49Z

superset/utils/dashboard_filter_scopes_converter.py

+        "filter_immune_slice_fields", {}
+    ).items():
+        for column in columns:
+            if immuned_by_column.get(column, None) is None:


Lets use a collections.defaultdict here.

graceguo-supercat · 2020-02-13T04:14:53Z

ping @john-bodley

john-bodley · 2020-02-14T17:31:20Z

superset/utils/dashboard_filter_scopes_converter.py

+        slice_params = json.loads(filter_slice.params or "{}")
+        configs = slice_params.get("filter_configs") or []
+
+        if slice_params.get("date_filter"):


How do we know this is the exhaustive list of slice parameters?

I was thinking about to add an iteration for all the possible date filter related keys. But think this function will be used by:

db migration, only run once,

dashboard import, to convert filter_immune metadata in old dashboard to use new filter_scopes
Given the usage is kind of backward compatible not future compatible, I feel enumeration probably is good enough?

pull-request-size bot added the size/L label Feb 10, 2020

graceguo-supercat force-pushed the gg-MigrateScopeFiltersMetadata branch 2 times, most recently from e29866a to 9331512 Compare February 10, 2020 19:13

graceguo-supercat changed the title ~~[migration] metadata for dashboard filters~~ [WIP][migration] metadata for dashboard filters Feb 10, 2020

graceguo-supercat force-pushed the gg-MigrateScopeFiltersMetadata branch 8 times, most recently from dd1e53d to 24206ab Compare February 11, 2020 05:12

graceguo-supercat changed the title ~~[WIP][migration] metadata for dashboard filters~~ [migration] metadata for dashboard filters Feb 11, 2020

graceguo-supercat force-pushed the gg-MigrateScopeFiltersMetadata branch from 24206ab to cd093b3 Compare February 11, 2020 05:27

serenajiang reviewed Feb 11, 2020

View reviewed changes

[migration] metadata for dashboard filters

159164a

graceguo-supercat force-pushed the gg-MigrateScopeFiltersMetadata branch from cd093b3 to 159164a Compare February 11, 2020 08:15

serenajiang reviewed Feb 11, 2020

View reviewed changes

graceguo-supercat force-pushed the gg-MigrateScopeFiltersMetadata branch 3 times, most recently from b396284 to 2f427c6 Compare February 11, 2020 22:26

fix comments

1a048c3

graceguo-supercat force-pushed the gg-MigrateScopeFiltersMetadata branch from 2f427c6 to 1a048c3 Compare February 11, 2020 23:03

john-bodley reviewed Feb 12, 2020

View reviewed changes

fix extra code review comments

008ce3f

graceguo-supercat force-pushed the gg-MigrateScopeFiltersMetadata branch from ae97bf7 to 008ce3f Compare February 12, 2020 17:45

graceguo-supercat requested a review from john-bodley February 12, 2020 19:09

john-bodley reviewed Feb 14, 2020

View reviewed changes

john-bodley approved these changes Feb 14, 2020

View reviewed changes

graceguo-supercat merged commit f4ad15e into apache:master Feb 14, 2020

This was referenced Feb 14, 2020

[dashboard] use filter_scopes metadata when import old dashboard #9145

Merged

[dashboard] clean up usage for old filter immune metadata #9146

Merged

graceguo-supercat mentioned this pull request Feb 21, 2020

[dashboard] fix filter_scopes when copy dashboard with duplicate_slices #9188

Merged

12 tasks

graceguo-supercat deleted the gg-MigrateScopeFiltersMetadata branch June 11, 2020 23:20

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.36.0 labels Feb 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[migration] metadata for dashboard filters #9109

[migration] metadata for dashboard filters #9109

graceguo-supercat commented Feb 10, 2020 •

edited

Loading

codecov-io commented Feb 10, 2020 •

edited

Loading

serenajiang left a comment

serenajiang Feb 11, 2020

graceguo-supercat Feb 11, 2020 •

edited

Loading

serenajiang Feb 11, 2020

graceguo-supercat Feb 11, 2020

serenajiang Feb 11, 2020

graceguo-supercat Feb 11, 2020

serenajiang Feb 11, 2020

graceguo-supercat Feb 11, 2020 •

edited

Loading

serenajiang Feb 11, 2020

graceguo-supercat Feb 11, 2020 •

edited

Loading

serenajiang Feb 11, 2020

graceguo-supercat Feb 11, 2020

serenajiang Feb 11, 2020

john-bodley left a comment

john-bodley Feb 12, 2020

graceguo-supercat Feb 12, 2020

john-bodley Feb 12, 2020

graceguo-supercat Feb 12, 2020

john-bodley Feb 12, 2020

graceguo-supercat Feb 12, 2020

graceguo-supercat commented Feb 13, 2020

john-bodley Feb 14, 2020

graceguo-supercat Feb 14, 2020

	if slice_params.get("date_filter", False):
	if slice_params.get("date_filter"):

[migration] metadata for dashboard filters #9109

[migration] metadata for dashboard filters #9109

Conversation

graceguo-supercat commented Feb 10, 2020 • edited Loading

CATEGORY

SUMMARY

TEST PLAN

ADDITIONAL INFORMATION

REVIEWERS

codecov-io commented Feb 10, 2020 • edited Loading

Codecov Report

serenajiang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graceguo-supercat Feb 11, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graceguo-supercat Feb 11, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graceguo-supercat Feb 11, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

john-bodley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graceguo-supercat commented Feb 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graceguo-supercat commented Feb 10, 2020 •

edited

Loading

codecov-io commented Feb 10, 2020 •

edited

Loading

graceguo-supercat Feb 11, 2020 •

edited

Loading

graceguo-supercat Feb 11, 2020 •

edited

Loading

graceguo-supercat Feb 11, 2020 •

edited

Loading