fix: Ensure table uniqueness on update #15909

john-bodley · 2021-07-27T15:32:49Z

SUMMARY

The DAO model which @dpgaspar, @villebro, et al. added goes a long way to making #5449 obsolete, i.e., it has the necessary existence checks when creating or updating a table.

There are however at least two issues,

The Datasource Editor still uses the legacy /datasource/save/ endpoint (per here) rather than the DAO model and so the existence logic is not invoked on update.
The legacy CRUD Editor does not use the DAO model.

meaning it is currently possible to rename a dataset resulting in multiple datasets with the same schema and table name.

Though these issues should be resolved once we fully deprecate the CRUD editor and migrate over to the DAO model, I felt it was prudent to add an interim check which enforces uniqueness. Given then various vectors I thought the simplest and safest path was to leverage the approach in #5449 and add a SQLAlchemy before update listener. Note although the UX may not be ideal, my sense is it is suffice as an interim measure.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Editing/Updating via the Datasource Editor

Editing/Updating via the legacy CRUD Editor

Though this UX is not ideal, i.e., it routes you to explorer, the dataset is not updated and there is messaging indicating that the dataset already exists.

TESTING INSTRUCTIONS

CI and manual testing.

ADDITIONAL INFORMATION

Has associated issue:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

john-bodley · 2021-07-27T22:44:09Z

tests/integration_tests/base_tests.py

@@ -254,13 +245,31 @@ def get_slice(
        return slc

    @staticmethod
-    def get_table_by_name(name: str) -> SqlaTable:
-        return get_table_by_name(name)
+    def get_table(


Cleanup to ensure consistency.

john-bodley · 2021-07-27T22:46:12Z

tests/integration_tests/datasource_tests.py



 class TestDatasource(SupersetTestCase):
    def setUp(self):
-        self.original_attrs = {}


I really have no idea why this logic is here and the before_update method exposed its fragility. Rather than using instance variables to save state and restore state, using a sub-transaction and rolling back seems considerably cleaner.

john-bodley · 2021-07-27T22:47:38Z

tests/integration_tests/datasource_tests.py

@@ -168,25 +158,21 @@ def test_save(self):
            else:
                self.assertEqual(resp[k], datasource_post[k])

-    def save_datasource_from_dict(self, datasource_dict):


Hmmm. The local variable was never used but instead the global variable was. Seems like a mistake.

john-bodley · 2021-07-27T22:48:01Z

tests/integration_tests/datasource_tests.py

        data = dict(data=json.dumps(datasource_post))
        resp = self.get_json_resp("/datasource/save/", data)
        return resp

+    @pytest.mark.usefixtures("load_birth_names_dashboard_with_slices")


I have no idea how this test worked without first loading the example data.

john-bodley · 2021-07-27T22:48:37Z

tests/integration_tests/datasource_tests.py


+        datasource_post = get_datasource_post()
        datasource_post["id"] = tbl.id


Yikes! This was updating the global variable.

john-bodley · 2021-07-27T22:49:31Z

tests/integration_tests/tasks/async_queries_tests.py

@@ -42,11 +42,6 @@
 from tests.integration_tests.test_app import app


-def get_table_by_name(name: str) -> SqlaTable:


This doesn't need to be redefined. Adhering to the DRY principle.

john-bodley · 2021-07-27T23:18:47Z

tests/integration_tests/import_export_tests.py

@@ -351,13 +345,6 @@ def test_import_2_slices_for_same_table(self):
        self.assert_slice_equals(slc_2, imported_slc_2)
        self.assertEqual(imported_slc_2.datasource.perm, imported_slc_2.perm)

-    def test_import_slices_for_non_existent_table(self):


I'm not sure what the purpose of this test was. If the table doesn't exist the get_table(...) precursor method with throw.

codecov · 2021-07-28T00:18:54Z

Codecov Report

Merging #15909 (444ac0d) into master (39db6a7) will increase coverage by 0.19%.
The diff coverage is 54.54%.

❗ Current head 444ac0d differs from pull request most recent head c08f541. Consider uploading reports for the commit c08f541 to get more accurate results

@@            Coverage Diff             @@
##           master   #15909      +/-   ##
==========================================
+ Coverage   76.69%   76.88%   +0.19%     
==========================================
  Files         995      995              
  Lines       52774    52797      +23     
  Branches     6691     6695       +4     
==========================================
+ Hits        40475    40594     +119     
+ Misses      12074    11978      -96     
  Partials      225      225

Flag	Coverage Δ
hive	`81.32% <93.33%> (?)`
mysql	`81.61% <93.33%> (+<0.01%)`	⬆️
postgres	`81.63% <93.33%> (+<0.01%)`	⬆️
presto	`81.42% <93.33%> (?)`
python	`82.15% <93.33%> (+0.42%)`	⬆️
sqlite	`81.28% <93.33%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...frontend/src/dashboard/components/Header/index.jsx	`63.74% <14.28%> (-1.74%)`	⬇️
...tend/src/explore/components/ExploreChartHeader.jsx	`56.33% <14.28%> (-3.96%)`	⬇️
.../ReportModal/HeaderReportActionsDropdown/index.tsx	`25.71% <33.33%> (+0.71%)`	⬆️
superset/connectors/sqla/models.py	`89.73% <92.85%> (+1.73%)`	⬆️
...perset-frontend/src/addSlice/AddSliceContainer.tsx	`78.78% <100.00%> (+0.66%)`	⬆️
superset/models/slice.py	`86.18% <100.00%> (ø)`
superset-frontend/src/reports/actions/reports.js	`10.86% <0.00%> (-10.87%)`	⬇️
superset-frontend/src/components/Select/Select.tsx	`73.05% <0.00%> (-1.37%)`	⬇️
superset/common/query_context.py	`90.74% <0.00%> (-0.47%)`	⬇️
superset/utils/core.py	`88.30% <0.00%> (+0.12%)`	⬆️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 39db6a7...c08f541. Read the comment docs.

etr2460 · 2021-07-29T20:46:25Z

superset/connectors/sqla/models.py

+    # not exist in the migrations, but is required by `import_from_dict` to ensure the
+    # correct filters are applied in order to identify uniqueness.
+    #
+    # The reason it does not physically exist is MySQL, PostgreSQL have a different


grammar nit: ...is MySQL and PostgreSQL

etr2460

lgtm with one nit. thanks for cleaning up this edge case

john-bodley · 2021-07-30T01:52:51Z

superset/models/slice.py

@@ -53,9 +53,9 @@
 logger = logging.getLogger(__name__)


-class Slice(
+class Slice(  # pylint: disable=too-many-instance-attributes,too-many-public-methods


I'm not sure why pylint was complaining on this file as it was unchanged.

Yes, I also faced this issue.

Same here. :-P

* fix: Ensure table uniqueness on update * Update models.py * Update slice.py * Update datasource_tests.py Co-authored-by: John Bodley <john.bodley@airbnb.com>

pull-request-size bot added the size/L label Jul 27, 2021

john-bodley force-pushed the john-bodley--table-uniqueness branch from 74a6352 to 915c28c Compare July 27, 2021 15:37

john-bodley marked this pull request as ready for review July 27, 2021 16:04

john-bodley requested a review from a team as a code owner July 27, 2021 16:04

john-bodley requested review from dpgaspar, etr2460, nytai and villebro July 27, 2021 16:04

john-bodley mentioned this pull request Jul 27, 2021

[schema] Updating the tables schema #5449

Closed

john-bodley force-pushed the john-bodley--table-uniqueness branch 9 times, most recently from b8a102c to 3735c81 Compare July 27, 2021 22:43

pull-request-size bot added size/XL and removed size/L labels Jul 27, 2021

john-bodley commented Jul 27, 2021

View reviewed changes

john-bodley force-pushed the john-bodley--table-uniqueness branch 2 times, most recently from 0a7a81e to 9175b6d Compare July 27, 2021 23:17

john-bodley commented Jul 27, 2021

View reviewed changes

john-bodley force-pushed the john-bodley--table-uniqueness branch 2 times, most recently from 9a8e25c to f76f2b5 Compare July 28, 2021 00:02

fix: Ensure table uniqueness on update

cade3cb

john-bodley force-pushed the john-bodley--table-uniqueness branch from f76f2b5 to cade3cb Compare July 28, 2021 00:57

etr2460 reviewed Jul 29, 2021

View reviewed changes

etr2460 approved these changes Jul 29, 2021

View reviewed changes

john-bodley added 2 commits July 30, 2021 09:48

Update models.py

11965f7

Update slice.py

feea25d

john-bodley commented Jul 30, 2021

View reviewed changes

john-bodley added 2 commits August 2, 2021 10:38

Merge branch 'master' into john-bodley--table-uniqueness

a2b81bd

Update datasource_tests.py

c08f541

john-bodley merged commit c0615c5 into apache:master Aug 2, 2021

john-bodley deleted the john-bodley--table-uniqueness branch August 2, 2021 19:45

exemplary-citizen mentioned this pull request Aug 3, 2021

feat: Add parquet upload #14449

Merged

8 tasks

jayakrishnankk mentioned this pull request Aug 25, 2021

Dashboard export/import doesn't work in v1.3 #16434

Closed

3 tasks

hughhhh mentioned this pull request Sep 27, 2021

fix: Fix Uniqueness check before update for Sqllab Overwrites #16859

Merged

9 tasks

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.3.0 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Ensure table uniqueness on update #15909

fix: Ensure table uniqueness on update #15909

john-bodley commented Jul 27, 2021 •

edited

john-bodley Jul 27, 2021

john-bodley Jul 27, 2021

john-bodley Jul 27, 2021 •

edited

john-bodley Jul 27, 2021

john-bodley Jul 27, 2021

john-bodley Jul 27, 2021

john-bodley Jul 27, 2021

codecov bot commented Jul 28, 2021 •

edited

etr2460 Jul 29, 2021

etr2460 left a comment

john-bodley Jul 30, 2021

zhaoyongjie Jul 30, 2021

betodealmeida Aug 3, 2021


		datasource_post = get_datasource_post()
		datasource_post["id"] = tbl.id

		@@ -42,11 +42,6 @@
		from tests.integration_tests.test_app import app


		def get_table_by_name(name: str) -> SqlaTable:

fix: Ensure table uniqueness on update #15909

fix: Ensure table uniqueness on update #15909

Conversation

john-bodley commented Jul 27, 2021 • edited

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Editing/Updating via the Datasource Editor

Editing/Updating via the legacy CRUD Editor

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Choose a reason for hiding this comment

Choose a reason for hiding this comment

john-bodley Jul 27, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 28, 2021 • edited

Codecov Report

Choose a reason for hiding this comment

etr2460 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

john-bodley commented Jul 27, 2021 •

edited

john-bodley Jul 27, 2021 •

edited

codecov bot commented Jul 28, 2021 •

edited