Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Migration to fix out of sync schema_perm in charts and datasets #24884

Merged
merged 6 commits into from
Aug 7, 2023

Conversation

jfrag1
Copy link
Member

@jfrag1 jfrag1 commented Aug 3, 2023

SUMMARY

Currently, a bug exists in Superset where sometimes when updating a dataset's schema, the dataset and any charts belonging to the dataset do not have their schema_perm attribute get updated properly. I've identified that this bug is the result of this line which checks if a dataset's schema was changed after a dataset update. Usually when the schema changes, both conditions are true/truthy, however sometimes history_schema.deleted is empty/falsy, which results in the schema_perm not getting updated. I was not able to find a cause for this inconsistency.

This PR does not actually fix this bug; it just fixes any currently affected charts and datasets. The reasoning behind this is that there was another bug recently fixed which caused some datasets to be created without a schema, resulting in users needing the update dataset schemas more frequently than usual. This made the schema_perm bug here more impactful.

Additionally, actually fixing this bug is not simple. I initially thought we could remove the history_schema.deleted condition, but it turns out that the other condition history_schema.has_changes() is pretty much always true even when the schema has not changed, which would result in many unnecessary queries. I believe the long-term fix for this is either to move away from sqlalchemy event listeners/attribute history and instead handle this logic in the DAO layer and/or to rehaul the data model for data access permissions so that these permission strings are no longer needed

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided - no downtime
  • Introduces new feature or API
  • Removes existing feature or API

@jfrag1 jfrag1 requested a review from a team as a code owner August 3, 2023 18:18
@codecov
Copy link

codecov bot commented Aug 3, 2023

Codecov Report

Merging #24884 (559e29f) into master (b71541f) will decrease coverage by 0.01%.
Report is 9 commits behind head on master.
The diff coverage is 71.42%.

❗ Current head 559e29f differs from pull request most recent head af8fefc. Consider uploading reports for the commit af8fefc to get more accurate results

@@            Coverage Diff             @@
##           master   #24884      +/-   ##
==========================================
- Coverage   69.00%   69.00%   -0.01%     
==========================================
  Files        1906     1906              
  Lines       74141    74142       +1     
  Branches     8209     8208       -1     
==========================================
- Hits        51161    51160       -1     
  Misses      20859    20859              
- Partials     2121     2123       +2     
Flag Coverage Δ
hive 54.13% <ø> (ø)
mysql 79.22% <ø> (ø)
postgres 79.32% <ø> (ø)
presto 54.03% <ø> (ø)
python 83.37% <ø> (ø)
sqlite 77.89% <ø> (ø)
unit 55.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
...s/legacy-plugin-chart-country-map/src/countries.ts 100.00% <ø> (ø)
...t-frontend/src/explore/actions/saveModalActions.js 95.52% <ø> (ø)
...c/features/datasets/AddDataset/LeftPanel/index.tsx 89.47% <ø> (ø)
superset-frontend/src/features/datasets/styles.ts 92.00% <50.00%> (-8.00%) ⬇️
...uperset-ui-core/src/query/types/AnnotationLayer.ts 100.00% <100.00%> (ø)
...tend/src/features/datasets/DatasetLayout/index.tsx 100.00% <100.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Member

@craig-rueda craig-rueda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eschutho eschutho merged commit 07992c1 into apache:master Aug 7, 2023
29 checks passed
@eschutho eschutho deleted the jack/fix-dataset-schema-perm branch August 7, 2023 16:34
sadpandajoe pushed a commit to preset-io/superset that referenced this pull request Aug 7, 2023
@sadpandajoe
Copy link
Contributor

🏷️ preset:2023.31

@michael-s-molina michael-s-molina added the v3.0 Label added by the release manager to track PRs to be included in the 3.0 branch label Aug 7, 2023
michael-s-molina pushed a commit that referenced this pull request Aug 7, 2023
@mistercrunch mistercrunch added 🍒 3.0.0 🍒 3.0.1 🍒 3.0.2 🍒 3.0.3 🍒 3.0.4 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.1.0 labels Mar 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/L v3.0 Label added by the release manager to track PRs to be included in the 3.0 branch 🍒 3.0.0 🍒 3.0.1 🍒 3.0.2 🍒 3.0.3 🍒 3.0.4 🚢 3.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants