feat: Adding configuration_method column to Database Model #14433

AAfghahi · 2021-04-30T15:14:08Z

SUMMARY

We are adding a new enum column to the Database model that identifies which method was used in its' creation or updating.

The logic for this is written here: #14451

Reason

This column is necessary so that front end components know what information to present to the user when they are editing a database.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

Coming Soon.

ADDITIONAL INFORMATION

Has associated issue:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

craig-rueda · 2021-04-30T15:58:09Z

I only see a single migration file. Where is the enum defined, etc? 🤔

AAfghahi · 2021-04-30T15:59:30Z

I only see a single migration file. Where is the enum defined, etc? 🤔

Hey @craig-rueda! I am working on all of that in another PR, didn't want to add anything more to a migration. I'll send you a link when it is done!

craig-rueda · 2021-04-30T16:27:04Z

Why not just combine the two? That keeps your relevant changes together, making things easier to reason about, test, etc.

github-actions · 2021-05-03T12:34:31Z

⚠️ @AAfghahi Your base branch master has just also updated superset/migrations.

❗ Please consider rebasing your branch to avoid db migration conflicts.

betodealmeida

@craig-rueda we've been doing the migration scripts on separate PRs because it makes it easier to cherry pick them if someone needs to cherry-pick a subsequent PR with a downstream migration (ie, we need to cherry pick a PR that has a migration that revises this one). Since this PR only adds a column it can be cherry-picked safely and the logic can be left out.

Additionally, we only merge this PR once the accompanying PR with the logic has also been approved and is ready to merge.

betodealmeida · 2021-05-03T15:34:41Z

superset/migrations/versions/453530256cea_add_save_option_column_to_db_model.py

+def upgrade():
+    with op.batch_alter_table("dbs") as batch_op:
+        batch_op.add_column(
+            sa.Column("save_option", sa.VARCHAR(255), server_default="SQL_ALCHEMY")


I think save_option is not a super informative nam), something like configuration_method would be better. And let's use SQLALCHEMY_URI as the default.

That sounds good, I appreciate you helping me with naming conventions, it is something that I definitely struggle with.

betodealmeida · 2021-05-03T15:46:23Z

@AAfghahi it might be better leaving this PR as draft until the one with the logic is implemented and ready to merge. Then you can also link them to each other in the description.

AAfghahi · 2021-05-03T15:51:24Z

@AAfghahi it might be better leaving this PR as draft until the one with the logic is implemented and ready to merge. Then you can also link them to each other in the description.

converted!

codecov · 2021-05-06T21:29:59Z

Codecov Report

Merging #14433 (d6d7bdc) into master (1df9384) will increase coverage by 0.16%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #14433      +/-   ##
==========================================
+ Coverage   77.15%   77.31%   +0.16%     
==========================================
  Files         958      959       +1     
  Lines       48241    48393     +152     
  Branches     5636     5636              
==========================================
+ Hits        37219    37416     +197     
+ Misses      10821    10776      -45     
  Partials      201      201

Flag	Coverage Δ
hive	`81.09% <ø> (+0.29%)`	⬆️
mysql	`81.36% <ø> (+0.30%)`	⬆️
postgres	`81.39% <ø> (+0.29%)`	⬆️
presto	`81.08% <ø> (+0.28%)`	⬆️
python	`81.91% <ø> (+0.28%)`	⬆️
sqlite	`81.00% <ø> (+0.30%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/connectors/base/models.py	`88.03% <0.00%> (-2.87%)`	⬇️
superset/reports/logs/api.py	`94.11% <0.00%> (-1.34%)`	⬇️
superset/exceptions.py	`98.70% <0.00%> (-1.30%)`	⬇️
superset/db_engine_specs/presto.py	`89.89% <0.00%> (-0.43%)`	⬇️
superset/app.py	`81.31% <0.00%> (-0.07%)`	⬇️
superset/views/core.py	`75.48% <0.00%> (-0.04%)`	⬇️
superset/errors.py	`100.00% <0.00%> (ø)`
superset/constants.py	`100.00% <0.00%> (ø)`
superset/utils/urls.py	`100.00% <0.00%> (ø)`
superset/examples/country_map.py	`26.31% <0.00%> (ø)`
... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1df9384...d6d7bdc. Read the comment docs.

robdiciuccio · 2021-05-06T22:23:31Z

superset/migrations/versions/453530256cea_add_save_option_column_to_db_model.py

+    with op.batch_alter_table("dbs") as batch_op:
+        batch_op.add_column(
+            sa.Column(
+                "configuration_method", sa.VARCHAR(255), server_default="SQLALCHEMY_URI"


Is this column nullable?

Hey, this is a good question. I think that it should not be nullable, because it is something assigned at creation.

agree.. any existing rows should default to the current method which is the sqlalchemy uri.

hughhhh · 2021-05-14T21:21:18Z

superset/models/core.py

@@ -99,6 +100,11 @@ class CssTemplate(Model, AuditMixinNullable):
    css = Column(Text, default="")


+class ConfigurationMethod(str, enum.Enum):
+    SQLALCHEMY_URI = "sqlalchemy_form"


can we make the var equal to the value

Suggested change

SQLALCHEMY_URI = "sqlalchemy_form"

SQLALCHEMY_FORM= "sqlalchemy_form"

done, pushing up now.

hughhhh · 2021-05-14T21:30:14Z

LGTM, but will let @betodealmeida do the approval

betodealmeida · 2021-05-14T21:32:29Z

superset/migrations/versions/453530256cea_add_save_option_column_to_db_model.py

+import sqlalchemy as sa
+from alembic import op
+
+from superset.models.core import ConfigurationMethod


Let's hard code this to ensure the migration script doesn't get broken if someone changes the enum in the future.

ok, I wasn't sure what method would be preferable, to hard code or to import, looking through the versions I saw a little bit of both.

superset/migrations/versions/453530256cea_add_save_option_column_to_db_model.py

jmistry · 2021-05-17T13:50:05Z

Hi I think this change may have introduced a bug on the front end, i get this error when adding a new database
"An error occurred while creating databases: (configuration_method) Missing data for required field."

AAfghahi · 2021-05-17T15:39:05Z

hey, I think that issue is set to be fixed via this: #14583

which creates a new database edit. @betodealmeida @eschutho I can make this field optional till then?

AAfghahi · 2021-05-17T17:07:08Z

@jmistry I pushed up a change to make it optional for the time being: #14668

) * db migration for dbs * changing naming conventions * added in new migration * made server default an enum * added enum to migration file * removed enum import

AAfghahi requested a review from a team as a code owner April 30, 2021 15:14

pull-request-size bot added the size/M label Apr 30, 2021

betodealmeida requested changes May 3, 2021

View reviewed changes

AAfghahi marked this pull request as draft May 3, 2021 15:51

AAfghahi marked this pull request as ready for review May 6, 2021 21:12

AAfghahi added 3 commits May 6, 2021 17:15

db migration for dbs

824961e

changing naming conventions

b4f5662

added in new migration

fec5ef2

AAfghahi force-pushed the ch8325_ApiUpdateForDB branch from 26fd564 to fec5ef2 Compare May 6, 2021 21:16

AAfghahi changed the title ~~feat: Adding save_option column to Database Model~~ feat: Adding configuration_method column to Database Model May 6, 2021

robdiciuccio reviewed May 6, 2021

View reviewed changes

made server default an enum

22508f5

AAfghahi force-pushed the ch8325_ApiUpdateForDB branch from 5117d8b to 22508f5 Compare May 7, 2021 15:05

hughhhh reviewed May 14, 2021

View reviewed changes

AAfghahi force-pushed the ch8325_ApiUpdateForDB branch from 29804e7 to 684e9d7 Compare May 14, 2021 21:27

betodealmeida reviewed May 14, 2021

View reviewed changes

added enum to migration file

277d4f9

AAfghahi force-pushed the ch8325_ApiUpdateForDB branch from 939625d to 277d4f9 Compare May 14, 2021 21:49

removed enum import

d6d7bdc

betodealmeida approved these changes May 14, 2021

View reviewed changes

betodealmeida merged commit b064cc1 into apache:master May 14, 2021

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.3.0 labels Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Adding configuration_method column to Database Model #14433

feat: Adding configuration_method column to Database Model #14433

AAfghahi commented Apr 30, 2021 •

edited by robdiciuccio

Loading

craig-rueda commented Apr 30, 2021

AAfghahi commented Apr 30, 2021

craig-rueda commented Apr 30, 2021

github-actions bot commented May 3, 2021

betodealmeida left a comment

betodealmeida May 3, 2021

AAfghahi May 3, 2021

betodealmeida commented May 3, 2021

AAfghahi commented May 3, 2021

codecov bot commented May 6, 2021 •

edited

Loading

robdiciuccio May 6, 2021

AAfghahi May 7, 2021 •

edited

Loading

eschutho May 12, 2021

hughhhh May 14, 2021

AAfghahi May 14, 2021

hughhhh commented May 14, 2021

betodealmeida May 14, 2021

AAfghahi May 14, 2021

jmistry commented May 17, 2021

AAfghahi commented May 17, 2021

AAfghahi commented May 17, 2021

	SQLALCHEMY_URI = "sqlalchemy_form"
	SQLALCHEMY_FORM= "sqlalchemy_form"

feat: Adding configuration_method column to Database Model #14433

feat: Adding configuration_method column to Database Model #14433

Conversation

AAfghahi commented Apr 30, 2021 • edited by robdiciuccio Loading

SUMMARY

Reason

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

craig-rueda commented Apr 30, 2021

AAfghahi commented Apr 30, 2021

craig-rueda commented Apr 30, 2021

github-actions bot commented May 3, 2021

betodealmeida left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

betodealmeida commented May 3, 2021

AAfghahi commented May 3, 2021

codecov bot commented May 6, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

AAfghahi May 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hughhhh commented May 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmistry commented May 17, 2021

AAfghahi commented May 17, 2021

AAfghahi commented May 17, 2021

AAfghahi commented Apr 30, 2021 •

edited by robdiciuccio

Loading

codecov bot commented May 6, 2021 •

edited

Loading

AAfghahi May 7, 2021 •

edited

Loading