refactor: Updates some database columns to MediumText

[michael-s-molina]

SUMMARY

Similarly to #24911, #24510, #20936, #20779, #5618, this PR updates the type of some database columns to MediumText to accommodate bigger text contents such as JSON, CSS or SQL. We have found some cases where SQL queries shared by users were truncated due to insufficient storage space. You can check the referenced PRs for more context on why the change is important.

TESTING INSTRUCTIONS

1 - Execute superset db upgrade and check the affected tables
2 - Make sure the nullable state is preserved
3 - Execute superset db downgrade and check the affected tables

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (152cd70) 67.16% compared to head (472e27f) 69.49%.
Report is 6 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #27119      +/-   ##
==========================================
+ Coverage   67.16%   69.49%   +2.32%     
==========================================
  Files        1900     1900              
  Lines       74447    74449       +2     
  Branches     8297     8297              
==========================================
+ Hits        50005    51737    +1732     
+ Misses      22387    20657    -1730     
  Partials     2055     2055              

Flag	Coverage Δ
hive	53.80% <100.00%> (?)
mysql	78.02% <100.00%> (+<0.01%)	⬆️
postgres	78.12% <100.00%> (+<0.01%)	⬆️
presto	53.75% <100.00%> (?)
python	83.09% <100.00%> (+4.82%)	⬆️
sqlite	77.64% <100.00%> (+<0.01%)	⬆️
unit	56.43% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zhaoyongjie]

@michael-s-molina, I encountered a database deadlock error when running the database migration script a few days ago. We are using PostgreSQL as Superset metadata. This change only affects MySQL, so could you please add a condition to detect if MySQL is used as metadata? If not, pass the upgrade and downgrade functions.

bind = op.get_bind()

if isinstance(bind.dialect, MySQLDialect):
    // the upgrade ......

[john-bodley]

@zhaoyongjie are these migrations a no-op if PostgreSQL doesn't differentiate between TEXT and MEDIUMTEXT?

[zhaoyongjie]

@john-bodley PG only has TEXT type for string-like column, I'm not sure what should be done under the SQLAlchemy core.

[michael-s-molina]

@john-bodley According to ChatGPT:

No, the alter_column function in SQLAlchemy is not a no-op if the existing type is the same as the new type. When you use the alter_column function in SQLAlchemy to alter a column, it will generate an ALTER TABLE statement that modifies the column definition in the database schema. Even if the existing type is the same as the new type, SQLAlchemy will still execute the ALTER TABLE statement to ensure that the column definition matches the intended type.
This behavior ensures consistency and helps to handle any other modifications or changes associated with the column, such as constraints or defaults.

[mistercrunch]

Had to look this up, but MediumText here is:

def MediumText() -> Variant:  # pylint:disable=invalid-name
    return Text().with_variant(MEDIUMTEXT(), "mysql")

[mistercrunch]

Can we make sure we only run this DDL for MySQL since MEDIUMTEXT is a MySQL-only construct? Something like if engine == 'mysql': within both upgrade and downgrade (?)

Unless SQLAlchemy is smart enough to understand this is a no-op for Postgres and others(?)

[mistercrunch]

Ooops just realized after my review that similar comments had been made :)

Side note - if that's helpful - we now have a utils module that can expose methods to be reused across migrations (somehow it was tricky to introduce this) -> https://github.com/apache/superset/blob/master/superset/migrations/migration_utils.py

Meaning you can create more complex or engine-specific logic here that we'll be able to re-use in the future.

[michael-s-molina]

so could you please add a condition to detect if MySQL is used as metadata?

Can we make sure we only run this DDL for MySQL since MEDIUMTEXT is a MySQL-only construct?

@zhaoyongjie @mistercrunch Added the MySQL condition 😉

Side note - if that's helpful - we now have a utils module that can expose methods to be reused across migrations

Nice! Thanks for sharing.

[zhaoyongjie]

LGTM, Thanks for the upgrading.