feat: Migrates TreeMap chart #23741

michael-s-molina · 2023-04-19T14:52:35Z

SUMMARY

This PR migrates the Treemap chart to the ECharts version.

Removes the legacy Treemap code and unregisters the plugin.
Removes old plugin references throughout the codebase
Updates unit tests
Updates E2E tests
Updates the example dashboards
Updates translation files

Note a previous migration existed however the legacy chart wasn't removed and thus new legacy Treemap charts could have been created.

AFTER SCREENSHOT

TESTING INSTRUCTIONS

1 - Make sure all Treemap (legacy) charts were converted to Treemap
2 - Make sure Treemap (legacy) is not available anymore in the viz picker
3 - Make sure that it's possible to revert the migration by executing superset db downgrade and reverting this PR

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

michael-s-molina · 2023-04-19T14:57:23Z

@villebro @kgabryje Can you execute the migration in your staging environments to make sure it runs successfully and the charts are correctly migrated? I executed it successfully at Airbnb.

codecov · 2023-04-19T15:07:16Z

Codecov Report

Merging #23741 (efecd50) into master (522eb97) will increase coverage by 0.06%.
The diff coverage is 100.00%.

❗ Current head efecd50 differs from pull request most recent head 6bd8548. Consider uploading reports for the commit 6bd8548 to get more accurate results

@@            Coverage Diff             @@
##           master   #23741      +/-   ##
==========================================
+ Coverage   68.42%   68.49%   +0.06%     
==========================================
  Files        1951     1946       -5     
  Lines       75472    75373      -99     
  Branches     8219     8218       -1     
==========================================
- Hits        51640    51624      -16     
+ Misses      21720    21638      -82     
+ Partials     2112     2111       -1

Flag	Coverage Δ
hive	`53.49% <ø> (+0.01%)`	⬆️
javascript	`54.91% <100.00%> (+0.08%)`	⬆️
mysql	`79.04% <ø> (+0.03%)`	⬆️
postgres	`79.11% <ø> (+0.03%)`	⬆️
presto	`53.41% <ø> (+0.01%)`	⬆️
python	`82.97% <ø> (+0.03%)`	⬆️
sqlite	`77.64% <ø> (+0.03%)`	⬆️
unit	`53.82% <ø> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ponents/controls/VizTypeControl/VizTypeGallery.tsx	`88.23% <ø> (ø)`
...-frontend/src/visualizations/presets/MainPreset.js	`100.00% <ø> (ø)`
superset/examples/supported_charts_dashboard.py	`0.00% <ø> (ø)`
superset/examples/world_bank.py	`30.66% <ø> (ø)`
superset/viz.py	`62.71% <ø> (+0.40%)`	⬆️
...src/dashboard/components/PropertiesModal/index.tsx	`64.32% <100.00%> (+0.28%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

villebro · 2023-04-19T16:00:32Z

@michael-s-molina please also 🔥 class TreemapViz(BaseViz) in viz.py!

EugeneTorap · 2023-04-19T16:38:31Z

Hello @michael-s-molina @villebro

We can create a discussion where we describe all the legacy chart viz that we want to migrate, as we did for ui pages structure in the src/pages structure proposal.
Do we plan to migrate SQL Lab to SPA in 3.0 version?

michael-s-molina · 2023-04-19T16:45:50Z

Hello @michael-s-molina @villebro

We can create a discussion where we describe all the legacy chart viz that we want to migrate, as we did for ui pages structure in the src/pages structure proposal.

Do we plan to migrate SQL Lab to SPA in 3.0 version?

Hi @EugeneTorap. The Superset 3.0 project answers both questions. There you'll find all legacy viz migrations that were voted and approved for 3.0. SQL Lab to SPA was not proposed for 3.0.

EugeneTorap · 2023-04-19T16:49:30Z

@rusackas Can we add move SQL Lab to SPA to the v3.0 proposal? This is super important for us 🙏

michael-s-molina · 2023-04-19T16:52:28Z

@rusackas Can we add move SQL Lab to SPA to the v3.0 proposal? This is super important for us 🙏

@EugeneTorap There's a 3.0 discussion going on for this type of request.

michael-s-molina · 2023-04-20T13:01:09Z

superset-frontend/cypress-base/cypress/integration/explore/visualizations/treemap.test.js

- * specific language governing permissions and limitations
- * under the License.
- */
-describe('Visualization > Treemap', () => {


I removed the test because it's not applicable anymore. By default, charts are rendered using canvas. Even if we created a test configuration to render the charts using SVG, the groupby colors are not static anymore, they suffer small variations depending on the value. This type of test is more suitable with visual regression testing using Applitools or Chromatic, which is not in the scope of this PR.

john-bodley · 2023-04-20T18:57:21Z

superset/migrations/versions/2023-04-19_11-23_4c5da39be729_migrate_treemap_chart.py

+    # Ensure `slice.params` and `slice.query_context`` in MySQL is MEDIUMTEXT
+    # before migration, as the migration will save a duplicate form_data backup
+    # which may significantly increase the size of these fields.
+    if isinstance(op.get_bind().dialect, MySQLDialect):


It would be great (possibly before) to make sure these columns are defined as a MediumText in the model and had a corresponding migration so we wouldn't need this logic.

@michael-s-molina has there been any update to the underlying model which would then make this redundant?

@john-bodley This migration is referenced by a previous script that was created a while ago. Because of that, we need to keep this piece of code even if the models were updated later.

michael-s-molina · 2023-04-21T17:57:07Z

superset-frontend/cypress-base/cypress/integration/dashboard/editmode.test.ts

@@ -535,12 +535,6 @@ describe('Dashboard edit', () => {
      applyChanges();
      saveChanges();

-      cy.get('.treemap #rect-sum__SP_POP_TOTL').should(


These kind of tests should be handled by Applitools or Chromatic. These selectors don't exist anymore now that we're using canvas to draw the charts.

villebro

LGTM

rusackas · 2023-04-26T15:27:53Z

superset/examples/configs/charts/Commute_Time.yaml

-    operator: '!='
-    sqlExpression: null
-    subject: communite_time
+    - clause: WHERE


Is this not over-indented?

Actually, it does look right - how was this not causing problems before? 😕

superset/viz.py

rusackas

Hooray! Happy to see this thing finally get killed off!!

— Treemap v1

pull-request-size bot added the size/XXL label Apr 19, 2023

michael-s-molina requested review from rusackas, john-bodley, kgabryje and villebro April 19, 2023 14:56

michael-s-molina marked this pull request as ready for review April 19, 2023 14:57

michael-s-molina requested a review from a team as a code owner April 19, 2023 14:57

michael-s-molina added risk:breaking-change Issues or PRs that will introduce breaking changes v3.0 Label added by the release manager to track PRs to be included in the 3.0 branch labels Apr 19, 2023

villebro mentioned this pull request Apr 19, 2023

feat: Migrates Pivot Table v1 to v2 #23712

Merged

9 tasks

michael-s-molina requested review from jinghua-qa, geido, eschutho and betodealmeida as code owners April 20, 2023 12:55

michael-s-molina commented Apr 20, 2023

View reviewed changes

john-bodley reviewed Apr 20, 2023

View reviewed changes

michael-s-molina force-pushed the migrate-treemap branch from e3396c2 to 1eedec7 Compare April 21, 2023 17:54

michael-s-molina commented Apr 21, 2023

View reviewed changes

michael-s-molina force-pushed the migrate-treemap branch from 1eedec7 to fdbcb43 Compare April 21, 2023 19:12

villebro approved these changes Apr 25, 2023

View reviewed changes

michael-s-molina force-pushed the migrate-treemap branch from 0647d15 to 4de9528 Compare April 25, 2023 20:09

michael-s-molina added the hold! On hold label Apr 26, 2023

rusackas reviewed Apr 26, 2023

View reviewed changes

superset/viz.py Show resolved Hide resolved

rusackas approved these changes Apr 26, 2023

View reviewed changes

michael-s-molina force-pushed the migrate-treemap branch 2 times, most recently from 3f24cea to b45898f Compare May 23, 2023 19:38

michael-s-molina force-pushed the migrate-treemap branch 3 times, most recently from 505121d to 1893774 Compare June 6, 2023 19:16

michael-s-molina added 9 commits June 8, 2023 10:17

feat: Migrates TreeMap chart

910379c

UPDATING.md

07762d9

Removes TreemapViz from viz.py

e847fd6

Removes more legacy references

18117fc

Fixes migration down_revision

ed762fa

Updates JSON translation files

dcfce66

Fixes revision order

1e1b5f9

Fixes E2E tests

4a966fc

Rebases

6bd8548

michael-s-molina force-pushed the migrate-treemap branch from 1893774 to 6bd8548 Compare June 8, 2023 13:23

michael-s-molina merged commit af24092 into apache:master Jun 8, 2023
29 checks passed

michael-s-molina removed the v3.0 Label added by the release manager to track PRs to be included in the 3.0 branch label Jun 26, 2023

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.0.0 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Migrates TreeMap chart #23741

feat: Migrates TreeMap chart #23741

michael-s-molina commented Apr 19, 2023 •

edited

michael-s-molina commented Apr 19, 2023

codecov bot commented Apr 19, 2023 •

edited

villebro commented Apr 19, 2023

EugeneTorap commented Apr 19, 2023 •

edited

michael-s-molina commented Apr 19, 2023

EugeneTorap commented Apr 19, 2023

michael-s-molina commented Apr 19, 2023

michael-s-molina Apr 20, 2023

john-bodley Apr 20, 2023

john-bodley May 22, 2023

michael-s-molina May 23, 2023

michael-s-molina Apr 21, 2023

villebro left a comment

rusackas Apr 26, 2023

rusackas Apr 26, 2023

rusackas left a comment

feat: Migrates TreeMap chart #23741

feat: Migrates TreeMap chart #23741

Conversation

michael-s-molina commented Apr 19, 2023 • edited

SUMMARY

AFTER SCREENSHOT

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

michael-s-molina commented Apr 19, 2023

codecov bot commented Apr 19, 2023 • edited

Codecov Report

villebro commented Apr 19, 2023

EugeneTorap commented Apr 19, 2023 • edited

michael-s-molina commented Apr 19, 2023

EugeneTorap commented Apr 19, 2023

michael-s-molina commented Apr 19, 2023

michael-s-molina Apr 20, 2023

Choose a reason for hiding this comment

john-bodley Apr 20, 2023

Choose a reason for hiding this comment

john-bodley May 22, 2023

Choose a reason for hiding this comment

michael-s-molina May 23, 2023

Choose a reason for hiding this comment

michael-s-molina Apr 21, 2023

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

rusackas Apr 26, 2023

Choose a reason for hiding this comment

rusackas Apr 26, 2023

Choose a reason for hiding this comment

rusackas left a comment

Choose a reason for hiding this comment

michael-s-molina commented Apr 19, 2023 •

edited

codecov bot commented Apr 19, 2023 •

edited

EugeneTorap commented Apr 19, 2023 •

edited