[SIP-65] Enhance, replace and deprecate data viz in Superset

[junlincc]

Note: This is 3/3 of a greater proposed effort in Data Visualization Enhancement, Reorganization and Revamp

Background
As of today, Superset has grown from a hackathon project to a full-fledged BI tool with over 50 out-of-the-box visualization plugins offered. Users appreciate Superset for providing a variety of visualization options, though the organic growth of the product throughout the years has led to increasing difficulty in navigating in the gallery page, and creating visualizations efficiently.

Goal
Posting 3 SIPs in related areas, and the shared end goal of all SIPs 65, 66, and 67 is to improve the time-to-value in creating meaningful charts in Superset.

• Less disorganized and stand alone chart types
• Cleaner Gallery modal and control layout
• More powerful charts!

Problem 3 Maintenance of legacy charts

During the last few rounds of chart audits, we realized that more than two thirds of the existing viz plug-ins have issues, functional or cosmetic, with some more critical than others. Libraries like NVD3 that we used for years may no longer fulfill our increasing enhancement needs. Many original contributors of customized charts are no longer active in the project. Without adequate documentation and proper knowledge transfer, maintaining customized legacy charts has also become harder. On the premise of ensuring that users can still achieve the same charting goals, we will deprecate those which are too difficult to maintain and replace them with a better option, for a more strategic long term investment.

Proposed Solutions for problem 3 Enhance, replace and deprecate

Solution: We come up with plan based on multiple chart audits, user feedback, chart usage and condition, and the core committer’s observation.

1. Functionalities we plan to add to each chart,
2. Plug-ins and libraries that we no longer support
3. Plug-ins that we are removing from Superset Gallery.
   There are different ways to approach deprecating charts, and each has its pros and cons:
   a. Burry the plugin behind a "legacy", "experimental" or "deprecated" category
   Pros: Minimal engineering effort, no change in ownership, least impact to all users
   Cons: Interaction with available broken charts leads to unpleasant UX
   b. Plugin off by default, new superset environments won't show the viz unless it's explicitly switch on by an admin
   Pros: Viz is retained and hidden, better UX
   Cons: Moderate impact on admin users and to releases
   c.moving the code out of our repo(s), offer for people to load as a dynamic plugin
   Pros: Improve Superset stability in general
   Cons: High engineering effort, a mature dynamic plugin system and UI solution are need, implementation takes longer
   d. fully deprecate, drop the code and offer no path forward
   Pros: Minimal engineering effort
   Cons: Impact small amount of users heavily

Chart	Condition	Usage	Library	Extensibility	Proposed Solution
Line Chart	Stable	Very High	NVD3	Poor	Migrate to Echarts(Line)Add dynamic control and non temporal X-axis support
Bar Chart	Stable	Very High	NVD3	Poor	Migrate to Echarts(Bar)Add dynamic control and non temporal X-axis support
Time-Series Bar Chart	Stable	Medium	NVD3	Poor	Migrate to Echarts(Bar)Integrate with Bar Chart after its Echarts migration
Area Chart	Stable	High	NVD3	Poor	Migrate to Echarts(Stream)Add dynamic control and non temporal X-axis support
Histogram	Stable	Medium	NVD3	Poor	Migrate to Echarts; Integrate with Bar chart after Echarts migrationAdd control to convert from Bar to Histogram; both could have separate entrypoint as well as in-place chart-type switch
Sunburst Chart	Stable	Medium	Custom	Poor	Migrate to Echarts(Sunburst)
Treemap	Unstable	Medium	NVD3	Poor	Migrate to Echarts(Treemap)
Sankey Diagram	Stable	Low	Custom	Poor	Migrate to Echarts(Sankey)
Heatmap	Unstable	Medium	NVD3?	Poor	Migrate to Echarts(Heatmap)
Calendar Heatmap	Unstable	Low	Custom	Poor	Migrate to Echarts(Heatmap)Add control to support Calendar Heatmap in Heatmap after migration; both have separate entrypoint as well as in-place chart-type switch
Bubble Chart	Stable	Low	NVD3	Poor	Migrate to Echarts(Scatter)Add control to set Size and Color in Echarts Scatter Plot; both could have separate entrypoint as well as in-place chart-type switch
World Map	Stable	Medium	Custom	OK	Migrate to Echarts(Map)
Nightingale Rose Chart	Stable	Medium	Custom	OK	Migrate to Echarts(Nightingale), both could have separate entrypoint as well as in-place chart-type switch
Chord Diagram	Stable	Low	Custom	OK	Migrate to Echarts(Chord)
Bullet Chart	Unstable	Low	NVD3	Poor	Migrate to Echarts(Gague) and remove from the gallery
Time-series Chart	Stable	High	Echarts	Good	Break down into separate chart types. and time series analytic functionality will be added to the line, bar, scatter, step etc.
Table	Improving	Very High	Custom	OK	Enhance and share the same entry point with Pivot Table, both could have separate entrypoint as well as in-place chart-type switch
Pivot table	Migrating	High	Custom	Poor	Migrate to React-pivottable.js. both could have separate entrypoint as well as in-place chart-type switch
Dual Line Chart	Unstable	Low	Custom	OK	Move to Echarts(combo) and remove it from the galleryBe a variation of Multi Y-axis combo chart; Deprecate
Pie Chart	Stable	Very High	Echarts	Good	Stabilize and enhance
Box Plot	Stable	Medium	Echarts	Good	Stabilize and enhance
Graph Chart	Stable	Low	Echarts	Good	Stabilize and enhance
Word Cloud	Stable	Medium	Custom	OK	Stabilize and enhance
Big Number	Stable	Very High	Custom	OK	Enhance and combine, add dynamic control for trendline option
BigNumber w Trendline	Stable	Very High	Custom	OK	Enhance and combine, same entry point as BigNumber
MapBox	Unstable	Medium	Custom	OK	Stabilize
Deck.gl Scatterplot(2D)	Unstable	Medium	Custom	OK	Group under the same category, each display as stand-alone chart type within the 2D Deck.gl category.
Deck.gl Screen Grid(2D)	Unstable	Low	Custom	OK
Deck.gl Geojson(2D)	Unstable	Low	Custom	Poor
Deck.gl Path(2D)	Unstable	Low	Custom	OK
Deck.gl Multiple Layers(2D)	Unstable	Very Low	Custom	Poor
Deck.gl Arc(2D)	Unstable	Very Low	Custom	OK
Deck.gl Polygon(3D)	Unstable	Low	Custom	OK	Group under the same category, each display as stand alone chart type within the 3D Deck.gl category.
Deck.gl Grid(3D)	Unstable	Low	Custom	OK
Deck.gl 3D Hexagon(3D)	Unstable	Low	Custom	OK
Time-series Percentage Change	Unstable	Very Low	Custom	Poor	Add a control option to Echarts Line chart when temporal column is selected as X-axis
Time-series Table	Stable	Medium	Custom	Poor	Move behind feature flag as it, volunteer chart maintainer is needed
Partition Chart	Unstable	Very Low	Custom	Poor	Move behind feature flag as it, volunteer chart maintainer is needed
Horizon Chart	Unstable	Very Low	Custom	Poor	Move behind feature flag as it, volunteer chart maintainer is needed
EventFlow	Broken	Very Low	Custom	OK	Move behind feature flag as it, volunteer chart maintainer is needed
Pair t-test Table	Broken	Very Low	Custom	Poor	Move behind feature flag as it, volunteer chart maintainer is needed
Parallel Coordinates	Stable	Very Low	Custom	Poor	Move behind feature flag as it, volunteer chart maintainer is needed
Time-series Period Pivot	Broken	Very Low	NVD2	Poor	Remove
Multi Line Charts	Stable	Low	NVD3	Poor	Remove, replaced by new Echarts combination chart
Filterbox	Stable	Very High	Custom	Poor	Migrate to native filter & Deprecate

Thanks for brainstorming: @villebro @mistercrunch @rusackas 🙏

[zufolo441]

I don't see in this list the country map chart, that we heavily use. We created more than 2.000 different territories to be used in charts.....

[andrewscarani]

Please keep deck.gl and make it more stable!

[andrewscarani]

Also, it would be great to consolidate all time-series chart functionality into a single chart type. It's so confusing having to select different versions rather than just select "time-series" and allow you to pick datasets for each series and have the full capabilities around period-over-period, forecasting, multiple Y-axises, etc. It would be great to get it to a place where it works more like Tableau.

[anadue]

We see deck.gl as a critical differentiator for superset vs other solutions. Please keep it and make it an integral part of the solution. I expect deck.gl usage will increase as analytics in general takes more location intelligence into account. Not many other solutions have a proper geospatial component, please consider deck.gl to be an important future component.

[riazarbi]

Hello, just adding a +1 for Deck.gl. Usage is probably low because it is unstable. We also view it as a critical differentiator. We'd also like to add support for deck.gl H3Hexagon maps - happy to do the legwork if I can get some guidance on how to go about it.

[zgramana]

+1 For deck.gl Multiple Layers

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.

[anadue]

We see deck.gl as a critical differentiator for superset vs other solutions. Please keep it and make it an integral part of the solution. I expect deck.gl usage will increase as analytics in general takes more location intelligence into account. Not many other solutions have a proper geospatial component, please consider deck.gl to be an important future component.

Has there been any movement in terms of improving support for deck.gl in Superset?

[txjmb]

Deck.gl LineLayer would be a very important add. Not every line on a map is an arc or a path.

[rusackas]

@junlincc is there intent to carry this forward to a vote? If so, it probably needs substantial updates. My personal feeling is that charts that are effectively replaced by newer (ECharts) variations can be considered breaking changes and rolled into the 3.0 voting/release process.

[rusackas]

I think we can close this out for a couple of reasons:

1. the thread and the discussion have gone cold and it doesn't sound like there's an intent to put it up for a vote?

2. The general theme/idea here is to deprecate and remove old charts in favor of new ones... I think this is not terribly controversial as long as the feature parity is reasonable enough to make the breaking change in any major version release. I believe there are a few that are ripe for this, and this'll be part of the 3.0 discussion as that draws near.

If anyone feels differently, of course anyone is welcome re-open this, update the proposal's specifics, and rekindle the discussion on the dev mailing list.