feat: Visualize SqlLab.Query model data in Explore 📈 #20281

hughhhh · 2022-06-06T17:06:19Z

SUMMARY

Explore view now can accept datasource_type = 'query, which represents sql_lab.Query model. Now users will be able to easily chart data from queries being executed in SQL Lab

Screen.Recording.2022-06-10.at.5.39.41.PM.mov

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

Go to Sqllab and run a query
Click Create Chart
Verify that Chart Source is Query with the SQL icon

***OR

Go to Sqllab and run a query
Then run a select id from query
Take one valid id from results returned
Enter the following string into the browser {ephemeral-url}/superset/explore/query/{id}

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

> columns are loading into page

hughhhh · 2022-06-08T15:45:44Z

superset-frontend/packages/superset-ui-core/src/query/DatasourceKey.ts

@@ -29,6 +29,7 @@ export default class DatasourceKey {
    this.id = parseInt(idStr, 10);
    this.type =
      typeStr === 'table' ? DatasourceType.Table : DatasourceType.Druid;
+    this.type = typeStr === 'query' ? DatasourceType.Query : this.type;


make this into a switch statement

you can remove DatasourceType.Druid

…ype is Query [41493] Initial commit to add ability for the a Query Preview Modal to be available when the data source type is Query and not Dataset Converted ModalTrigger to a functional TypeScript component

> allow for all records to be displayed > fix select with all columns queries > filters are now working

AAfghahi · 2022-06-10T16:57:01Z

superset-frontend/src/explore/components/controls/DatasourceControl/index.jsx

@@ -244,7 +244,13 @@ class DatasourceControl extends React.PureComponent {
    return (
      <Styles data-test="datasource-control" className="DatasourceControl">
        <div className="data-container">
-          <Icons.DatasetPhysical className="dataset-svg" />
+          {/* todo(hughhh): move this into a function */}
+          {datasource.type === DatasourceType.Table && (


Is this supposed to be an if and then statement?

AAfghahi · 2022-06-10T16:57:35Z

superset/common/query_context_factory.py

@@ -80,6 +80,11 @@ def create(

    # pylint: disable=no-self-use
    def _convert_to_model(self, datasource: DatasourceDict) -> BaseDatasource:
-        return ConnectorRegistry.get_datasource(
-            str(datasource["type"]), int(datasource["id"]), db.session
+        from superset.dao.datasource.dao import DatasourceDAO


Is there a benefit to importing this in the function versus globally

we are currently getting circuliar dependency issues if we don't wrap the imports in the functions, we have another ticket addressing how to fix this.

AAfghahi · 2022-06-10T16:58:40Z

superset/models/sql_lab.py

-        return self.extra.get("columns", [])
+    def columns(self) -> List[ResultSetColumnType]:
+        # todo(hughhh): move this logic into a base class
+        from superset.utils.core import GenericDataType


Also why import this here?

…hart-power-query

…pes from DatasourceTypes options

…/query-preview-modal-for-chart-source-query

yousoph · 2022-06-13T23:14:52Z

/testenv up

…hart-power-query

…wErrorMessage

pkdotson · 2022-07-15T22:14:56Z

superset-frontend/src/explore/components/ExploreChartPanel.jsx

+  const { useLegacyApi } = metaDataRegistry.get(vizType);
+  const vizTypeNeedsDataset = useLegacyApi && datasource.type !== 'dataset';
+
+  // added boolean column to below show boolean so that the errors aren't overlapping


liking all the comments!

zhaoyongjie · 2022-07-16T03:03:18Z

@hughhhh @yousoph have we really done testing? The explore page of the master branch seems doesn't work at all.

SQLLab -> Explore was broken

explore.is.broken.mov

unable to change datasource

can.not.change.datasource.mov

hughhhh · 2022-07-18T17:10:14Z

@zhaoyongjie I have fixes out for these 2 bugs
#20747
#20751

Antonio-RiveroMartnez · 2022-07-20T12:53:49Z

superset/explore/commands/get.py

@@ -152,11 +153,6 @@ def run(self) -> Optional[Dict[str, Any]]:
        except (SupersetException, SQLAlchemyError):
            dataset_data = dummy_dataset_data

-        if dataset:


@hughhhh DatasourceEditor uses it, without it you would get the Owners is invalid error every time you try to edit a datasource in explore and you would also see undefined undefined as owner when loading its info, so before putting it back I'd like to ask if there was any particular reason why we removed it?

eschutho · 2022-07-25T21:27:30Z

superset/models/sql_lab.py

+
+            computed_column["column_name"] = col.get("name")
+            computed_column["groupby"] = True
+            columns.append(computed_column)


Do you think it would it be easier here in the long run to return a column class? columns.append(Column(computed_column)). Then you wouldn't have to worry about type checking every time you reference the property from a dataset vs a query.

eschutho · 2022-07-25T21:32:10Z

superset/models/sql_lab.py

@@ -167,8 +173,54 @@ def sql_tables(self) -> List[Table]:
        return list(ParsedQuery(self.sql).tables)

    @property
-    def columns(self) -> List[Table]:
-        return self.extra.get("columns", [])
+    def columns(self) -> List[ResultSetColumnType]:


is there a situation where you think we may need a setter on this property?

eschutho · 2022-07-25T21:34:06Z

superset/models/sql_lab.py

+
+    @property
+    def cache_timeout(self) -> int:
+        return 0


the viz class checks if this is None and if not, it will look at the database. Should we set this to None instead?

add POC ExploreMixin

6fc435d

pull-request-size bot added the size/XXL label Jun 6, 2022

hughhhh force-pushed the chart-power-query branch 2 times, most recently from 012e881 to fb58c99 Compare June 6, 2022 20:35

Working POC

632ce21

> columns are loading into page

hughhhh force-pushed the chart-power-query branch from fb58c99 to 632ce21 Compare June 7, 2022 17:37

lyndsiWilliams and others added 3 commits June 7, 2022 16:42

Created/tested query dataset dropdown

6f22618

Add isValidDatasourceType to @superset-ui/core and hide query dropdown

236c489

fix merge conflicts

24261d1

hughhhh force-pushed the chart-power-query branch from 4cb2626 to c8caf51 Compare June 8, 2022 13:29

hughhhh commented Jun 8, 2022

View reviewed changes

Visual updates to explore datasource panel

bd3b6a9

hughhhh force-pushed the chart-power-query branch from c8caf51 to 7d6fd62 Compare June 8, 2022 18:39

lyndsiWilliams and others added 5 commits June 8, 2022 13:53

Temporarily make Query icon visible

64f4181

Remove Query icon visibility

7c07534

Removed isValidDatasourceType check

6fab57f

Added Query preview Modal from DatasourceControl if the data source t…

3a46152

…ype is Query [41493] Initial commit to add ability for the a Query Preview Modal to be available when the data source type is Query and not Dataset Converted ModalTrigger to a functional TypeScript component

> fix integration point with frontend

613d78a

> allow for all records to be displayed > fix select with all columns queries > filters are now working

hughhhh changed the title ~~Chart-power-query~~ feat: Visualize Query model data in Explore 📈 Jun 10, 2022

hughhhh changed the title ~~feat: Visualize Query model data in Explore 📈~~ feat: Visualize SqlLab.Query model data in Explore 📈 Jun 10, 2022

betodealmeida requested review from michael-s-molina and betodealmeida and removed request for michael-s-molina June 10, 2022 17:21

AAfghahi reviewed Jun 10, 2022

View reviewed changes

Merge branch 'master' of https://github.com/preset-io/superset into c…

e076f13

…hart-power-query

hughhhh force-pushed the chart-power-query branch from 7d6fd62 to e076f13 Compare June 13, 2022 20:01

eric-briscoe added 3 commits June 13, 2022 14:01

Adjusts conditional logic approach to be extensible for additional ty…

31dfdd2

…pes from DatasourceTypes options

Merge branch 'chart-power-query' into lyndsi/dataset-panel-updates

bbb91e0

Merge branch 'lyndsi/dataset-panel-updates' into ericbriscoe/sc-41493…

8513cc5

…/query-preview-modal-for-chart-source-query

remove viz error code and addressed more concerns

15d530c

hughhhh force-pushed the chart-power-query branch from f58c753 to 15d530c Compare July 14, 2022 22:23

Merge branch 'master' of https://github.com/preset-io/superset into c…

48f40a6

…hart-power-query

hughhhh requested review from eschutho and AAfghahi July 14, 2022 23:10

hughhhh force-pushed the chart-power-query branch 2 times, most recently from 7a70c8d to 2a57a4f Compare July 15, 2022 00:51

address comments v2

555d26b

hughhhh force-pushed the chart-power-query branch from 2a57a4f to 555d26b Compare July 15, 2022 00:55

AAfghahi and others added 5 commits July 15, 2022 12:17

Merge remote-tracking branch 'origin/chart-power-query' into arash/ne…

cc7a2d0

…wErrorMessage

addressed last comment

eb99ad4

addressed last comment

4fa62e9

failing test

afa34fd

Merge pull request #231 from preset-io/arash/newErrorMessage

c0d7c14

AAfghahi approved these changes Jul 15, 2022

View reviewed changes

Merge branch 'master' into chart-power-query

0666568

pkdotson reviewed Jul 15, 2022

View reviewed changes

hide feature

0e5767b

hughhhh merged commit e5e8867 into apache:master Jul 15, 2022

Antonio-RiveroMartnez reviewed Jul 20, 2022

View reviewed changes

zhaoyongjie mentioned this pull request Jul 21, 2022

feat: the samples endpoint supports filters and pagination #20683

Merged

9 tasks

eschutho reviewed Jul 25, 2022

View reviewed changes

lyndsiWilliams mentioned this pull request Aug 2, 2022

chore(explore): Visual updates to explore datasource panel #20317

Closed

9 tasks

villebro mentioned this pull request Dec 13, 2022

fix(sqla): copy temporal range logic to helper #22405

Merged

9 tasks

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 2.1.0 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Visualize SqlLab.Query model data in Explore 📈 #20281

feat: Visualize SqlLab.Query model data in Explore 📈 #20281

hughhhh commented Jun 6, 2022 •

edited

Loading

hughhhh Jun 8, 2022

eschutho Jun 22, 2022

AAfghahi Jun 10, 2022

AAfghahi Jun 10, 2022

hughhhh Jun 21, 2022

AAfghahi Jun 10, 2022

yousoph commented Jun 13, 2022

pkdotson Jul 15, 2022

zhaoyongjie commented Jul 16, 2022 •

edited

Loading

hughhhh commented Jul 18, 2022

Antonio-RiveroMartnez Jul 20, 2022 •

edited

Loading

eschutho Jul 25, 2022

eschutho Jul 25, 2022

eschutho Jul 25, 2022

feat: Visualize SqlLab.Query model data in Explore 📈 #20281

feat: Visualize SqlLab.Query model data in Explore 📈 #20281

Conversation

hughhhh commented Jun 6, 2022 • edited Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yousoph commented Jun 13, 2022

Choose a reason for hiding this comment

zhaoyongjie commented Jul 16, 2022 • edited Loading

SQLLab -> Explore was broken

unable to change datasource

hughhhh commented Jul 18, 2022

Antonio-RiveroMartnez Jul 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hughhhh commented Jun 6, 2022 •

edited

Loading

zhaoyongjie commented Jul 16, 2022 •

edited

Loading

Antonio-RiveroMartnez Jul 20, 2022 •

edited

Loading