feat: add generic type to column payload #14547

villebro · 2021-05-10T12:51:03Z

SUMMARY

Adds the generic datatype to the dataset column payload to avoid having to parse the raw datatypes in the frontend. Also bump superset-ui to bring in updates to superset-ui/core that are required to make this function properly: apache-superset/superset-ui#1102

BEFORE

Currently the numeric datatypes on the FCC 2018 example dataset show up as unidentified due to the frontend not being able to detect the generic datatype of the raw datatype from the database ("DOUBLE PRECISION"):

AFTER

Now numeric types are being picked up automatically. In the future only datatypes that the db engine spec doesn't identify are shown as question marks:

TEST PLAN

Added datatype tests for Postgres

ADDITIONAL INFORMATION

Has associated issue:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

codecov · 2021-05-10T13:11:12Z

Codecov Report

Merging #14547 (eab110a) into master (31f406a) will not change coverage.
The diff coverage is 77.77%.

❗ Current head eab110a differs from pull request most recent head 8853b1a. Consider uploading reports for the commit 8853b1a to get more accurate results

@@           Coverage Diff           @@
##           master   #14547   +/-   ##
=======================================
  Coverage   77.46%   77.46%           
=======================================
  Files         959      959           
  Lines       48450    48450           
  Branches     5680     5680           
=======================================
  Hits        37532    37532           
  Misses      10718    10718           
  Partials      200      200

Flag	Coverage Δ
javascript	`72.51% <0.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...rontend/src/filters/components/Range/buildQuery.ts	`40.00% <ø> (ø)`
...ers/components/TimeGrain/TimeGrainFilterPlugin.tsx	`0.00% <0.00%> (ø)`
superset/db_engine_specs/base.py	`88.42% <ø> (ø)`
superset/connectors/base/models.py	`90.90% <77.77%> (ø)`
superset/connectors/sqla/models.py	`90.07% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 31f406a...8853b1a. Read the comment docs.

junlincc · 2021-05-10T19:18:07Z

/testenv up

github-actions · 2021-05-10T19:20:16Z

@junlincc Ephemeral environment spinning up at http://54.188.138.197:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

junlincc · 2021-05-10T19:34:04Z

thank you Ville! for some reasons i am still seeing "?" as the icon locally

villebro · 2021-05-11T05:34:04Z

thank you Ville! for some reasons i am still seeing "?" as the icon locally

The question mark won't go away until apache-superset/superset-ui#1102 is released and bumped on the main repo. But since this is not a breaking change I think it makes sense to merge this first to avoid having to make a superset-ui bump coincide with this PR.

junlincc · 2021-05-11T19:49:56Z

gotcha, @zhaoyongjie please help review, i wanna get this PR and apache-superset/superset-ui#1102 in soon. 🙏 @villebro can we get the test pass?

superset/connectors/base/models.py

junlincc · 2021-05-12T16:28:50Z

/testenv up

junlincc · 2021-05-12T17:27:53Z

hmmm still seeing "?" this one seems low risk, i am comfortable of merging if code is good. will look at it right it's in master

github-actions · 2021-05-12T20:01:43Z

@junlincc Ephemeral environment spinning up at http://18.236.153.44:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

superset-frontend/cypress-base/cypress/integration/dashboard/filter.test.ts

villebro · 2021-05-13T05:32:33Z

superset-frontend/cypress-base/cypress/integration/dashboard/dashboard.helper.ts

-    .get(`[data-test="chart-grid-component"][data-test-chart-name="${name}"]`)
+    .get(`[data-test="chart-grid-component"][data-test-chart-name="${name}"]`, {
+      timeout: 30000,
+    })


This was resolving very close to the default 5 secs, and apparently started causing widespread flakiness on cypress. I bumped this to the 30s seen a few lines below, now all tests pass.

zhaoyongjie

LGTM! thanks for the improvment.
Tested in postgres

github-actions · 2021-05-13T06:36:31Z

Ephemeral environment shutdown and build artifacts deleted.

kgabryje · 2021-05-13T06:58:02Z

superset-frontend/spec/javascripts/datasource/fixtures.tsx

@@ -29,7 +29,8 @@ export const columns: ColumnMeta[] = [
    id: 516,
    is_dttm: false,
    python_date_format: null,
-    type: ColumnType.DOUBLE,
+    type: 'DOUBLE',


Why don't we use ColumnType here anymore? Comment applies to all other "ColumnType -> hardcoded string" changes

ColumnType has been removed from superset-ui/core and replaced with string, as it's the raw column type coming back from the database. ColumnType was really just a generalization that mostly complied with ANSI SQL types, but this was not a scalable solution, as the frontend mostly isn't aware of which database is being used. As an example, the datatype DOUBLE PRECISION wasn't being picked up on the frontend due to it being a datatype that's mostly only available on dialects of Postgres. In corner cases two datatypes with the same name can mean different things on different databases (the boolean type on MySQL comes to mind, which is actually an integer). As these are now handled in the backend and mapped to GenericDataType, that's what we're now going to be using for the most part.

* feat: add generic type to column payload * feat: add generic type to column payload * xit flaky test

superset-github-bot bot added the preset-io label May 10, 2021

pull-request-size bot added the size/M label May 10, 2021

villebro force-pushed the villebro/type-generic branch 3 times, most recently from 1e44a08 to 181ab7d Compare May 10, 2021 14:33

pull-request-size bot added size/L and removed size/M labels May 10, 2021

villebro changed the title ~~[WIP] feat: add generic type to column payload~~ feat: add generic type to column payload May 10, 2021

junlincc added the release:note label May 10, 2021

junlincc added the hold:testing! On hold for testing label May 10, 2021

villebro force-pushed the villebro/type-generic branch from 181ab7d to a1625ac Compare May 11, 2021 05:38

junlincc removed the hold:testing! On hold for testing label May 11, 2021

junlincc requested a review from zhaoyongjie May 11, 2021 19:49

zhaoyongjie reviewed May 12, 2021

View reviewed changes

superset/connectors/base/models.py Show resolved Hide resolved

villebro mentioned this pull request May 12, 2021

feat(core): clean up datatype logic apache-superset/superset-ui#1102

Merged

villebro force-pushed the villebro/type-generic branch 4 times, most recently from 6f0615b to b4b79e6 Compare May 12, 2021 16:03

rusackas added the bash! label May 12, 2021

feat: add generic type to column payload

6ad1d31

feat: add generic type to column payload

232e4b2

junlincc added the need:pass-check label May 13, 2021

villebro force-pushed the villebro/type-generic branch from b4b79e6 to 545b954 Compare May 13, 2021 03:52

villebro mentioned this pull request May 13, 2021

feat: bumping echarts plugin, adding new treemap plugin #14560

Merged

8 tasks

villebro commented May 13, 2021

View reviewed changes

superset-frontend/cypress-base/cypress/integration/dashboard/filter.test.ts Outdated Show resolved Hide resolved

villebro force-pushed the villebro/type-generic branch from 545b954 to f7f252f Compare May 13, 2021 04:22

xit flaky test

8853b1a

villebro force-pushed the villebro/type-generic branch from f7f252f to 8853b1a Compare May 13, 2021 05:08

villebro commented May 13, 2021

View reviewed changes

junlincc removed the need:pass-check label May 13, 2021

villebro added the data:connect:postgres Related to Postgres label May 13, 2021

zhaoyongjie approved these changes May 13, 2021

View reviewed changes

villebro merged commit 3f6bd1e into apache:master May 13, 2021

villebro deleted the villebro/type-generic branch May 13, 2021 06:36

kgabryje reviewed May 13, 2021

View reviewed changes

villebro mentioned this pull request May 14, 2021

perf: memoize db_engine_spec in database #14638

Merged

8 tasks

junlincc added rush! Requires immediate attention and removed rush! Requires immediate attention labels May 18, 2021

cccs-RyanS pushed a commit to CybercentreCanada/superset that referenced this pull request Dec 17, 2021

feat: add generic type to column payload (apache#14547)

f33c9c6

* feat: add generic type to column payload * feat: add generic type to column payload * xit flaky test

QAlexBall pushed a commit to QAlexBall/superset that referenced this pull request Dec 29, 2021

feat: add generic type to column payload (apache#14547)

5fc45d2

* feat: add generic type to column payload * feat: add generic type to column payload * xit flaky test

cccs-rc pushed a commit to CybercentreCanada/superset that referenced this pull request Mar 6, 2024

feat: add generic type to column payload (apache#14547)

aeccfc8

* feat: add generic type to column payload * feat: add generic type to column payload * xit flaky test

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.3.0 labels Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add generic type to column payload #14547

feat: add generic type to column payload #14547

villebro commented May 10, 2021 •

edited

Loading

codecov bot commented May 10, 2021 •

edited

Loading

junlincc commented May 10, 2021

github-actions bot commented May 10, 2021

junlincc commented May 10, 2021

villebro commented May 11, 2021

junlincc commented May 11, 2021 •

edited

Loading

junlincc commented May 12, 2021

junlincc commented May 12, 2021 •

edited

Loading

github-actions bot commented May 12, 2021

villebro May 13, 2021 •

edited

Loading

zhaoyongjie left a comment •

edited

Loading

github-actions bot commented May 13, 2021

kgabryje May 13, 2021

villebro May 14, 2021

feat: add generic type to column payload #14547

feat: add generic type to column payload #14547

Conversation

villebro commented May 10, 2021 • edited Loading

SUMMARY

BEFORE

AFTER

TEST PLAN

ADDITIONAL INFORMATION

codecov bot commented May 10, 2021 • edited Loading

Codecov Report

junlincc commented May 10, 2021

github-actions bot commented May 10, 2021

junlincc commented May 10, 2021

villebro commented May 11, 2021

junlincc commented May 11, 2021 • edited Loading

junlincc commented May 12, 2021

junlincc commented May 12, 2021 • edited Loading

github-actions bot commented May 12, 2021

villebro May 13, 2021 • edited Loading

Choose a reason for hiding this comment

zhaoyongjie left a comment • edited Loading

Choose a reason for hiding this comment

github-actions bot commented May 13, 2021

kgabryje May 13, 2021

Choose a reason for hiding this comment

villebro May 14, 2021

Choose a reason for hiding this comment

villebro commented May 10, 2021 •

edited

Loading

codecov bot commented May 10, 2021 •

edited

Loading

junlincc commented May 11, 2021 •

edited

Loading

junlincc commented May 12, 2021 •

edited

Loading

villebro May 13, 2021 •

edited

Loading

zhaoyongjie left a comment •

edited

Loading