fix: share column type matching between model and result set #9161
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #9161 +/- ##
=======================================
Coverage 59.06% 59.06%
=======================================
Files 372 372
Lines 11922 11922
Branches 2919 2919
=======================================
Hits 7042 7042
Misses 4698 4698
Partials 182 182Continue to review full report at Codecov.
|
willbarrett
approved these changes
Feb 18, 2020
john-bodley
reviewed
Feb 19, 2020
villebro
commented
Mar 3, 2020
Comment on lines
+147
to
+168
| # default matching patterns for identifying column types | ||
| db_column_types: Dict[utils.DbColumnType, Tuple[Pattern, ...]] = { | ||
| utils.DbColumnType.NUMERIC: ( | ||
| re.compile(r".*DOUBLE.*", re.IGNORECASE), | ||
| re.compile(r".*FLOAT.*", re.IGNORECASE), | ||
| re.compile(r".*INT.*", re.IGNORECASE), | ||
| re.compile(r".*NUMBER.*", re.IGNORECASE), | ||
| re.compile(r".*LONG.*", re.IGNORECASE), | ||
| re.compile(r".*REAL.*", re.IGNORECASE), | ||
| re.compile(r".*NUMERIC.*", re.IGNORECASE), | ||
| re.compile(r".*DECIMAL.*", re.IGNORECASE), | ||
| re.compile(r".*MONEY.*", re.IGNORECASE), | ||
| ), | ||
| utils.DbColumnType.STRING: ( | ||
| re.compile(r".*CHAR.*", re.IGNORECASE), | ||
| re.compile(r".*STRING.*", re.IGNORECASE), | ||
| ), | ||
| utils.DbColumnType.TEMPORAL: ( | ||
| re.compile(r".*DATE.*", re.IGNORECASE), | ||
| re.compile(r".*TIME.*", re.IGNORECASE), | ||
| ), | ||
| } |
There was a problem hiding this comment.
These regexes essentially do the same as the previous string matches
Before:
"CHAR" in "NVARCHAR"
After:
re.compile(r".*CHAR.*", re.IGNORECASE).match("NVARCHAR")
villebro
commented
Mar 4, 2020
tests/sqla_models_tests.py
Outdated
Comment on lines
29
to
40
| col = TableColumn(column_name="__time", type="INTEGER") | ||
|
|
||
| database = Database(database_name="druid_db", sqlalchemy_uri="druid://db") | ||
| tbl = SqlaTable(table_name="druid_tbl", database=database) | ||
| col = TableColumn(column_name="__time", type="INTEGER", table=tbl) | ||
| self.assertEqual(col.is_dttm, None) | ||
| DruidEngineSpec.alter_new_orm_column(col) | ||
| self.assertEqual(col.is_dttm, True) | ||
|
|
||
| col = TableColumn(column_name="__not_time", type="INTEGER") | ||
| col = TableColumn(column_name="__not_time", type="INTEGER", table=tbl) | ||
| self.assertEqual(col.is_time, False) |
There was a problem hiding this comment.
I intended to refactor out alter_new_orm_column in this PR, but will leave that for a later date to avoid convoluting this fix.
dpgaspar
reviewed
Mar 4, 2020
dpgaspar
approved these changes
Mar 4, 2020
12 tasks
john-bodley
added a commit
to john-bodley/superset
that referenced
this pull request
Apr 24, 2020
john-bodley
added a commit
that referenced
this pull request
Apr 24, 2020
mistercrunch
added
🏷️ bot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CATEGORY
Choose one
SUMMARY
When investigating a bug affecting BigQuery time grains, I noticed that Exploring SQL Lab queries don't correctly detect
TIMEandDATEtypes due to the logic being different in the result set code vs the SqlAlchemy model. This PR adds a method toBaseEngineSpecfor matching database specific column types (e.g.NVARCHAR,DATETIME,BIGINT) to generic types (temporal, numeric and date). In addition, type matching logic inSupersetResultSetis replaced with said logic, ensuring that type inference is uniform across models.TEST PLAN
Test locally + CI (old + new tests)
ADDITIONAL INFORMATION
REVIEWERS