Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Add Druid SQL time grains for parity with Druid NoSQL #15320

Conversation

john-bodley
Copy link
Member

@john-bodley john-bodley commented Jun 23, 2021

SUMMARY

Adding Druid SQL time grains for parity with Druid NoSQL per here. Note the Druid NoSQL connector supports simple natural language whereas the SQL connector does not.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

AFTER

Screen Shot 2021-06-23 at 10 22 16 AM

Screen Shot 2021-06-23 at 10 22 25 AM

TESTING INSTRUCTIONS

Verified the SQL logic adhered to other database engine specifications.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@codecov
Copy link

codecov bot commented Jun 23, 2021

Codecov Report

Merging #15320 (454f831) into master (b295c6a) will decrease coverage by 0.15%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #15320      +/-   ##
==========================================
- Coverage   77.16%   77.01%   -0.16%     
==========================================
  Files         973      973              
  Lines       50523    50523              
  Branches     6184     6184              
==========================================
- Hits        38986    38909      -77     
- Misses      11331    11408      +77     
  Partials      206      206              
Flag Coverage Δ
hive ?
mysql 81.70% <ø> (ø)
postgres 81.72% <ø> (ø)
python 81.81% <ø> (-0.30%) ⬇️
sqlite 81.36% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/db_engine_specs/base.py 87.97% <ø> (-0.41%) ⬇️
superset/db_engine_specs/druid.py 88.63% <ø> (ø)
superset/db_engines/hive.py 0.00% <0.00%> (-82.15%) ⬇️
superset/db_engine_specs/hive.py 69.20% <0.00%> (-17.21%) ⬇️
superset/views/database/mixins.py 81.03% <0.00%> (-1.73%) ⬇️
superset/db_engine_specs/presto.py 83.36% <0.00%> (-1.06%) ⬇️
superset/connectors/sqla/models.py 88.22% <0.00%> (-0.24%) ⬇️
superset/utils/core.py 88.93% <0.00%> (-0.13%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b295c6a...454f831. Read the comment docs.

@john-bodley john-bodley force-pushed the john-bodley--druid-sql-time-grain-expressions branch from f8318d2 to 454f831 Compare June 23, 2021 17:23
@john-bodley
Copy link
Member Author

john-bodley commented Jun 23, 2021

@villebro would you mind re-reviewing this PR? I added all the additional time grains (not just week starting/ending) which have not previously been defined or the SQL connector. Let me know if this should be added in db_engine_specs/base.py (as defined) or whether db_engine_specs/druid.py should override the get_time_grains() method.

Furthermore I'm not sure if additional work is required, i.e., for Pandas post processing per here. Additionally I'm not sure if the custom time grain configuration logic actually works if these need to be defined within the post processing as mentioned above, i.e., can Superset really support custom time grains?

@villebro
Copy link
Member

@villebro would you mind re-reviewing this PR? I added all the additional time grains (not just week starting/ending) which have not previously been defined or the SQL connector. Let me know if this should be added in db_engine_specs/base.py (as defined) or whether db_engine_specs/druid.py should override the get_time_grains() method.

I think this is fine. We have plans to introduce freeform timegrains (where easily supported), via regex or other pattern matching schemes. This would make it possible to do arbitrary time grains, like PT7M. Most engines support this fairly easily, but for others that are more restricted we would only support the built-in ones that have to be explicitly written out.

Furthermore I'm not sure if additional work is required, i.e., for Pandas post processing per here. Additionally I'm not sure if the custom time grain configuration logic actually works if these need to be defined within the post processing as mentioned above, i.e., can Superset really support custom time grains?

I think it's ok, at least for now, to just support the built-in time grains in the forecasting feature. But here we could probably fairly easily also support more custom increments via regex. I'd be happy to add this as soon as someone requests it or when we start working on the arbitrary precision time grain feature.

Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@john-bodley john-bodley merged commit accee50 into apache:master Jun 25, 2021
@john-bodley john-bodley deleted the john-bodley--druid-sql-time-grain-expressions branch June 25, 2021 14:21
cccs-RyanS pushed a commit to CybercentreCanada/superset that referenced this pull request Dec 17, 2021
…5320)

Co-authored-by: John Bodley <john.bodley@airbnb.com>
QAlexBall pushed a commit to QAlexBall/superset that referenced this pull request Dec 29, 2021
…5320)

Co-authored-by: John Bodley <john.bodley@airbnb.com>
cccs-rc pushed a commit to CybercentreCanada/superset that referenced this pull request Mar 6, 2024
…5320)

Co-authored-by: John Bodley <john.bodley@airbnb.com>
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.3.0 labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/S 🚢 1.3.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants