Add time grain blacklist and addons to config.py #5380

villebro · 2018-07-12T11:43:38Z

Refactor time grain logic to avoid the use of conflicting names and durations
Harmonize discrepancies in time grains, e.g. P3M (Druid) vs P0.25Y (all others)
Add TIME_GRAIN_BLACKLIST to config.py to be able to remove unwanted time grains, e.g. 5/10/15 mins
Add TIME_GRAIN_ADDONS and TIME_GRAIN_ADDON_FUNCTIONS to config.py to add custom time grains not natively supported by Superset
Add tests for blacklist and validity of defined time grains

codecov-io · 2018-07-12T12:27:45Z

Codecov Report

Merging #5380 into master will increase coverage by 0.03%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #5380      +/-   ##
==========================================
+ Coverage   63.28%   63.31%   +0.03%     
==========================================
  Files         349      349              
  Lines       22121    22141      +20     
  Branches     2457     2457              
==========================================
+ Hits        13999    14019      +20     
  Misses       8108     8108              
  Partials       14       14

Impacted Files	Coverage Δ
superset/models/core.py	`86.95% <100%> (ø)`	⬆️
superset/db_engine_specs.py	`55.26% <100%> (+1.2%)`	⬆️
superset/config.py	`92.48% <100%> (+0.17%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b6cafc...df61c68. Read the comment docs.

mistercrunch · 2018-07-12T14:20:47Z

@betodealmeida

villebro · 2018-07-18T14:04:03Z

Also added functionality to be able to add new custom time grains in config.py (TIME_GRAIN_ADDONS and TIME_GRAIN_ADDON_FUNCTIONS). This could be used to move some less used time grains out of db_engine_specs, while still allowing for end users to define their own custom time grains where needed.

Question: in config.py, which would be preferable: using a blacklist to remove predefined options (e.g. VIZ_TYPE_BLACKLIST), or effectively defining a whitelist (e.g. LANGUAGES)?

villebro · 2018-07-23T11:06:26Z

Comments would be much appreciated @betodealmeida @mistercrunch . This PR would make it easier to manage and add new time grains in production environments. If this PR is not aligned with the future direction of Superset I will be happy make necessary changes or close the PR.

betodealmeida

This looks great, @villebro! It greatly simplifies the definition of new durations and reduces a lot of the duplication.

I have just a few comments on the code, mostly nits, and there's one place where the function is wrong (probably due to an accidental deletion).

Question: in config.py, which would be preferable: using a blacklist to remove predefined options (e.g. VIZ_TYPE_BLACKLIST), or effectively defining a whitelist (e.g. LANGUAGES)?

I think doing a blacklist works better here.

betodealmeida · 2018-07-23T18:15:04Z

superset/db_engine_specs.py

+    'P1Y': 'year',
+    '1969-12-28T00:00:00Z/P1W': 'week_start_sunday',
+    '1969-12-29T00:00:00Z/P1W': 'week_start_monday',
+    'P1W/1970-01-03T00:00:00Z': 'week_ending_saturday',


Can you also add 'P1W/1970-01-04T00:00:00Z': 'week_ending_sunday', here, for completeness?

Good idea, done.

betodealmeida · 2018-07-23T18:22:01Z

superset/db_engine_specs.py

+
+
+def _create_time_grains_tuple(time_grains, time_grain_functions, blacklist):
+    ret_list = list()


Nit: use [] to instantiate an empty list. It's negligibly faster, but the preferred style for Superset.

Thanks, will try to remember this in the future.

betodealmeida · 2018-07-23T18:25:04Z

superset/db_engine_specs.py

+    blacklist = blacklist if blacklist else []
+    for duration, func in time_grain_functions.items():
+        if duration not in blacklist:
+            name = time_grains.get(duration, None)


Nit: {}.get already returns None when the key is not present, so this can be written simply as:

name = time_grains.get(duration)

Good point.

betodealmeida · 2018-07-23T18:25:21Z

superset/db_engine_specs.py

@@ -65,12 +94,22 @@ class BaseEngineSpec(object):
    """Abstract class for database engine specific configurations"""

    engine = 'base'  # str as defined in sqlalchemy.engine.engine
-    time_grains = tuple()
+    time_grain_functions = dict()


Nit: Use {} when instantiating an empty dictionary.

betodealmeida · 2018-07-23T18:26:09Z

superset/db_engine_specs.py

+        'P1M': "DATE_TRUNC('month', {col}) AT TIME ZONE 'UTC'",
+        'P0.25Y': "DATE_TRUNC('quarter', {col}) AT TIME ZONE 'UTC'",
+        'P1Y': "DATE_TRUNC('year', {col}) AT TIME ZONE 'UTC'",
+    }


Nice! This is much cleaner now. Love it!

betodealmeida · 2018-07-23T18:47:59Z

superset/db_engine_specs.py

@@ -52,6 +52,35 @@

 Grain = namedtuple('Grain', 'name label function duration')


I think we can get rid of the name attribute in Grain. I checked the code and it seems like it's not being used anywhere, and grains can be uniquely identified by the duration now that they're ISO durations.

Good point, I misremembered that core.grains_dict() created a dict that mapped duration -> grain and name -> grain, but the latter was in fact label -> grain. I've now removed name.

betodealmeida · 2018-07-23T18:52:30Z

superset/db_engine_specs.py

+        grain_functions = cls.time_grain_functions.copy()
+        grain_addon_functions = config.get('TIME_GRAIN_ADDON_FUNCTIONS', {})
+        grain_functions.update(grain_addon_functions.get(cls.engine, {}))
+        return _create_time_grains_tuple(grains, grain_functions, blacklist)


I think it might be better to call this get_time_grains, in order to be more explicit and also because it has changed from an attribute to a method.

betodealmeida · 2018-07-23T18:53:18Z

superset/db_engine_specs.py

-    )
+    time_grain_functions = {
+        None: '{col}',
+        'PT1S': "{col}) AT TIME ZONE 'UTC'",


'PT1S': "DATE_TRUNC('second', {col}) AT TIME ZONE 'UTC'",

Whoops, good catch!

betodealmeida · 2018-07-23T18:57:45Z

superset/db_engine_specs.py

+        None: '{col}',
+        'PT1S': "DATE_TRUNC('SECOND', {col})",
+        'PT1M': "DATE_TRUNC('MINUTE', {col})",
+        'PT5M': "DATEADD(MINUTE, FLOOR(DATE_PART(MINUTE, {col}) / 5) * 5, \


Nice! Were you able to test these against Snowflake?

Yes; not sure if it's the most efficient way of doing it, but managed to get good performance on a semi-large dataset, so should be good to go.

villebro

All proposed changes implemented.

villebro · 2018-07-24T13:08:05Z

@betodealmeida I tried to remove name from Grain and updating time_column_grains() and data() in connectors/sqla/models.py, but got slightly overwhelmed by the frontend code. For instance, controls.jsx seems to have similar but slightly different grains, and am unsure where these definitions are used:

https://github.com/apache/incubator-superset/blob/bea0a0aa1590d073365ecf74d75a054643dbe3b7/superset/assets/src/explore/controls.jsx#L750-L776

I'm sure I will be able to sort out how the frontend logic works, but it will take time, so I propose merging this and removing name in a separate PR.

betodealmeida · 2018-07-24T18:26:40Z

@villebro, no worries, I'll take a stab at removing name from Grain.

betodealmeida

Awesome! Thank you so much for doing this, @villebro!

villebro · 2018-07-24T19:18:41Z

Great, thanks @betodealmeida !

villebro · 2018-07-28T07:34:34Z

@betodealmeida FYI rebased to resolve conflict, any chance of getting this merged out of the way?

betodealmeida · 2018-07-30T16:32:47Z

@mistercrunch, can you merge this? I still don't have merging privilege.

@betodealmeida

* Add interim grains * Refactor and add blacklist * Change PT30M to PT0.5H * Linting * Linting * Add time grain addons to config.py and refactor engine spec logic * Remove redundant import and clean up config.py * Fix bad rebase * Implement changes proposed by @betodealmeida * Revert removal of name from Grain * Linting

villebro changed the title ~~Add time grain blacklist and tests for validity~~ Add time grain blacklist and addons to config.py Jul 21, 2018

betodealmeida requested changes Jul 23, 2018

View reviewed changes

villebro commented Jul 24, 2018

View reviewed changes

betodealmeida approved these changes Jul 24, 2018

View reviewed changes

villebro added 11 commits July 28, 2018 09:51

Add interim grains

d77bb1b

Refactor and add blacklist

e996c83

Change PT30M to PT0.5H

36fbbb9

Linting

5ee77ed

Linting

8da5b1a

Add time grain addons to config.py and refactor engine spec logic

4ec96d5

Remove redundant import and clean up config.py

afdc590

Fix bad rebase

381e65f

Implement changes proposed by @betodealmeida

34f7f82

Revert removal of name from Grain

5e2f8a4

Linting

df61c68

mistercrunch merged commit c1e6c68 into apache:master Jul 31, 2018

villebro mentioned this pull request Dec 16, 2018

Refactor teradata to new time_grain_functions spec #6539

Merged

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.28.0 labels Feb 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add time grain blacklist and addons to config.py #5380

Add time grain blacklist and addons to config.py #5380

villebro commented Jul 12, 2018 •

edited

Loading

codecov-io commented Jul 12, 2018 •

edited

Loading

mistercrunch commented Jul 12, 2018

villebro commented Jul 18, 2018

villebro commented Jul 23, 2018 •

edited

Loading

betodealmeida left a comment

betodealmeida Jul 23, 2018

villebro Jul 24, 2018

betodealmeida Jul 23, 2018

villebro Jul 24, 2018

betodealmeida Jul 23, 2018

villebro Jul 24, 2018

betodealmeida Jul 23, 2018

villebro Jul 24, 2018

betodealmeida Jul 23, 2018

villebro Jul 24, 2018

betodealmeida Jul 23, 2018

villebro Jul 24, 2018

betodealmeida Jul 23, 2018

villebro Jul 24, 2018

betodealmeida Jul 23, 2018

villebro Jul 24, 2018

betodealmeida Jul 23, 2018

villebro Jul 24, 2018

villebro left a comment

villebro commented Jul 24, 2018

betodealmeida commented Jul 24, 2018

betodealmeida left a comment

villebro commented Jul 24, 2018

villebro commented Jul 28, 2018

betodealmeida commented Jul 30, 2018



		def _create_time_grains_tuple(time_grains, time_grain_functions, blacklist):
		ret_list = list()

		@@ -52,6 +52,35 @@

		Grain = namedtuple('Grain', 'name label function duration')

Add time grain blacklist and addons to config.py #5380

Add time grain blacklist and addons to config.py #5380

Conversation

villebro commented Jul 12, 2018 • edited Loading

codecov-io commented Jul 12, 2018 • edited Loading

Codecov Report

mistercrunch commented Jul 12, 2018

villebro commented Jul 18, 2018

villebro commented Jul 23, 2018 • edited Loading

betodealmeida left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

villebro commented Jul 24, 2018

betodealmeida commented Jul 24, 2018

betodealmeida left a comment

Choose a reason for hiding this comment

villebro commented Jul 24, 2018

villebro commented Jul 28, 2018

betodealmeida commented Jul 30, 2018

villebro commented Jul 12, 2018 •

edited

Loading

codecov-io commented Jul 12, 2018 •

edited

Loading

villebro commented Jul 23, 2018 •

edited

Loading