[db-engine-spec] Aligning Hive/Presto partition logic #7007

john-bodley · 2019-03-11T18:00:22Z

This PR remedies a couple of issues related to Hive/Presto partitions. Specifically:

For Hive there was no check to see whether the Pandas Dataframe was empty and thus an exception was raised when trying to obtain the _latest_partition_from_df. For context this same issue was resolved for Presto in [sql lab] fix partitionned table has no partitions #2740.
Added a check that if the partition value is explicitly None to ensure that the query isn't mutated to include a non-valid WHERE clause. The methods simply return False (which is somewhat of a strange return value) which is consistent with other invalid scenarios.

Note for (1) this wasn't an issue for where_latest_partition as the exception was caught, however it was problematic for the extra_table_metadata which is used in SQL Lab for fetching the preview.

to: @michellethomas @mistercrunch

codecov-io · 2019-03-11T19:47:18Z

Codecov Report

Merging #7007 into master will decrease coverage by <.01%.
The diff coverage is 55.55%.

@@            Coverage Diff             @@
##           master    #7007      +/-   ##
==========================================
- Coverage   64.46%   64.45%   -0.01%     
==========================================
  Files         421      421              
  Lines       20521    20523       +2     
  Branches     2245     2245              
==========================================
  Hits        13229    13229              
- Misses       7165     7167       +2     
  Partials      127      127

Impacted Files	Coverage Δ
superset/db_engine_specs.py	`56.88% <55.55%> (-0.15%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6a4d507...f1d2c1e. Read the comment docs.

mistercrunch

LGTM

(cherry picked from commit 05be866)

* Sparkline dates aren't formatting in Time Series Table (#6976) * Exclude venv for python linter to ignore * Fix NaN error * Fix the white background shown in SQL editor on drag (#7021) This PR sets the background-color css property on `.ace_scroller` instead of `.ace_content` to prevent the white background shown during resizing of the SQL editor before drag ends. * Show tooltip with time frame (#6979) * Fix time filter control (#6978) * Enhancement of query context and object. (#6962) * added more functionalities for query context and object. * fixed cache logic * added default value for groupby * updated comments and removed print (cherry picked from commit d5b9795) * [fix] /superset/slice/id url is too long (#6989) (cherry picked from commit 6a4d507) * [WIP] fix user specified JSON metadata not updating dashboard on refresh (#7027) (cherry picked from commit cc58f0e) * feat: add ability to change font size in big number (#7003) * Add ability to change font sizes in Big Number * rename big number to header * Add comment to clarify font size values * Allow LIMIT to be specified in parameters (#7052) * [fix] Cursor jumping when editing chart and dashboard titles (#7038) (cherry picked from commit fc1770f) * Changing time table viz to pass formatTime a date (#7020) (cherry picked from commit 7f3c145) * [db-engine-spec] Aligning Hive/Presto partition logic (#7007) (cherry picked from commit 05be866) * [fix] explore chart from dashboard missed slice title (#7046) (cherry picked from commit a6d48d4) * fix inaccurate data calculation with adata rolling and contribution (#7035) (cherry picked from commit 0782e83) * Adding warning message for sqllab save query (#7028) (cherry picked from commit ead3d48) * [datasource] Ensuring consistent behavior of datasource editing/saving. (#7037) * Update datasource.py * Update datasource.py (cherry picked from commit c771625) * [csv-upload] Fixing message encoding (#6971) (cherry picked from commit 48431ab) * [sql-parse] Fixing LIMIT exceptions (#6963) (cherry picked from commit 3e076cb) * Adding custom control overrides (#6956) * Adding extraOverrides to line chart * Updating extraOverrides to fit with more cases * Moving extraOverrides to index.js * Removing webpack-merge in package.json * Fixing metrics control clearing metric (cherry picked from commit e619405) * [sqlparse] Fixing table name extraction for ill-defined query (#7029) (cherry picked from commit 07c340c) * [missing values] Removing replacing missing values (#4905) (cherry picked from commit 61add60) * [SQL Lab] Improved query and results tabs rendering reliability (#7082) closes #7080 (cherry picked from commit 9b58e9f) * Fix filter_box migration PR #6523 (#7066) * Fix filter_box migration PR #6523 * Fix druid-related bug (cherry picked from commit b210742) * SQL editor layout makeover (#7102) This PR includes the following layout and css tweaks: - Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox - Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane - Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars - css tweaks on the basic.html page so the loading animation appears in the center of the page across the board (cherry picked from commit 71f1bbd) * [forms] Fix handling of NULLs (cherry picked from commit e83a07d) * handle null column_name in sqla and druid models (cherry picked from commit 2ff721a) * Use metric name instead of metric in filter box (#7106) (cherry picked from commit 003364e) * Bump python lib croniter to an existing version (#7132) Package maintainers should really never delete packages, but it appears this happened with croniter and resulted in breaking our builds. This PR bumps to a more recent existing version of the library (cherry picked from commit 215ed39) * Revert PR #6933 (#7162) * [bugfix] SQL Lab 'Filter Results' doesn't stick (#7104) When using a "Search Results" criteria, the subset of rows that match the criteria get displayed. While this the filter is applied, if another query is run, the filter is still active, but not displayed in the input text box. After this change, the state of the input box sticks after subsequent queries. (cherry picked from commit d5e8d66) * Injectable statsd client (#7138) * Add ability to inject statsd client; some py test/reqs updates - Updated the metrics logger to allow construction with an existing statsd client, so that it can be configured by external systems or libs. - added requirements to requirements-dev.txt which are needed to run tests-eg coverage, nose - removed dependency on mock lib, it is in python stdlib now - updated tox.ini to remove the now-superfluous deps * add license to test file, and remove blank line at EOF (cherry picked from commit ba19a62) * [Lyft-GA] Enable color consistency in a dashboard (#7135) * Enable color consistency in a dashboard Moved actions, minor UI, allowed dashboard copy Fix linting errors Undo unintentional change Updated and added unit tests Fail quietly if package has not been updated Fail quietly on dashboard copy if package is old * Update packages * Remove unnecessary code * Addressed Grace's comments * Small fix for item key * Reset chart's color during exploration * Do not reset chart form data when exploring chart * Fix double scroll bars when content of sql result table overflows horizontally (#7168) The PR substracts the scrollbar height from the height of the container of the react virtualized table so we don't see double scrollbars. (cherry picked from commit 7ffcabd) * Change number format default * Use smart formatter instead * fix merge issues * Use SMART_NUMBER

…equests (#7032) * Sparkline dates aren't formatting in Time Series Table (#6976) * Exclude venv for python linter to ignore * Fix NaN error * Fix the white background shown in SQL editor on drag (#7021) This PR sets the background-color css property on `.ace_scroller` instead of `.ace_content` to prevent the white background shown during resizing of the SQL editor before drag ends. * Show tooltip with time frame (#6979) * Fix time filter control (#6978) * Enhancement of query context and object. (#6962) * added more functionalities for query context and object. * fixed cache logic * added default value for groupby * updated comments and removed print (cherry picked from commit d5b9795) * [fix] /superset/slice/id url is too long (#6989) (cherry picked from commit 6a4d507) * [WIP] fix user specified JSON metadata not updating dashboard on refresh (#7027) (cherry picked from commit cc58f0e) * feat: add ability to change font size in big number (#7003) * Add ability to change font sizes in Big Number * rename big number to header * Add comment to clarify font size values * Allow LIMIT to be specified in parameters (#7052) * [fix] Cursor jumping when editing chart and dashboard titles (#7038) (cherry picked from commit fc1770f) * Changing time table viz to pass formatTime a date (#7020) (cherry picked from commit 7f3c145) * [db-engine-spec] Aligning Hive/Presto partition logic (#7007) (cherry picked from commit 05be866) * [fix] explore chart from dashboard missed slice title (#7046) (cherry picked from commit a6d48d4) * fix inaccurate data calculation with adata rolling and contribution (#7035) (cherry picked from commit 0782e83) * Adding warning message for sqllab save query (#7028) (cherry picked from commit ead3d48) * [datasource] Ensuring consistent behavior of datasource editing/saving. (#7037) * Update datasource.py * Update datasource.py (cherry picked from commit c771625) * [csv-upload] Fixing message encoding (#6971) (cherry picked from commit 48431ab) * [sql-parse] Fixing LIMIT exceptions (#6963) (cherry picked from commit 3e076cb) * Adding custom control overrides (#6956) * Adding extraOverrides to line chart * Updating extraOverrides to fit with more cases * Moving extraOverrides to index.js * Removing webpack-merge in package.json * Fixing metrics control clearing metric (cherry picked from commit e619405) * [sqlparse] Fixing table name extraction for ill-defined query (#7029) (cherry picked from commit 07c340c) * [missing values] Removing replacing missing values (#4905) (cherry picked from commit 61add60) * [SQL Lab] Improved query and results tabs rendering reliability (#7082) closes #7080 (cherry picked from commit 9b58e9f) * Fix filter_box migration PR #6523 (#7066) * Fix filter_box migration PR #6523 * Fix druid-related bug (cherry picked from commit b210742) * SQL editor layout makeover (#7102) This PR includes the following layout and css tweaks: - Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox - Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane - Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars - css tweaks on the basic.html page so the loading animation appears in the center of the page across the board (cherry picked from commit 71f1bbd) * [forms] Fix handling of NULLs (cherry picked from commit e83a07d) * handle null column_name in sqla and druid models (cherry picked from commit 2ff721a) * Use metric name instead of metric in filter box (#7106) (cherry picked from commit 003364e) * Bump python lib croniter to an existing version (#7132) Package maintainers should really never delete packages, but it appears this happened with croniter and resulted in breaking our builds. This PR bumps to a more recent existing version of the library (cherry picked from commit 215ed39) * Revert PR #6933 (#7162) * Add decorator for etag cache * Fetch charts with GET * Small fixes * Fix typo * Compute correct cache key; fix logging * Check perms on cached response * Revert change * If perms fail, return naked response * Fix lint * Compute cache key from all form data * Pass extra_filters in GET request * Fix pylint * Fix flake8 * Use ETags even if no cache is set * Handle adhoc filters * Raise in debug mode * Rename actions * Fix integration tests * Do POST request on new charts * Set extra/adhoc filters only in GET requests * Raise if check_perms fails * Refactor auth * Fix flake8 * Fix js unit tests * Fix js unit tests that fail in lyftga * Fix js * Sparkline dates aren't formatting in Time Series Table (#6976) * Exclude venv for python linter to ignore * Fix NaN error * Changing time table viz to pass formatTime a date (#7020) (cherry picked from commit 7f3c145) * SQL editor layout makeover (#7102) This PR includes the following layout and css tweaks: - Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox - Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane - Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars - css tweaks on the basic.html page so the loading animation appears in the center of the page across the board (cherry picked from commit 71f1bbd) * Add decorator for etag cache * Fetch charts with GET * Small fixes * Fix typo * Compute correct cache key; fix logging * Check perms on cached response * Revert change * If perms fail, return naked response * Fix lint * Compute cache key from all form data * Pass extra_filters in GET request * Fix pylint * Fix flake8 * Use ETags even if no cache is set * Handle adhoc filters * Raise in debug mode * Rename actions * Fix integration tests * Do POST request on new charts * Set extra/adhoc filters only in GET requests * Raise if check_perms fails * Refactor auth * Fix flake8 * Fix js unit tests * Fix js unit tests that fail in lyftga * Fix js * Fix bad merge * Use far future when max_age=0

* Sparkline dates aren't formatting in Time Series Table (#6976) * Exclude venv for python linter to ignore * Fix NaN error * Fix the white background shown in SQL editor on drag (#7021) This PR sets the background-color css property on `.ace_scroller` instead of `.ace_content` to prevent the white background shown during resizing of the SQL editor before drag ends. * Show tooltip with time frame (#6979) * Fix time filter control (#6978) * Enhancement of query context and object. (#6962) * added more functionalities for query context and object. * fixed cache logic * added default value for groupby * updated comments and removed print (cherry picked from commit d5b9795) * [fix] /superset/slice/id url is too long (#6989) (cherry picked from commit 6a4d507) * [WIP] fix user specified JSON metadata not updating dashboard on refresh (#7027) (cherry picked from commit cc58f0e) * feat: add ability to change font size in big number (#7003) * Add ability to change font sizes in Big Number * rename big number to header * Add comment to clarify font size values * Allow LIMIT to be specified in parameters (#7052) * [fix] Cursor jumping when editing chart and dashboard titles (#7038) (cherry picked from commit fc1770f) * Changing time table viz to pass formatTime a date (#7020) (cherry picked from commit 7f3c145) * [db-engine-spec] Aligning Hive/Presto partition logic (#7007) (cherry picked from commit 05be866) * [fix] explore chart from dashboard missed slice title (#7046) (cherry picked from commit a6d48d4) * fix inaccurate data calculation with adata rolling and contribution (#7035) (cherry picked from commit 0782e83) * Adding warning message for sqllab save query (#7028) (cherry picked from commit ead3d48) * [datasource] Ensuring consistent behavior of datasource editing/saving. (#7037) * Update datasource.py * Update datasource.py (cherry picked from commit c771625) * [csv-upload] Fixing message encoding (#6971) (cherry picked from commit 48431ab) * [sql-parse] Fixing LIMIT exceptions (#6963) (cherry picked from commit 3e076cb) * Adding custom control overrides (#6956) * Adding extraOverrides to line chart * Updating extraOverrides to fit with more cases * Moving extraOverrides to index.js * Removing webpack-merge in package.json * Fixing metrics control clearing metric (cherry picked from commit e619405) * [sqlparse] Fixing table name extraction for ill-defined query (#7029) (cherry picked from commit 07c340c) * [missing values] Removing replacing missing values (#4905) (cherry picked from commit 61add60) * [SQL Lab] Improved query and results tabs rendering reliability (#7082) closes #7080 (cherry picked from commit 9b58e9f) * Fix filter_box migration PR #6523 (#7066) * Fix filter_box migration PR #6523 * Fix druid-related bug (cherry picked from commit b210742) * SQL editor layout makeover (#7102) This PR includes the following layout and css tweaks: - Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox - Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane - Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars - css tweaks on the basic.html page so the loading animation appears in the center of the page across the board (cherry picked from commit 71f1bbd) * [forms] Fix handling of NULLs (cherry picked from commit e83a07d) * handle null column_name in sqla and druid models (cherry picked from commit 2ff721a) * Use metric name instead of metric in filter box (#7106) (cherry picked from commit 003364e) * Bump python lib croniter to an existing version (#7132) Package maintainers should really never delete packages, but it appears this happened with croniter and resulted in breaking our builds. This PR bumps to a more recent existing version of the library (cherry picked from commit 215ed39) * Revert PR #6933 (#7162) * Celery worker for warming up cache * Remove testing changes * Add documentation * Fix lint * WIP dashboard filters * Use new cache so it works with dashboards * Add more unit tests, fix old ones * Fix flake8 and docs * Sparkline dates aren't formatting in Time Series Table (#6976) * Exclude venv for python linter to ignore * Fix NaN error * Changing time table viz to pass formatTime a date (#7020) (cherry picked from commit 7f3c145) * SQL editor layout makeover (#7102) This PR includes the following layout and css tweaks: - Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox - Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane - Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars - css tweaks on the basic.html page so the loading animation appears in the center of the page across the board (cherry picked from commit 71f1bbd) * Celery worker for warming up cache * Remove testing changes * Add documentation * Fix lint * WIP dashboard filters * Use new cache so it works with dashboards * Add more unit tests, fix old ones * Fix flake8 and docs * Fix bad merge and pylint

john-bodley added the !deprecated-label:bug Deprecated label - Use #bug instead label Mar 11, 2019

[db-engine-spec] Aligning Hive/Presto partition logic

f1d2c1e

john-bodley force-pushed the john-bodley--latest-partition branch from bf70267 to f1d2c1e Compare March 11, 2019 18:25

mistercrunch approved these changes Mar 12, 2019

View reviewed changes

john-bodley merged commit 05be866 into apache:master Mar 13, 2019

john-bodley deleted the john-bodley--latest-partition branch March 13, 2019 20:22

graceguo-supercat pushed a commit to graceguo-supercat/superset that referenced this pull request Mar 18, 2019

[db-engine-spec] Aligning Hive/Presto partition logic (apache#7007)

c54b067

(cherry picked from commit 05be866)

xtinec pushed a commit that referenced this pull request Mar 21, 2019

[db-engine-spec] Aligning Hive/Presto partition logic (#7007)

771d212

(cherry picked from commit 05be866)

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.34.0 labels Feb 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[db-engine-spec] Aligning Hive/Presto partition logic #7007

[db-engine-spec] Aligning Hive/Presto partition logic #7007

john-bodley commented Mar 11, 2019 •

edited

codecov-io commented Mar 11, 2019

mistercrunch left a comment

[db-engine-spec] Aligning Hive/Presto partition logic #7007

[db-engine-spec] Aligning Hive/Presto partition logic #7007

Conversation

john-bodley commented Mar 11, 2019 • edited

codecov-io commented Mar 11, 2019

Codecov Report

mistercrunch left a comment

Choose a reason for hiding this comment

john-bodley commented Mar 11, 2019 •

edited