Merge master into lyft-release-sp8 (#7484)

* filter out all nan series (#7313) * improve not rich tooltip (#7345) * Create issue_label_bot.yaml (#7341) * fix: do not save colors without a color scheme (#7347) * [wtforms] Strip leading/trailing whitespace (#7084) * [schema] Updating the datasources schema (#5451) * limit tables/views returned if schema is not provided (#7358) * limit tables/views returned if schema is not provided * fix typo * improve code performance * handle the case when table name or view name does not present a schema * Add type anno (#7342) * Updated local dev instructions to include missing step * First pass at type annotations * [schema] Updating the base column schema (#5452) * Update 937d04c16b64_update_datasources.py (#7361) * Feature flag for client cache (#7348) * Feature flag for client cache * Fix integration test * Revert "Fix integration test" This reverts commit 58434ab. * Feature flag for client cache * Fix integration tests * Add feature flag to config.py * Add another feature check * Fix more integration tests * Fix raw HTML in SliceAdder (#7338) * remove backendSync.json (#7331) * [bubbles] issue when using duplicated metrics (#7087) * SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359) * SUPERSET-8: Update text in docs copyright footer (#7360) * SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 * SUPERSET-8: Extra text in docs copyright footer * [schema] Adding commits and removing unnecessary foreign-key definitions (#7371) * Store last selected dashboard in sessionStorage (#7181) * Store last selected dashboard in sessionStorage * Fix tests * [schema] Updating the base metric schema (#5453) * Fix NoneType bug & fill the test recipients with original recipients if empty (#7365) * Added living goods as among the users of Superset (#7407) * Added living goods as among the users of Superset Living Goods is a non profit organisation with operation in africa and the middle east. We work in community health use data heavily on day to day. Superset is our platform of choice for dashboards. * Update README.md * [dashboard] allow user re-order top-level tabs (#7390) * [SQL Lab] Increase timeout threshold for offline check (#7411) * Bump FAB to 2.0.0 (#7323) * Bump FAB to 2.0.0 * [tests] whitelist SecurityApi login and refresh endpoints * [style] Fix, C812 missing trailing commas * [security] Remove SUPERSET_UPDATE_PERMS flag Registering sources needs to be performed after the views are initialized on UPDATE_PERMS=False configuration * [docs] New, FAB_UPDATE_PERMS and flask fab cli * [docs] Fix, db upgrade needs to come first, create-admin needs a db * [cli] New, superset init bootstraps all permissions for FAB and Superset * [style] Fix, flakes * [annotations] Improves UX on annotation validation, start_dttm, end_dttm (#7326) * Setting renderTrigger on label_colors (#7410) * Refactor out controlUtils.js module + unit tests (#7350) * [WiP]refactor out a controlUtils.js file * unit tests * add missing license * Addressing comments * feature: see Presto row and array data types (#7413) * Merge lastest from master into lyft-release-sp8 (#7405) * filter out all nan series (#7313) * improve not rich tooltip (#7345) * Create issue_label_bot.yaml (#7341) * fix: do not save colors without a color scheme (#7347) * [wtforms] Strip leading/trailing whitespace (#7084) * [schema] Updating the datasources schema (#5451) * limit tables/views returned if schema is not provided (#7358) * limit tables/views returned if schema is not provided * fix typo * improve code performance * handle the case when table name or view name does not present a schema * Add type anno (#7342) * Updated local dev instructions to include missing step * First pass at type annotations * [schema] Updating the base column schema (#5452) * Update 937d04c16b64_update_datasources.py (#7361) * Feature flag for client cache (#7348) * Feature flag for client cache * Fix integration test * Revert "Fix integration test" This reverts commit 58434ab. * Feature flag for client cache * Fix integration tests * Add feature flag to config.py * Add another feature check * Fix more integration tests * Fix raw HTML in SliceAdder (#7338) * remove backendSync.json (#7331) * [bubbles] issue when using duplicated metrics (#7087) * SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359) * SUPERSET-8: Update text in docs copyright footer (#7360) * SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 * SUPERSET-8: Extra text in docs copyright footer * [schema] Adding commits and removing unnecessary foreign-key definitions (#7371) * Store last selected dashboard in sessionStorage (#7181) * Store last selected dashboard in sessionStorage * Fix tests * [schema] Updating the base metric schema (#5453) * Fix NoneType bug & fill the test recipients with original recipients if empty (#7365) * feat: see Presto row and array data types (#7391) * feat: see Presto row and array data types * fix: address PR comments * fix: lint and build issues * fix: add types * add stronger type hints where possible * fix: lint issues and add select_star func in Hive * add missing pkg init * fix: build issues * fix: pylint issues * fix: use logging instead of print * Removed --console-log and superset runserver (#7421) * Fixes dashboard export button missing download and #7353 (#7427) * Added additional German translations to string file (#6604) * Added additional German translations to string file Updates to German translation files as per directions * Removed messages.json * [fix] Fixing SQL parsing issue (#7374) * add chinese translate (#7402) * Quick fix to address deadlock issue (#7434) * feat: view presto row objects in data grid (#7445) * Merge lastest from master into lyft-release-sp8 (#7405) * filter out all nan series (#7313) * improve not rich tooltip (#7345) * Create issue_label_bot.yaml (#7341) * fix: do not save colors without a color scheme (#7347) * [wtforms] Strip leading/trailing whitespace (#7084) * [schema] Updating the datasources schema (#5451) * limit tables/views returned if schema is not provided (#7358) * limit tables/views returned if schema is not provided * fix typo * improve code performance * handle the case when table name or view name does not present a schema * Add type anno (#7342) * Updated local dev instructions to include missing step * First pass at type annotations * [schema] Updating the base column schema (#5452) * Update 937d04c16b64_update_datasources.py (#7361) * Feature flag for client cache (#7348) * Feature flag for client cache * Fix integration test * Revert "Fix integration test" This reverts commit 58434ab. * Feature flag for client cache * Fix integration tests * Add feature flag to config.py * Add another feature check * Fix more integration tests * Fix raw HTML in SliceAdder (#7338) * remove backendSync.json (#7331) * [bubbles] issue when using duplicated metrics (#7087) * SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359) * SUPERSET-8: Update text in docs copyright footer (#7360) * SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 * SUPERSET-8: Extra text in docs copyright footer * [schema] Adding commits and removing unnecessary foreign-key definitions (#7371) * Store last selected dashboard in sessionStorage (#7181) * Store last selected dashboard in sessionStorage * Fix tests * [schema] Updating the base metric schema (#5453) * Fix NoneType bug & fill the test recipients with original recipients if empty (#7365) * feat: see Presto row and array data types (#7391) * feat: see Presto row and array data types * fix: address PR comments * fix: lint and build issues * fix: add types * Incorporate feedback from initial PR (prematurely merged to lyft-release-sp8) (#7415) * add stronger type hints where possible * fix: lint issues and add select_star func in Hive * add missing pkg init * fix: build issues * fix: pylint issues * fix: use logging instead of print * feat: view presto row objects in data grid * fix: address feedback * fix: spacing * feat: Scheduling queries from SQL Lab (#7416) (#7446) * Merge lastest from master into lyft-release-sp8 (#7405) * filter out all nan series (#7313) * improve not rich tooltip (#7345) * Create issue_label_bot.yaml (#7341) * fix: do not save colors without a color scheme (#7347) * [wtforms] Strip leading/trailing whitespace (#7084) * [schema] Updating the datasources schema (#5451) * limit tables/views returned if schema is not provided (#7358) * limit tables/views returned if schema is not provided * fix typo * improve code performance * handle the case when table name or view name does not present a schema * Add type anno (#7342) * Updated local dev instructions to include missing step * First pass at type annotations * [schema] Updating the base column schema (#5452) * Update 937d04c16b64_update_datasources.py (#7361) * Feature flag for client cache (#7348) * Feature flag for client cache * Fix integration test * Revert "Fix integration test" This reverts commit 58434ab. * Feature flag for client cache * Fix integration tests * Add feature flag to config.py * Add another feature check * Fix more integration tests * Fix raw HTML in SliceAdder (#7338) * remove backendSync.json (#7331) * [bubbles] issue when using duplicated metrics (#7087) * SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359) * SUPERSET-8: Update text in docs copyright footer (#7360) * SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 * SUPERSET-8: Extra text in docs copyright footer * [schema] Adding commits and removing unnecessary foreign-key definitions (#7371) * Store last selected dashboard in sessionStorage (#7181) * Store last selected dashboard in sessionStorage * Fix tests * [schema] Updating the base metric schema (#5453) * Fix NoneType bug & fill the test recipients with original recipients if empty (#7365) * feat: see Presto row and array data types (#7391) * feat: see Presto row and array data types * fix: address PR comments * fix: lint and build issues * fix: add types * Incorporate feedback from initial PR (prematurely merged to lyft-release-sp8) (#7415) * add stronger type hints where possible * fix: lint issues and add select_star func in Hive * add missing pkg init * fix: build issues * fix: pylint issues * fix: use logging instead of print * feat: view presto row objects in data grid * fix: address feedback * fix: spacing * Workaround for no results returned (#7442) * feat: view presto row objects in data grid (#7436) * feat: view presto row objects in data grid * fix: address feedback * fix: spacing * feat: Scheduling queries from SQL Lab (#7416) * Lightweight pipelines POC * Add docs * Minor fixes * Remove Lyft URL * Use enum * Minor fix * Fix unit tests * Mark props as required * feat: Add `validate_sql_json` endpoint for checking that a given sql query is valid for the chosen database (#7422) (#7462) merge from lyft-release-sp8 to master * Adds missing metric sum__SP_RUR_TOTL (#7452) * Late import for optional lib pyhive (#7471) * Late import for optional lib pyhive * fix * fix: calendar heatmap examples (#7375) Fixing a set of examples that trip on ValueError vs TypeError * bugfix: Improve support for special characters in schema and table names (#7297) * Bugfix to SQL Lab to support tables and schemas with characters that require quoting * Remove debugging prints * Add uri encoding to secondary tables call * Quote schema names for presto * Quote selected_schema on Snowflake, MySQL and Hive * Remove redundant parens * Add python unit tests * Add js unit test * Fix flake8 linting error * [dashboard] After update filter, trigger new queries when charts are visible (#7233) * trigger query when chart is visible * add integration test * fix: alter sql columns to long text #7463 (#7476) Merge lyft-release-sp8@7bfe7bc to master * Refactor ConsoleLog (#7428) * Revised Chinese translation (#7464) * add chinese translate * edit chinese translation * druid connector: avoid using 'dimensions' for scan queries (#7377) After the following PyDruid change (contained in version 0.5.2) the Superset Histogram charts rendered with Druid data are broken: druid-io/pydruid@0a59a70 Bump the pydruid requirements accordingly in setup.py Issue: #7368
apache · May 10, 2019 · a1c0051 · a1c0051
1 parent 7bfe7bc
commit a1c0051
Show file tree

Hide file tree

Showing 65 changed files with 1,385 additions and 2,811 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -328,10 +328,19 @@ FLASK_ENV=development flask run -p 8088 --with-threads --reload --debugger
 
 #### Logging to the browser console
 
-This feature is only available on Python 3. When debugging your application, you can have the server logs sent directly to the browser console:
+This feature is only available on Python 3. When debugging your application, you can have the server logs sent directly to the browser console using the [ConsoleLog](https://github.com/betodealmeida/consolelog) package. You need to mutate the app, by adding the following to your `config.py` or `superset_config.py`:
+
+```python
+from console_log import ConsoleLog
+
+def FLASK_APP_MUTATOR(app):
+    app.wsgi_app = ConsoleLog(app.wsgi_app, app.logger)
+```
+
+Then make sure you run your WSGI server using the right worker type:
 
 ```bash
-FLASK_ENV=development flask run -p 8088 --with-threads --reload --debugger --console-log
+FLASK_ENV=development gunicorn superset:app -k "geventwebsocket.gunicorn.workers.GeventWebSocketWorker" -b 127.0.0.1:8088 --reload
 ```
 
 You can log anything to the browser console, including objects:

diff --git a/README.md b/README.md
@@ -195,6 +195,7 @@ the world know they are using Superset. Join our growing community!
  1. [Konfío](http://konfio.mx)
  1. [Kuaishou](https://www.kuaishou.com/)
  1. [Lime](https://www.limebike.com/)
+ 1. [Living Goods](https://www.livinggoods.org)
  1. [Lyft](https://www.lyft.com/)
  1. [Maieutical Labs](https://maieuticallabs.it)
  1. [Myra Labs](http://www.myralabs.com/)

diff --git a/UPDATING.md b/UPDATING.md
@@ -21,7 +21,7 @@ under the License.
 This file documents any backwards-incompatible changes in Superset and
 assists people when migrating to a new version.
 
-## Superset 0.34.0
+## Next Version
 
 * [5451](https://github.com/apache/incubator-superset/pull/5451): a change
 which adds missing non-nullable fields to the `datasources` table. Depending on
@@ -31,7 +31,11 @@ the integrity of the data, manual intervention may be required.
 which adds missing non-nullable fields and uniqueness constraints to the
 `columns`and `table_columns` tables. Depending on the integrity of the data,
 manual intervention may be required.
-
+* `fabmanager` command line is deprecated since Flask-AppBuilder 2.0.0, use
+the new `flask fab <command>` integrated with *Flask cli*.
+* `SUPERSET_UPDATE_PERMS` environment variable was replaced by 
+`FAB_UPDATE_PERMS` config boolean key. To disable automatic
+creation of permissions set `FAB_UPDATE_PERMS = False` on config.
 * [5453](https://github.com/apache/incubator-superset/pull/5453): a change
 which adds missing non-nullable fields and uniqueness constraints to the metrics
 and sql_metrics tables. Depending on the integrity of the data, manual

diff --git a/docs/faq.rst b/docs/faq.rst
@@ -186,8 +186,8 @@ by setting the ``stagger_refresh`` to ``false`` and modify the stagger period by
 Here, the entire dashboard will refresh at once if periodic refresh is on. The stagger time of
 2.5 seconds is ignored.
 
-Why does fabmanager or superset freezed/hung/not responding when started (my home directory is NFS mounted)?
-------------------------------------------------------------------------------------------------------------
+Why does 'flask fab' or superset freezed/hung/not responding when started (my home directory is NFS mounted)?
+-------------------------------------------------------------------------------------------------------------
 By default, superset creates and uses an sqlite database at ``~/.superset/superset.db``. Sqlite is known to `don't work well if used on NFS`__ due to broken file locking implementation on NFS.
 
 __ https://www.sqlite.org/lockingv3.html

diff --git a/docs/installation.rst b/docs/installation.rst
@@ -170,20 +170,21 @@ Follow these few simple steps to install Superset.::
     # Install superset
     pip install superset
 
-    # Create an admin user (you will be prompted to set a username, first and last name before setting a password)
-    fabmanager create-admin --app superset
-
     # Initialize the database
     superset db upgrade
 
+    # Create an admin user (you will be prompted to set a username, first and last name before setting a password)
+    $ export FLASK_APP=superset
+    flask fab create-admin
+
     # Load some data to play with
     superset load_examples
 
     # Create default roles and permissions
     superset init
 
     # To start a development web server on port 8088, use -p to bind to another port
-    superset runserver -d
+    flask run -p 8080 --with-threads --reload --debugger
 
 
 After installation, you should be able to point your browser to the right
@@ -236,17 +237,11 @@ workers this creates a lot of contention and race conditions when defining
 permissions and views.
 
 To alleviate this issue, the automatic updating of permissions can be disabled
-by setting the environment variable
-`SUPERSET_UPDATE_PERMS` environment variable to `0`.
-The value `1` enables it, `0` disables it. Note if undefined the functionality
-is enabled to maintain backwards compatibility.
+by setting `FAB_UPDATE_PERMS = False` (defaults to True).
 
 In a production environment initialization could take on the following form:
 
-  export SUPERSET_UPDATE_PERMS=1
   superset init
-
-  export SUPERSET_UPDATE_PERMS=0
   gunicorn -w 10 ... superset:app
 
 Configuration behind a load balancer

diff --git a/requirements-dev.txt b/requirements-dev.txt
@@ -14,7 +14,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-console_log==0.2.10
 coverage==4.5.3
 flake8-commas==2.0.0
 flake8-import-order==0.18

diff --git a/requirements.txt b/requirements.txt
@@ -6,7 +6,9 @@
 #
 alembic==1.0.0            # via flask-migrate
 amqp==2.3.2               # via kombu
+apispec[yaml]==1.2.0      # via flask-appbuilder
 asn1crypto==0.24.0        # via cryptography
+attrs==19.1.0             # via jsonschema
 babel==2.6.0              # via flask-babel
 billiard==3.5.0.4         # via celery
 bleach==3.0.2
@@ -21,10 +23,11 @@ croniter==0.3.29
 cryptography==2.4.2
 decorator==4.3.0          # via retry
 defusedxml==0.5.0         # via python3-openid
-flask-appbuilder==1.12.5
+flask-appbuilder==2.0.0
 flask-babel==0.11.1       # via flask-appbuilder
 flask-caching==1.4.0
 flask-compress==1.4.0
+flask-jwt-extended==3.18.1  # via flask-appbuilder
 flask-login==0.4.1        # via flask-appbuilder
 flask-migrate==2.1.1
 flask-openid==1.2.5       # via flask-appbuilder
@@ -38,19 +41,25 @@ idna==2.6
 isodate==0.6.0
 itsdangerous==0.24        # via flask
 jinja2==2.10              # via flask, flask-babel
+jsonschema==3.0.1         # via flask-appbuilder
 kombu==4.2.1              # via celery
 mako==1.0.7               # via alembic
 markdown==3.0
 markupsafe==1.0           # via jinja2, mako
+marshmallow-enum==1.4.1   # via flask-appbuilder
+marshmallow-sqlalchemy==0.16.2  # via flask-appbuilder
+marshmallow==2.19.2       # via flask-appbuilder, marshmallow-enum, marshmallow-sqlalchemy
 numpy==1.15.2             # via pandas
 pandas==0.23.4
 parsedatetime==2.0.0
 pathlib2==2.3.0
 polyline==1.3.2
+prison==0.1.0             # via flask-appbuilder
 py==1.7.0                 # via retry
 pycparser==2.19           # via cffi
 pydruid==0.5.2
-pyjwt==1.7.1              # via flask-appbuilder
+pyjwt==1.7.1              # via flask-appbuilder, flask-jwt-extended
+pyrsistent==0.14.11       # via jsonschema
 python-dateutil==2.6.1
 python-editor==1.0.3      # via alembic
 python-geohash==0.8.5
@@ -61,14 +70,14 @@ requests==2.20.0
 retry==0.9.2
 selenium==3.141.0
 simplejson==3.15.0
-six==1.11.0               # via bleach, cryptography, isodate, pathlib2, polyline, pydruid, python-dateutil, sqlalchemy-utils, wtforms-json
+six==1.11.0               # via bleach, cryptography, flask-jwt-extended, isodate, jsonschema, pathlib2, polyline, prison, pydruid, pyrsistent, python-dateutil, sqlalchemy-utils, wtforms-json
 sqlalchemy-utils==0.32.21
 sqlalchemy==1.3.1
 sqlparse==0.2.4
 unicodecsv==0.14.1
 urllib3==1.22             # via requests, selenium
 vine==1.1.4               # via amqp
 webencodings==0.5.1       # via bleach
-werkzeug==0.14.1          # via flask
+werkzeug==0.14.1          # via flask, flask-jwt-extended
 wtforms-json==0.3.3
 wtforms==2.2.1            # via flask-wtf, wtforms-json
diff --git a/setup.py b/setup.py
@@ -77,7 +77,7 @@ def get_git_sha():
         'croniter>=0.3.28',
         'cryptography>=2.4.2',
         'flask>=1.0.0, <2.0.0',
-        'flask-appbuilder>=1.12.5, <2.0.0',
+        'flask-appbuilder>=2.0.0, <2.3.0',
         'flask-caching',
         'flask-compress',
         'flask-migrate',
@@ -92,7 +92,7 @@ def get_git_sha():
         'parsedatetime',
         'pathlib2',
         'polyline',
-        'pydruid>=0.4.3',
+        'pydruid>=0.5.2',
         'python-dateutil',
         'python-geohash',
         'pyyaml>=3.13',
@@ -108,7 +108,6 @@ def get_git_sha():
     ],
     extras_require={
         'cors': ['flask-cors>=2.0.0'],
-        'console_log': ['console_log==0.2.10'],
         'hive': [
             'pyhive[hive]>=0.6.1',
             'tableschema',

diff --git a/superset/__init__.py b/superset/__init__.py
@@ -34,8 +34,7 @@
 from superset import config
 from superset.connectors.connector_registry import ConnectorRegistry
 from superset.security import SupersetSecurityManager
-from superset.utils.core import (
-    get_update_perms_flag, pessimistic_connection_handling, setup_cache)
+from superset.utils.core import pessimistic_connection_handling, setup_cache
 
 wtforms_json.init()
 
@@ -196,14 +195,14 @@ def index(self):
          not FAB's security manager.
          See [4565] in UPDATING.md""")
 
-appbuilder = AppBuilder(
-    app,
-    db.session,
-    base_template='superset/base.html',
-    indexview=MyIndexView,
-    security_manager_class=custom_sm,
-    update_perms=get_update_perms_flag(),
-)
+with app.app_context():
+    appbuilder = AppBuilder(
+        app,
+        db.session,
+        base_template='superset/base.html',
+        indexview=MyIndexView,
+        security_manager_class=custom_sm,
+    )
 
 security_manager = appbuilder.sm
 
@@ -226,11 +225,6 @@ def is_feature_enabled(feature):
     return get_feature_flags().get(feature)
 
 
-# Registering sources
-module_datasource_map = app.config.get('DEFAULT_MODULE_DS_MAP')
-module_datasource_map.update(app.config.get('ADDITIONAL_MODULE_DS_MAP'))
-ConnectorRegistry.register_sources(module_datasource_map)
-
 # Flask-Compress
 if conf.get('ENABLE_FLASK_COMPRESS'):
     Compress(app)
@@ -242,3 +236,8 @@ def is_feature_enabled(feature):
     flask_app_mutator(app)
 
 from superset import views  # noqa
+
+# Registering sources
+module_datasource_map = app.config.get('DEFAULT_MODULE_DS_MAP')
+module_datasource_map.update(app.config.get('ADDITIONAL_MODULE_DS_MAP'))
+ConnectorRegistry.register_sources(module_datasource_map)
diff --git a/superset/assets/cypress/integration/dashboard/dashboard.helper.js b/superset/assets/cypress/integration/dashboard/dashboard.helper.js
@@ -16,7 +16,8 @@
  * specific language governing permissions and limitations
  * under the License.
  */
-export const WORLD_HEALTH_DASHBOARD = '/superset/dashboard/world_health';
+export const WORLD_HEALTH_DASHBOARD = '/superset/dashboard/world_health/';
+export const TABBED_DASHBOARD = '/superset/dashboard/tabbed_dash/';
 
 export const CHECK_DASHBOARD_FAVORITE_ENDPOINT = '/superset/favstar/Dashboard/*/count';
 
diff --git a/superset/assets/cypress/integration/dashboard/index.test.js b/superset/assets/cypress/integration/dashboard/index.test.js
@@ -22,6 +22,7 @@ import DashboardFavStarTest from './fav_star';
 import DashboardFilterTest from './filter';
 import DashboardLoadTest from './load';
 import DashboardSaveTest from './save';
+import DashboardTabsTest from './tabs';
 
 describe('Dashboard', () => {
   DashboardControlsTest();
@@ -30,4 +31,5 @@ describe('Dashboard', () => {
   DashboardFilterTest();
   DashboardLoadTest();
   DashboardSaveTest();
+  DashboardTabsTest();
 });