feat(api): bump marshmallow and FAB to version 3 #9964

dpgaspar · 2020-06-02T11:35:33Z

SUMMARY

Bump Flask-AppBuilder to version 3 that implies marshmallow version 3 that is a major (breaking) release

Marshmallow upgrading to version 3: https://github.com/marshmallow-code/marshmallow/blob/31b784fc24ed637c82d87eac35d1a1bd58851f59/docs/upgrading.rst

FAB 3 changelog:

Improvements and Bug fixes on 3.0.0

Fix, swagger test (fix: swagger test dpgaspar/Flask-AppBuilder#1423)
Fix, change openapi tags and swagger access URL (breaking) (fix, change openapi tags and swagger access URL (breaking) dpgaspar/Flask-AppBuilder#1422)
Fix: replace deprecated flask-oauthlib with authlib (fix: replace deprecated flask-oauthlib with authlib dpgaspar/Flask-AppBuilder#1411)
Refactor: interface query on m-m joins and select specific columns (refactor: interface query on m-m joins and select specific columns dpgaspar/Flask-AppBuilder#1398)
Fix, docs on has_view_access (Fix, docs on has_view_access dpgaspar/Flask-AppBuilder#1419)
New, Examples/react api ([WiP] Examples/react api dpgaspar/Flask-AppBuilder#1071)
Fix, action confirmation disabling (Fix, action confirmation disabling dpgaspar/Flask-AppBuilder#1408)
New, add API descriptions and examples to OpenAPI spec (New, add API descriptions and examples to OpenAPI spec dpgaspar/Flask-AppBuilder#1396)
New, Dynamic user registration role (New, Dynamic user registration role dpgaspar/Flask-AppBuilder#1410)
Fix, typos and improved bit of the German translation (Fix: fixed typos and improved bit of the German translation dpgaspar/Flask-AppBuilder#1406)
New, Added Dutch language to docs (New, Added Dutch language to docs dpgaspar/Flask-AppBuilder#1393)
New, Added Dutch translation (New, Added Dutch translation dpgaspar/Flask-AppBuilder#1387)
Fix, load options and limits for many to many truncating results (Fix, load options and limits for many to many truncating results dpgaspar/Flask-AppBuilder#1389)
Fix, SQLAlchemyAutoSchema needs marshmallow-sqlalchemy>=0.22.0 (SQLAlchemyAutoSchema needs marshmallow-sqlalchemy>=0.22.0 dpgaspar/Flask-AppBuilder#1392)
New, [api] support marshmallow 3 ([dependencies] Fix, support marshmallow 3 dpgaspar/Flask-AppBuilder#1334)
Fix, hardcoded url on oauth (Fix, hardcoded url in oauth template, fix for #1182 dpgaspar/Flask-AppBuilder#1331)
Fix, [examples] Update models.py (Update models.py dpgaspar/Flask-AppBuilder#1380)
Fix: add a panel body to panel_begin/panel_end macros (fix: add a panel body to panel_begin/panel_end macros dpgaspar/Flask-AppBuilder#1377)
Fix, name column resizing in ab_view_menu table Column/Metric description tooltips missing #1367 (Fix, name column resizing in ab_view_menu table #1367 dpgaspar/Flask-AppBuilder#1368)
Fix, typos in the documentation (Fix, possible typos in the documentation dpgaspar/Flask-AppBuilder#1375)

BREAKING CHANGES:

New, [api] support marshmallow 3 (Apply appropriate type conversion for viz form_data #1334)
swaggerview/v1 is now exposed on swagger/v1
Oauth now uses authlib and configuration is slightly different: https://flask-appbuilder.readthedocs.io/en/latest/security.html#authentication-oauth

Fixes: #10232

ADDITIONAL INFORMATION

Has associated issue: Welcome page pagination is inconsistent #10232
Changes UI
Requires DB Migration.
Confirm DB Migration upgrade and downgrade tested.
Introduces new feature or API
Removes existing feature or API

codecov-commenter · 2020-06-02T11:41:39Z

Codecov Report

Merging #9964 into master will increase coverage by 0.58%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #9964      +/-   ##
==========================================
+ Coverage   70.31%   70.90%   +0.58%     
==========================================
  Files         594      404     -190     
  Lines       31608    13108   -18500     
  Branches     3221     3221              
==========================================
- Hits        22225     9294   -12931     
+ Misses       9275     3706    -5569     
  Partials      108      108

Flag	Coverage Δ
#cypress	`53.45% <ø> (+0.07%)`	⬆️
#javascript	`59.44% <ø> (ø)`
#python	`?`

Impacted Files	Coverage Δ
superset/db_engine_specs/kylin.py
superset/connectors/druid/__init__.py
superset/db_engines/hive.py
superset/connectors/base/models.py
superset/db_engine_specs/elasticsearch.py
superset/db_engine_specs/mssql.py
superset/views/chart/mixin.py
superset/examples/countries.py
superset/models/datasource_access_request.py
superset/examples/world_bank.py
... and 168 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fdd28c1...7e046a8. Read the comment docs.

# Conflicts: # requirements.txt # superset/charts/api.py

villebro · 2020-07-03T15:06:42Z

Tagging this as 0.37 due to bugs it fixes in master + to make it easier to cherry pick for patch releases on 0.37 version.

villebro

A few first pass comments. Looking forward to getting aboard {FAB,marshmallow}3! 👍

villebro · 2020-07-05T21:00:09Z

superset/charts/api.py

+            query_context = ChartDataQueryContextSchema().load(json_body)
        except KeyError:
            return self.response_400(message="Request is incorrect")
+        except ValidationError as err:
+            return self.response_400(
+                _("Request is incorrect: %(error)s", error=err.messages)
+            )


Is except KeyError still needed here? Based on the docs, load only raises ValidationError in 3.0+: https://marshmallow.readthedocs.io/en/stable/_modules/marshmallow/schema.html#Schema.load

Yes it seems weird, this may possible come from our custom post_load https://github.com/apache/incubator-superset/blob/master/superset/charts/schemas.py#L736 and then from https://github.com/apache/incubator-superset/blob/master/superset/common/query_context.py#L57

superset/charts/schemas.py

villebro · 2020-07-05T21:09:59Z

superset/dashboards/api.py

+        except ValidationError as err:
+            return self.response_400(message=err.messages)


Being picky, but is it more pythonic to assign error as opposed to err? (Applies to other places, too)

villebro · 2020-07-05T21:19:00Z

tests/charts/schema_tests.py

+        try:
+            _ = ChartDataQueryContextSchema().load(payload)
+        except ValidationError as errors:
+            self.assertIn("row_limit", errors.messages["queries"][0])
+            self.assertIn("row_offset", errors.messages["queries"][0])


If this doesn't raise, won't the assertions just be skipped?

good catch!

villebro

A few small second pass comments

superset/charts/schemas.py

tests/datasets/api_tests.py

villebro

Would suggest bumping to latest patch of marshmallow, other than that LGTM 🎉

requirements.txt

* feat(api): bump marshmallow and FAB to version 3 * revert query context tests changes * obey mypy * fix tests * ignore types that collide with marshmallow * preparing for RC2 * fix tests for marshmallow 3 * typing fixes for marshmallow * fix tests and black * fix tests * bump to RC3 and lint * Test RC4 * Final 3.0.0 * Address comments, fix tests, better naming, docs * fix test * couple of fixes, addressing comments * bumping marshmallow

feat(api): bump marshmallow and FAB to version 3

46c681f

superset-github-bot bot added the preset-io label Jun 2, 2020

pull-request-size bot added the size/L label Jun 2, 2020

dpgaspar added 13 commits June 2, 2020 13:06

revert query context tests changes

377cd26

obey mypy

ce7cab5

fix tests

18522ce

ignore types that collide with marshmallow

7a05ed9

preparing for RC2

875d8eb

Merge branch 'master' into dep/marshmallow3

e464929

# Conflicts: # requirements.txt # superset/charts/api.py

fix tests for marshmallow 3

1e68ad7

typing fixes for marshmallow

7e046a8

fix tests and black

7e50e0c

fix tests

600d404

bump to RC3 and lint

126e942

Test RC4

9c64267

Final 3.0.0

15d75b4

dpgaspar marked this pull request as ready for review July 3, 2020 14:52

dpgaspar requested review from villebro, bkyryliuk and etr2460 July 3, 2020 14:54

villebro added the v0.37 label Jul 3, 2020

villebro reviewed Jul 5, 2020

View reviewed changes

dpgaspar added 2 commits July 6, 2020 09:52

Address comments, fix tests, better naming, docs

b16094a

fix test

66a47d0

dpgaspar requested a review from villebro July 6, 2020 09:16

villebro requested changes Jul 6, 2020

View reviewed changes

superset/charts/schemas.py Outdated Show resolved Hide resolved

tests/datasets/api_tests.py Outdated Show resolved Hide resolved

tests/datasets/api_tests.py Outdated Show resolved Hide resolved

couple of fixes, addressing comments

eda8b83

dpgaspar requested a review from villebro July 6, 2020 09:48

villebro approved these changes Jul 6, 2020

View reviewed changes

requirements.txt Outdated Show resolved Hide resolved

bumping marshmallow

1c40698

bkyryliuk approved these changes Jul 6, 2020

View reviewed changes

dpgaspar merged commit 878dbcd into apache:master Jul 7, 2020

dpgaspar deleted the dep/marshmallow3 branch July 7, 2020 12:27

mistercrunch added the changelog-highlight label Jul 16, 2020

villebro mentioned this pull request Aug 10, 2020

OAuth provider blocking failure #10568

Closed

3 tasks

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.37.0 labels Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(api): bump marshmallow and FAB to version 3 #9964

feat(api): bump marshmallow and FAB to version 3 #9964

dpgaspar commented Jun 2, 2020 •

edited

codecov-commenter commented Jun 2, 2020 •

edited

villebro commented Jul 3, 2020

villebro left a comment

villebro Jul 5, 2020

dpgaspar Jul 6, 2020

villebro Jul 5, 2020

villebro Jul 5, 2020

dpgaspar Jul 6, 2020

dpgaspar Jul 6, 2020

villebro left a comment

villebro left a comment

		except ValidationError as err:
		return self.response_400(message=err.messages)

feat(api): bump marshmallow and FAB to version 3 #9964

feat(api): bump marshmallow and FAB to version 3 #9964

Conversation

dpgaspar commented Jun 2, 2020 • edited

SUMMARY

Improvements and Bug fixes on 3.0.0

ADDITIONAL INFORMATION

codecov-commenter commented Jun 2, 2020 • edited

Codecov Report

villebro commented Jul 3, 2020

villebro left a comment

Choose a reason for hiding this comment

villebro Jul 5, 2020

Choose a reason for hiding this comment

dpgaspar Jul 6, 2020

Choose a reason for hiding this comment

villebro Jul 5, 2020

Choose a reason for hiding this comment

villebro Jul 5, 2020

Choose a reason for hiding this comment

dpgaspar Jul 6, 2020

Choose a reason for hiding this comment

dpgaspar Jul 6, 2020

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

dpgaspar commented Jun 2, 2020 •

edited

codecov-commenter commented Jun 2, 2020 •

edited