feat: upgrade docker image to py38 and add support for py39 #16889

villebro · 2021-09-29T07:25:00Z

SUMMARY

With Python 3.10 right around the corner (2021-10-04) and multiple performance improvements when going from 3.7 to 3.8 (see Python Speed Center benchmark summary below), we should move to 3.8 and start preparing for transitioning to 3.9. This PR bumps the official image from Python 3.7 to 3.8 and adds support for Python 3.9. CI is also bumped from 3.7 to 3.8 (btw, tox was already confgured with basepython = python3.8), and dual tests for 3.7 and 3.8 are replaced with 3.8 and 3.9.

Performance benchmark

On most benchmark tests, 3.8 (blue) performs slightly better than 3.7 (orange):

TESTING INSTRUCTIONS

Test locally that everything works as expected

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

villebro · 2021-09-29T07:29:37Z

requirements/base.txt

-sqlalchemy-utils==0.36.8
+sqlalchemy-utils==0.37.8


sqlalchemy-utils==0.36.8 has been yanked from PyPI, causing an ugly warning when installing deps. In addition, py39 was added to the officially supported versions right after 0.37.8 was cut (kvesteri/sqlalchemy-utils@b5b48b1), so I assume 0.37.8 to be fully supported by 3.9 in practice.

villebro · 2021-09-29T07:30:07Z

requirements/development.in

-pillow>=7.0.0,<8.0.0
+pillow>=8.3.1,<9


Pillow 7.x caused problems on Python 3.9

codecov · 2021-09-29T10:46:34Z

Codecov Report

Merging #16889 (38d3b1b) into master (9a8911f) will increase coverage by 0.08%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #16889      +/-   ##
==========================================
+ Coverage   76.85%   76.93%   +0.08%     
==========================================
  Files        1030     1030              
  Lines       55017    55017              
  Branches     7464     7464              
==========================================
+ Hits        42282    42330      +48     
+ Misses      12482    12434      -48     
  Partials      253      253

Flag	Coverage Δ
hive	`81.44% <100.00%> (+<0.01%)`	⬆️
javascript	`70.92% <ø> (ø)`
mysql	`81.89% <100.00%> (+0.06%)`	⬆️
postgres	`81.91% <100.00%> (+0.01%)`	⬆️
presto	`81.77% <100.00%> (?)`
python	`82.41% <100.00%> (+0.16%)`	⬆️
sqlite	`81.58% <100.00%> (+0.06%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/utils/memoized.py	`86.84% <100.00%> (+0.73%)`	⬆️
superset/views/database/views.py	`88.41% <0.00%> (-0.05%)`	⬇️
superset/reports/commands/execute.py	`91.22% <0.00%> (+0.38%)`	⬆️
superset/models/core.py	`90.00% <0.00%> (+0.73%)`	⬆️
superset/connectors/sqla/models.py	`87.24% <0.00%> (+1.41%)`	⬆️
superset/commands/importers/v1/utils.py	`91.30% <0.00%> (+2.17%)`	⬆️
superset/db_engines/hive.py	`85.18% <0.00%> (+3.04%)`	⬆️
superset/reports/commands/log_prune.py	`89.28% <0.00%> (+3.57%)`	⬆️
superset/db_engine_specs/presto.py	`90.37% <0.00%> (+6.06%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a8911f...38d3b1b. Read the comment docs.

villebro · 2021-09-29T13:41:07Z

superset/utils/memoized.py

-        return functools.partial(self.__call__, obj)
+        func = functools.partial(self.__call__, obj)
+        func.__func__ = self.func  # type: ignore
+        return func


I wasn't quite able to figure out why this is necessary, but running doctests on memoized methods started failing on 3.9 due to the doctests failing to find the __func__ attribute on memoized methods. The only thing I found is this which seems like the same problem, but is said to affect Python 3.8, not 3.9: https://bugs.python.org/issue12790 Hacky workaround, but does the trick. I'll try to figure out how to solve this more elegantly before this gets merged

amitmiran137

LGTM
Minor change is needed

amitmiran137 · 2021-09-29T15:02:14Z

setup.py

@@ -168,5 +168,6 @@ def get_git_sha() -> str:
    classifiers=[
        "Programming Language :: Python :: 3.7",


Suggested change

"Programming Language :: Python :: 3.7",

I think we're still going to support 3.7 (we support last 3 python versions officially). Once 3.10 is out, we'll remove 3.7 support in a major version bump

I want to deprecate 3.7 too, but I think we should give it some more time (it's still currently the recommended version) - let's leave it alive for a month or so so everyone can get upgraded to at least 3.8

@etr2460 @villebro do we—Apache Superset—have an official policy about supporting the last three Python versions? Just asking. If that's the case I think this is great as it means we're not unnecessarily burdened by older less feature rich versions.

@john-bodley @etr2460 I think we just lightly agreed on the last 3 versions on a community meeting. Making it an official policy is a good idea to let users know what to expect and plan ahead

@john-bodley TBH until this PR we were only really supporting two 😄 But yeah, I think there was an unwritten agreement to try to support 3. I'm fine cutting it down to "at least 2" but trying to support 3 if possible.

villebro · 2021-09-30T08:20:33Z

Opened a task on Apache Infra to bump required tests: https://issues.apache.org/jira/browse/INFRA-22377

…6889) * feat: upgrade docker image to py38 and add support for py39 * update required tests (cherry picked from commit 82601ab)

…6889) * feat: upgrade docker image to py38 and add support for py39 * update required tests

superset-github-bot bot added the preset-io label Sep 29, 2021

pull-request-size bot added the size/M label Sep 29, 2021

villebro commented Sep 29, 2021

View reviewed changes

requirements/development.in

pillow>=7.0.0,<8.0.0

pillow>=8.3.1,<9

Copy link

Member Author

villebro Sep 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pillow 7.x caused problems on Python 3.9

villebro force-pushed the villebro/upgrade-python branch from 9ecbb2d to 6dea8d0 Compare September 29, 2021 07:32

villebro changed the title ~~deps: upgrade docker image to py38 and add support for py39~~ feat: upgrade docker image to py38 and add support for py39 Sep 29, 2021

villebro force-pushed the villebro/upgrade-python branch 2 times, most recently from 1476881 to b417b90 Compare September 29, 2021 07:42

feat: upgrade docker image to py38 and add support for py39

f0583ff

villebro force-pushed the villebro/upgrade-python branch from b417b90 to f0583ff Compare September 29, 2021 10:24

villebro commented Sep 29, 2021

View reviewed changes

amitmiran137 approved these changes Sep 29, 2021

View reviewed changes

dpgaspar approved these changes Sep 30, 2021

View reviewed changes

villebro added 2 commits October 5, 2021 08:43

Merge branch 'master' into villebro/upgrade-python

8b7b170

update required tests

dd2cd65

villebro mentioned this pull request Oct 5, 2021

test: update required tests to python 3.8 #16973

Merged

9 tasks

Merge branch 'master' into villebro/upgrade-python

38d3b1b

villebro merged commit 82601ab into apache:master Oct 5, 2021

villebro deleted the villebro/upgrade-python branch October 5, 2021 08:37

villebro added the v1.4 label Oct 12, 2021

opus-42 pushed a commit to opus-42/incubator-superset that referenced this pull request Nov 14, 2021

feat: upgrade docker image to py38 and add support for py39 (apache#1…

164ba8a

…6889) * feat: upgrade docker image to py38 and add support for py39 * update required tests

QAlexBall pushed a commit to QAlexBall/superset that referenced this pull request Dec 28, 2021

feat: upgrade docker image to py38 and add support for py39 (apache#1…

d85ef55

…6889) * feat: upgrade docker image to py38 and add support for py39 * update required tests

ad-m mentioned this pull request Jan 27, 2022

fix(tests): Remove walrus operator for Python 3.7 compatiblity #18205

Merged

9 tasks

mistercrunch added 🍒 1.4.0 🍒 1.4.1 🍒 1.4.2 labels Mar 13, 2024

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.5.0 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: upgrade docker image to py38 and add support for py39 #16889

feat: upgrade docker image to py38 and add support for py39 #16889

villebro commented Sep 29, 2021 •

edited

villebro Sep 29, 2021

villebro Sep 29, 2021

codecov bot commented Sep 29, 2021 •

edited

villebro Sep 29, 2021 •

edited

dpgaspar Sep 30, 2021

amitmiran137 left a comment

amitmiran137 Sep 29, 2021

etr2460 Sep 29, 2021

villebro Sep 29, 2021

john-bodley Sep 29, 2021

dpgaspar Sep 29, 2021

villebro Sep 29, 2021

villebro commented Sep 30, 2021

		@@ -168,5 +168,6 @@ def get_git_sha() -> str:
		classifiers=[
		"Programming Language :: Python :: 3.7",

		sqlalchemy-utils==0.36.8
		sqlalchemy-utils==0.37.8

feat: upgrade docker image to py38 and add support for py39 #16889

feat: upgrade docker image to py38 and add support for py39 #16889

Conversation

villebro commented Sep 29, 2021 • edited

SUMMARY

Performance benchmark

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 29, 2021 • edited

Codecov Report

villebro Sep 29, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amitmiran137 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

villebro commented Sep 30, 2021

villebro commented Sep 29, 2021 •

edited

codecov bot commented Sep 29, 2021 •

edited

villebro Sep 29, 2021 •

edited