Support for Python 3.9 #1197

shagunsodhani · 2020-12-11T19:58:23Z

Motivation

Support for Python 3.9

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

CI

Related Issues and PRs

(Is this PR part of a group of changes? Link the other relevant PRs and Issues here. Use https://help.github.com/en/articles/closing-issues-using-keywords for help on GitHub syntax)

#1062

shagunsodhani · 2020-12-12T00:04:59Z

Code changes:

importlib_resources is not available in Python 3.9 and we switch to importlib.resources. Relevant change
In Python 3.9, traceback displays the absolute path for __main__ module frames. This breaks some tests where we match the stack trace of the error. For example

Stack trace with Python < 3.9

...
"File "my_app.py", line 13, in my_app" 
...

Stack trace with Python 3.9

...
"File "/private/home/sodhani/projects/hydra/tests/test_apps/app_with_runtime_config_error/my_app.py", line 13, in my_app"
...

We have added a function to perform regex-like match on the stack trace.

Questions

Supporting windows: We havent supported windows since 3.7. I tried enabling support for 3.9 but ran into a bunch of errors. What is the stance on support for windows + Python 3.9
How do we go about updating the plugins? I can update the setup.py for all of them in this PR or make n other PRs.

shagunsodhani · 2020-12-12T00:18:28Z

One more question: I had to bump hydra version in setup.py of discovery test plugin. I understand why it should be bumped. I am not sure why did this not throw an error earlier (since we have been on Hydra 1.1* for some time now).

omry · 2020-12-12T07:34:47Z

Try to enable windows support before landing this:
I disabled it in Re-enable CI for Windows #1056 due to some issues mentioned in there.
plugins:
You can add Python 3.9 testing for all plugins as a part of this PR. If any of them is causing issue you can create a separate issue to track it.
discovery test:
dunno why it worked before. I think you can just require hydra-core without a version (it should be installed from the repo anyway).

omry

Very nice.
a few comments and see my answer to your questions.

hydra/_internal/core_plugins/importlib_resources_config_source.py

tests/standalone_apps/discovery_test_plugin/setup.py

tests/test_hydra.py

shagunsodhani · 2020-12-12T12:25:16Z

Try to enable windows support before landing this:
I disabled it in Re-enable CI for Windows #1056 due to some issues mentioned in there.

The suggested fix there is to use older version of virtualenv (== 20.0.33) which was released one day before python 3.9 :(

omry · 2020-12-12T17:36:29Z

Try to enable windows support before landing this:
I disabled it in Re-enable CI for Windows #1056 due to some issues mentioned in there.

The suggested fix there is to use older version of virtualenv (== 20.0.33) which was released one day before python 3.9 :(

Is pinning virtualenv to 20.0.33 for now a viable solution until conda and virtualenv figure out their shit?
If now, we may try to skip conda on Windows and install directly in a virtualenv on circle.

.circleci/config.yml

hydra/_internal/core_plugins/importlib_resources_config_source.py

hydra/test_utils/test_utils.py

noxfile.py

shagunsodhani · 2020-12-12T17:57:07Z

Is pinning virtualenv to 20.0.33 for now a viable solution until conda and virtualenv figure out their shit?
If now, we may try to skip conda on Windows and install directly in a virtualenv on circle.

I mean that python 3.9 does not virtualenv==20.0.33. Can try direct installation on virtualenv

omry · 2020-12-12T18:04:42Z

Is pinning virtualenv to 20.0.33 for now a viable solution until conda and virtualenv figure out their shit?
If now, we may try to skip conda on Windows and install directly in a virtualenv on circle.

I mean that python 3.9 does not virtualenv==20.0.33. Can try direct installation on virtualenv

What does that mean?

…t match

shagunsodhani · 2020-12-14T14:15:05Z

Force pushed to rebase

plugins/hydra_optuna_sweeper/setup.py

omry

Looking good, please clean up your comments.

hydra/_internal/core_plugins/importlib_resources_config_source.py

hydra/test_utils/test_utils.py

plugins/hydra_nevergrad_sweeper/setup.py

plugins/hydra_optuna_sweeper/tests/test_optuna_sweeper_plugin.py

omry

lgtm

* Fix importlib error when using py3.9 * Fix lint errors * Add a method to match lines based on regex * fix lint error * bump hydra version in discovery test plugin * Add python 3.9 in setup.py * Skip python 3.9 tests on windows * use assert_text_same when lengths of actual and expected output do not match * Remove dependency on hydra version * Check python version to decide if importlib_resources should be imported * Update the install script for plugins to reflect support for python 3.9 * pin virtualenv to 20.0.33 for windows * Fix lint error * Enable CI for 3.9 on windows * Update circle ci config * Update circle ci config * Fix issues with windows * Skip Ray test for Python 3.9 * Incorporate feedback * Fix circleci config * Fix lint errors * Fix circleci config * Fix some ci errors * Fix lint error * Fix lint error * Update setup.py for some plugins * Disable windows 9 support * Ax plugin doesnt support 3.9 for now * Nevergrad python 3.9 support depends on Scikit-learn * Add news entries * Accidentally removed support for optuna on windows * Incorporate feedback * Disable windows test for optuna

* Support for Python 3.9 (#1197) * Fix importlib error when using py3.9 * Fix lint errors * Add a method to match lines based on regex * fix lint error * bump hydra version in discovery test plugin * Add python 3.9 in setup.py * Skip python 3.9 tests on windows * use assert_text_same when lengths of actual and expected output do not match * Remove dependency on hydra version * Check python version to decide if importlib_resources should be imported * Update the install script for plugins to reflect support for python 3.9 * pin virtualenv to 20.0.33 for windows * Fix lint error * Enable CI for 3.9 on windows * Update circle ci config * Update circle ci config * Fix issues with windows * Skip Ray test for Python 3.9 * Incorporate feedback * Fix circleci config * Fix lint errors * Fix circleci config * Fix some ci errors * Fix lint error * Fix lint error * Update setup.py for some plugins * Disable windows 9 support * Ax plugin doesnt support 3.9 for now * Nevergrad python 3.9 support depends on Scikit-learn * Add news entries * Accidentally removed support for optuna on windows * Incorporate feedback * Disable windows test for optuna * Remove some commented statements

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 11, 2020

shagunsodhani marked this pull request as draft December 11, 2020 19:58

shagunsodhani marked this pull request as ready for review December 12, 2020 00:12

shagunsodhani requested review from omry and jieru-hu and removed request for omry December 12, 2020 00:12

omry reviewed Dec 12, 2020

View reviewed changes

hydra/_internal/core_plugins/importlib_resources_config_source.py Outdated Show resolved Hide resolved

tests/standalone_apps/discovery_test_plugin/setup.py Outdated Show resolved Hide resolved

tests/test_hydra.py Show resolved Hide resolved

omry reviewed Dec 12, 2020

View reviewed changes

shagunsodhani requested a review from omry December 14, 2020 13:52

shagunsodhani added 13 commits December 14, 2020 06:13

Fix importlib error when using py3.9

7b360ab

Fix lint errors

b0733d5

Add a method to match lines based on regex

c46ceef

fix lint error

d7ac56c

bump hydra version in discovery test plugin

fd7d851

Add python 3.9 in setup.py

d6a8353

Skip python 3.9 tests on windows

6a6919d

use assert_text_same when lengths of actual and expected output do no…

75564c3

…t match

Remove dependency on hydra version

0cd2c91

Check python version to decide if importlib_resources should be imported

cb08a11

Update the install script for plugins to reflect support for python 3.9

821b4d5

pin virtualenv to 20.0.33 for windows

a21dca9

Fix lint error

0a871a0

shagunsodhani added 14 commits December 14, 2020 06:13

Update circle ci config

7c44f29

Fix issues with windows

a3bb8b9

Skip Ray test for Python 3.9

d115c8f

Incorporate feedback

8364afb

Fix circleci config

a267dd5

Fix lint errors

18f1c3c

Fix circleci config

e55b904

Fix some ci errors

f9f4555

Fix lint error

f5fdf25

Fix lint error

c6da725

Update setup.py for some plugins

c79f7f0

Disable windows 9 support

fec551d

Ax plugin doesnt support 3.9 for now

720abcd

Nevergrad python 3.9 support depends on Scikit-learn

ab772c5

shagunsodhani force-pushed the py3.9 branch from 2461389 to ab772c5 Compare December 14, 2020 14:14

Add news entries

7c7a850

jieru-hu reviewed Dec 15, 2020

View reviewed changes

plugins/hydra_optuna_sweeper/setup.py Show resolved Hide resolved

Accidentally removed support for optuna on windows

2ffac4b

omry reviewed Dec 16, 2020

View reviewed changes

Incorporate feedback

4c69b25

shagunsodhani mentioned this pull request Dec 16, 2020

[Optuna-Plugin] Add support for Python 3.9 #1204

Closed

Disable windows test for optuna

70e64d2

omry approved these changes Dec 18, 2020

View reviewed changes

shagunsodhani merged commit 93eea6d into facebookresearch:master Dec 18, 2020

noChamp mentioned this pull request Dec 23, 2020

Release new version of Hydra #1234

Closed

shagunsodhani mentioned this pull request Dec 25, 2020

[Feature Request] Add support for Python 3.9 #1062

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Python 3.9 #1197

Support for Python 3.9 #1197

shagunsodhani commented Dec 11, 2020

shagunsodhani commented Dec 12, 2020 •

edited

shagunsodhani commented Dec 12, 2020

omry commented Dec 12, 2020

omry left a comment

shagunsodhani commented Dec 12, 2020

omry commented Dec 12, 2020

shagunsodhani commented Dec 12, 2020

omry commented Dec 12, 2020

shagunsodhani commented Dec 14, 2020

omry left a comment

omry left a comment

Support for Python 3.9 #1197

Support for Python 3.9 #1197

Conversation

shagunsodhani commented Dec 11, 2020

Motivation

Have you read the Contributing Guidelines on pull requests?

Test Plan

Related Issues and PRs

shagunsodhani commented Dec 12, 2020 • edited

Code changes:

Questions

shagunsodhani commented Dec 12, 2020

omry commented Dec 12, 2020

omry left a comment

Choose a reason for hiding this comment

shagunsodhani commented Dec 12, 2020

omry commented Dec 12, 2020

shagunsodhani commented Dec 12, 2020

omry commented Dec 12, 2020

shagunsodhani commented Dec 14, 2020

omry left a comment

Choose a reason for hiding this comment

omry left a comment

Choose a reason for hiding this comment

shagunsodhani commented Dec 12, 2020 •

edited