Bump Django to 1.11.17 and add Python 3.7 in test matrix #4551

benjaoming · 2018-12-04T21:40:39Z

Summary

This adds Python 3.7 compatibility, assuming that it's confirmed with by CI...

Reviewer guidance

Any discussion points about targeting this for 0.11.1?

References

#4548

Contributor Checklist

Contributor has fully tested the PR manually
PR has the correct target branch and milestone
PR has 'needs review' or 'work-in-progress' label
If PR is ready for review, a reviewer has been added. (Don't use 'Assignees')
If there are any front-end changes, before/after screenshots are included
If this is an important user-facing change, PR or related issue has a 'changelog' label

Reviewer Checklist

Automated test coverage is satisfactory
Reviewer has fully tested the PR manually
PR has been tested for accessibility regressions
External dependencies files were updated (yarn and pip)
Documentation is updated
Link to diff of internal dependency change is included
CHANGELOG.rst is updated for high-level changes
Contributor is in AUTHORS.rst

indirectlylit · 2018-12-05T20:27:08Z

unclear to me why tests are failing:

Traceback (most recent call last):
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/bin/kolibri", line 11, in <module>
    load_entry_point('kolibri', 'console_scripts', 'kolibri')()
  File "/home/travis/build/learningequality/kolibri/kolibri/utils/cli.py", line 642, in main
    manage(command, args=django_args)
  File "/home/travis/build/learningequality/kolibri/kolibri/utils/cli.py", line 432, in manage
    execute_from_command_line(argv=argv)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/core/management/base.py", line 327, in execute
    self.check()
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/core/management/base.py", line 359, in check
    include_deployment_checks=include_deployment_checks,
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/core/management/base.py", line 346, in _run_checks
    return checks.run_checks(**kwargs)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/core/checks/registry.py", line 81, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/core/checks/urls.py", line 16, in check_url_config
    return check_resolver(resolver)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/core/checks/urls.py", line 26, in check_resolver
    return check_method()
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/urls/resolvers.py", line 256, in check
    for pattern in self.url_patterns:
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/urls/resolvers.py", line 407, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/urls/resolvers.py", line 400, in urlconf_module
    return import_module(self.urlconf_name)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/travis/build/learningequality/kolibri/kolibri/deployment/default/urls.py", line 40, in <module>
    url(r'', include('kolibri.core.urls')),
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/conf/urls/__init__.py", line 50, in include
    urlconf_module = import_module(urlconf_module)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/travis/build/learningequality/kolibri/kolibri/core/urls.py", line 56, in <module>
    url(r'^api/', include('kolibri.core.api_urls')),
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/conf/urls/__init__.py", line 50, in include
    urlconf_module = import_module(urlconf_module)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/travis/build/learningequality/kolibri/kolibri/core/api_urls.py", line 8, in <module>
    url(r'^tasks/', include('kolibri.core.tasks.api_urls')),
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/django/conf/urls/__init__.py", line 50, in include
    urlconf_module = import_module(urlconf_module)
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/travis/build/learningequality/kolibri/kolibri/core/tasks/api_urls.py", line 3, in <module>
    from .api import TasksViewSet
  File "/home/travis/build/learningequality/kolibri/kolibri/core/tasks/api.py", line 7, in <module>
    from iceqube.common.classes import State
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/iceqube/common/classes.py", line 5, in <module>
    from iceqube.common.utils import import_stringified_func, stringify_func
  File "/home/travis/build/learningequality/kolibri/.tox/py3.7/lib/python3.7/site-packages/iceqube/common/utils.py", line 9
    from iceqube import async
                            ^

benjaoming · 2018-12-05T20:48:38Z

@indirectlylit it's because of async, its a syntax keyword now.

learningequality/iceqube#34

https://docs.python.org/3/library/asyncio.html

benjaoming · 2018-12-07T15:00:18Z

Doing some work for the Iceqube CI, so we can have automated tests approve these changes...

benjaoming · 2018-12-13T20:54:19Z

This PR is pending that I have credentials for PyPi to release another Iceqube...

rtibbles · 2019-01-03T23:05:48Z

Hrm, sorry, I don't have creds for icequebe on pypi either.

benjaoming · 2019-01-04T09:52:58Z

Iceqube 0.1b1 is released!

codecov · 2019-01-04T09:54:26Z

Codecov Report

Merging #4551 into release-v0.11.x will increase coverage by 1.79%.
The diff coverage is n/a.

@@                 Coverage Diff                 @@
##           release-v0.11.x    #4551      +/-   ##
===================================================
+ Coverage            50.13%   51.92%   +1.79%     
===================================================
  Files                  866      739     -127     
  Lines                28715    24385    -4330     
  Branches              4154     3304     -850     
===================================================
- Hits                 14395    12661    -1734     
+ Misses               13503    10959    -2544     
+ Partials               817      765      -52

Impacted Files	Coverage Δ
.../learn/assets/src/views/ContentUnavailablePage.vue	`75% <0%> (-25%)`	⬇️
...olibri/core/assets/src/views/CoachContentLabel.vue	`20% <0%> (-20%)`	⬇️
...libri/core/assets/src/mixins/responsive-element.js	`85.71% <0%> (-14.29%)`	⬇️
...assets/src/views/assignments/RecipientSelector.vue	`42.1% <0%> (-12.74%)`	⬇️
kolibri/core/content/views.py	`82.08% <0%> (-10.86%)`	⬇️
...coach/assets/src/views/reports/ChannelListPage.vue	`62.16% <0%> (-10.82%)`	⬇️
...t/assets/src/views/AvailableChannelsPage/index.vue	`73.41% <0%> (-10.24%)`	⬇️
.../core/assets/src/views/CoreMenu/CoreMenuOption.vue	`70% <0%> (-10%)`	⬇️
kolibri/core/assets/src/views/CoreMenu/index.vue	`16.66% <0%> (-9.65%)`	⬇️
...ent_epub_render/assets/src/views/SearchSideBar.vue	`33.33% <0%> (-9.34%)`	⬇️
... and 267 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 03c69e9...3d67508. Read the comment docs.

codecov · 2019-01-04T09:54:26Z

Codecov Report

❗ No coverage uploaded for pull request base (release-v0.11.x@b549aa7). Click here to learn what that means.
The diff coverage is n/a.

@@                Coverage Diff                 @@
##             release-v0.11.x    #4551   +/-   ##
==================================================
  Coverage                   ?   51.96%           
==================================================
  Files                      ?      739           
  Lines                      ?    24403           
  Branches                   ?     3308           
==================================================
  Hits                       ?    12682           
  Misses                     ?    10955           
  Partials                   ?      766

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b549aa7...bd7a422. Read the comment docs.

benjaoming · 2019-01-04T10:14:34Z

What what what. @lyw07 have you seen this before?

python build_tools/install_cexts.py --file "requirements/cext.txt" # pip install c extensions
  Could not find a version that satisfies the requirement cffi!=1.11.3,>=1.7 (from cryptography==2.3) (from versions: 0.7.1, 0.8, 0.8.4, 1.1.1, 1.1.2, 1.4.1, 1.5.2)
No matching distribution found for cffi!=1.11.3,>=1.7 (from cryptography==2.3)
 
Download failed for package cryptography-2.3-cp36-cp36m-linux_armv7l.whl.

lyw07 · 2019-01-04T10:18:31Z

Ah yes. I just created a PR(#4678) targeting 0.9 to fix it. Will get the fix in to branches above 0.9 after the PR is merged.
The error is because in the arm wheel repository we use, they have cryptography py36 but don’t have cffi py36. Their build for cffi py36 might be failed. I have commented on their github issue to ask about it. But before that, I think the workaround would be to skip downloading python 3.6 Linux arm cryptography package..

benjaoming · 2019-01-04T10:28:06Z

Thanks so much, that makes total sense -- sorry haven't familiarized myself with all the latest work of 2019 :)

benjaoming · 2019-01-04T12:17:02Z

There are a handful of Python 3.7 related issues in tests that I will investigate, and the issue with the missing ARM build for Cryptography isn't really a blocker for fixing this.

I'll try and dig into it and assess if its feasible to fix for 0.11.1 -- and if not, retargetting this PR is easy.

rtibbles

This is looking good, and I think would be good to include in 0.11.1 if at all possible.

rtibbles · 2019-01-08T18:36:19Z

.travis.yml

@@ -32,9 +32,17 @@ matrix:
    - python: "3.6"
      env:
        - TOX_ENV=pythonbuild3.6
+    - python: 3.7
+      dist: xenial
+      sudo: true


Is this a travis requirement to run with sudo true for Python3.7?

Yes, it's because it obtains Python 3.7 from some addition source and installs it. We cannot specify "Python 3.7" without these extra options. It's been the case for quite some time now, but Travis seems to need some time for releasing new images.

kolibri/core/content/templatetags/content_tags.py

rtibbles · 2019-01-08T18:38:16Z

kolibri/core/webpack/utils.py

+
+    # Backwards compatibility: The reserved keyword 'async' was used before.
+    # May be removed in 0.12+.
+    is_async = kwargs.pop('async', is_async)


Good good - yeah, maybe add a deprecation warning if kwargs.pop('async') returns something?

Good point, let's have them visible! Fixed!

rtibbles · 2019-01-08T18:39:04Z

kolibri/plugins/coach/templatetags/coach_tags.py

 from django import template
-from kolibri.core.webpack.utils import webpack_asset_render


I thought we had already linted every file? When we run CI linting, does it not run our import linting too?

This may not be seen by a linting rule but is likely only caused by changes to our pre-commit hooks.

rtibbles

Mr. Travis will have the final say, but this gets my vote!

rtibbles · 2019-01-09T00:31:16Z

Two tiny linting errors on the Python3 lint.

Ignore undefined linting of Py2 only globals in Py2 only code blocks.

benjaoming · 2019-01-09T13:25:31Z

Yup! CC: @radinamatic - we could need a bit of Python 3.7 testing before announcing the support. I'll have a look at building a Debian package that doesn't force any Python downgrades.

benjaoming added TODO: needs review Waiting for review changelog Important user-facing changes TAG: tech debt labels Dec 4, 2018

benjaoming added this to the 0.11.1 milestone Dec 4, 2018

benjaoming requested review from rtibbles and indirectlylit December 4, 2018 21:40

benjaoming mentioned this pull request Dec 5, 2018

Rename reserved keyword "async" for Python 3.7 compatibility. learningequality/iceqube#34

Merged

indirectlylit added TAG: tech update / debt Change not visible to user and removed TAG: tech update / debt Change not visible to user TAG: tech debt labels Dec 5, 2018

benjaoming force-pushed the django1.11.17 branch from 128e71b to 3d67508 Compare January 4, 2019 09:54

benjaoming force-pushed the django1.11.17 branch from 6fda6b4 to 5dbc61b Compare January 8, 2019 20:38

rtibbles reviewed Jan 8, 2019

View reviewed changes

benjaoming force-pushed the django1.11.17 branch 2 times, most recently from ad4f887 to 8d77ed9 Compare January 8, 2019 22:49

Bump Django to 1.11.17 and add Python 3.7 in test matrix

6277d24

Benjamin Bach added 4 commits January 9, 2019 00:14

Bump Iceqube to 0.1b1

d26bc4b

Fix occurrences of async Py 3.7 syntax keyword

7069f64

Separate linting into python 2 and 3 and run both on Travis

63cf7a0

Raise a deprecation warning when using deprecated keyword

888ee85

benjaoming force-pushed the django1.11.17 branch from 8d77ed9 to 888ee85 Compare January 8, 2019 23:14

rtibbles approved these changes Jan 8, 2019

View reviewed changes

rtibbles and others added 2 commits January 8, 2019 16:37

Ignore undefined linting of Py2 only globals in Py2 only code blocks.

e6157c9

Merge pull request #6 from rtibbles/django1.11.17

bd7a422

Ignore undefined linting of Py2 only globals in Py2 only code blocks.

benjaoming merged commit ec644bf into learningequality:release-v0.11.x Jan 9, 2019

rtibbles mentioned this pull request Feb 25, 2019

Lint for all Python versions #551

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump Django to 1.11.17 and add Python 3.7 in test matrix #4551

Bump Django to 1.11.17 and add Python 3.7 in test matrix #4551

benjaoming commented Dec 4, 2018

indirectlylit commented Dec 5, 2018

benjaoming commented Dec 5, 2018

benjaoming commented Dec 7, 2018

benjaoming commented Dec 13, 2018

rtibbles commented Jan 3, 2019

benjaoming commented Jan 4, 2019

codecov bot commented Jan 4, 2019

codecov bot commented Jan 4, 2019 •

edited

Loading

benjaoming commented Jan 4, 2019

lyw07 commented Jan 4, 2019 •

edited

Loading

benjaoming commented Jan 4, 2019

benjaoming commented Jan 4, 2019

rtibbles left a comment

rtibbles Jan 8, 2019

benjaoming Jan 8, 2019

rtibbles Jan 8, 2019

benjaoming Jan 8, 2019

rtibbles Jan 8, 2019

benjaoming Jan 8, 2019

rtibbles left a comment

rtibbles commented Jan 9, 2019

benjaoming commented Jan 9, 2019

		from django import template
		from kolibri.core.webpack.utils import webpack_asset_render

Bump Django to 1.11.17 and add Python 3.7 in test matrix #4551

Bump Django to 1.11.17 and add Python 3.7 in test matrix #4551

Conversation

benjaoming commented Dec 4, 2018

Summary

Reviewer guidance

References

Contributor Checklist

Reviewer Checklist

indirectlylit commented Dec 5, 2018

benjaoming commented Dec 5, 2018

benjaoming commented Dec 7, 2018

benjaoming commented Dec 13, 2018

rtibbles commented Jan 3, 2019

benjaoming commented Jan 4, 2019

codecov bot commented Jan 4, 2019

Codecov Report

codecov bot commented Jan 4, 2019 • edited Loading

Codecov Report

benjaoming commented Jan 4, 2019

lyw07 commented Jan 4, 2019 • edited Loading

benjaoming commented Jan 4, 2019

benjaoming commented Jan 4, 2019

rtibbles left a comment

Choose a reason for hiding this comment

rtibbles Jan 8, 2019

Choose a reason for hiding this comment

benjaoming Jan 8, 2019

Choose a reason for hiding this comment

rtibbles Jan 8, 2019

Choose a reason for hiding this comment

benjaoming Jan 8, 2019

Choose a reason for hiding this comment

rtibbles Jan 8, 2019

Choose a reason for hiding this comment

benjaoming Jan 8, 2019

Choose a reason for hiding this comment

rtibbles left a comment

Choose a reason for hiding this comment

rtibbles commented Jan 9, 2019

benjaoming commented Jan 9, 2019

codecov bot commented Jan 4, 2019 •

edited

Loading

lyw07 commented Jan 4, 2019 •

edited

Loading