unbundle requests and use requests.get instead of urllib.urlretrieve by benjaoming · Pull Request #5138 · learningequality/ka-lite

benjaoming · 2016-06-06T16:42:42Z

Summary

Fixes issues with m2crypto not able to handle https connections.

TODO

If not all TODOs are marked, this PR is considered WIP (work in progress)

Have tests been written for the new code? If you're fixing a bug, write a regression test (or have a really good reason for not writing one... and I mean really good!)
Has documentation been written/updated?
New dependencies (if any) added to requirements file

Reviewer guidance

If there's any concern about including this PR in 0.16.6, please state it in the discussion, but I would caution that the old download method was probably less reliable than using the latest version of requests @jamalex

Issues addressed

#5126

kalite/distributed/models.py

-# (this is done in this file because putting it in __init__.py causes circular imports,
-# and putting it in urls.py doesn't get loaded by management commands)
-base_headers = requests.defaults.defaults["base_headers"]
-base_headers["User-Agent"] = ("ka-lite/%s " % VERSION) + base_headers["User-Agent"]


benjaoming · 2016-06-06T17:46:43Z

This is where the requests library is used currently, I don't see any reason of concern, but should we change the User-Agent on any of these requests @jamalex ?

kalite/distributed/static/js/distributed/perseus/ke/build/update_title_tags.py
9:import requests
13:    r = requests.get("http://www.khanacademy.org/api/internal/exercises")

kalite/distributed/static/js/distributed/perseus/ke/build/close-old-issues.py
10:import requests
39:        r = requests.get(url, auth=github_auth)
52:    r = requests.get("https://api.github.com/repos/Khan/khan-exercises/issues/"
65:    r = requests.post("https://api.github.com/repos/Khan/khan-exercises/"
73:    except requests.HTTPError:
83:    r = requests.post("https://api.github.com/repos/Khan/khan-exercises/"
91:    except requests.HTTPError:

kalite/i18n/management/commands/create_dummy_language_pack.py
13:import requests
57:        resp = requests.get(url)
59:    except requests.ConnectionError as e:

kalite/i18n/management/commands/setup_in_context_translations.py
5:import requests
40:        resp = requests.get(url)

kalite/i18n/tests/create_dummy_language_pack_tests.py
1:import requests
17:    @patch.object(requests, "get", autospec=True)

kalite/static/js/distributed/perseus/ke/build/update_title_tags.py
9:import requests
13:    r = requests.get("http://www.khanacademy.org/api/internal/exercises")

kalite/static/js/distributed/perseus/ke/build/close-old-issues.py
10:import requests
39:        r = requests.get(url, auth=github_auth)
52:    r = requests.get("https://api.github.com/repos/Khan/khan-exercises/issues/"
65:    r = requests.post("https://api.github.com/repos/Khan/khan-exercises/"
73:    except requests.HTTPError:
83:    r = requests.post("https://api.github.com/repos/Khan/khan-exercises/"
91:    except requests.HTTPError:

kalite/contentload/management/commands/unpack_assessment_zip.py
1:import requests
47:            r = requests.get(ziplocation, prefetch=False)

kalite/contentload/tests/unpack_assessment_zip.py
4:import requests
52:    @patch.object(requests, "get")
60:        self.assertEqual(get_method.call_count, 0, "requests.get was called even if we should've skipped!")
66:    @patch.object(requests, "get", autospec=True)

benjaoming · 2016-06-06T19:24:20Z

Confirmed working on Raspberry Pi with m2crypto 0.21.1 installed as a dependency of ka-lite-raspberry-pi.

python-packages/fle_utils/internet/download.py

-
-    response = None
-    last_error = None
-    for retries in range(1, 1 + max_retries):


jamalex · 2016-06-06T21:22:14Z

This is where the requests library is used currently, I don't see any reason of concern, but should we change the User-Agent on any of these requests @jamalex ?

Most of the ones you list there don't seem important (as they're not hitting our central server, so we won't be getting logs with the user agent in it for those), but that doesn't seem to be an exhaustive list. Here are some that are more important:
https://github.com/learningequality/ka-lite/blob/develop/python-packages/fle_utils/internet/functions.py#L22
https://github.com/learningequality/ka-lite/blob/develop/python-packages/securesync/api_client.py#L14
https://github.com/learningequality/ka-lite/blob/develop/python-packages/securesync/devices/views.py#L100
(I know these are under python-packages, but these aren't actually external python packages; that code is only used in the context of KA Lite)

Might be good to pull out this header calculation logic into a reusable function:
https://github.com/learningequality/ka-lite/pull/5138/files#diff-5e841025f58a948893e991da20eeebaaR64
(though not sure the best way to avoid having securesync and fle_utils depend on kalite...)

python-packages/fle_utils/videos.py

-                "Content-Length" not in response or
-                not str(len(open(thumb_filepath, "rb").read())) == response["Content-Length"]):
+                "content-length" not in response.headers or
+                not len(open(thumb_filepath, "rb").read()) == int(response.headers['content-length'])):


benjaoming · 2016-06-06T22:18:46Z

(though not sure the best way to avoid having securesync and fle_utils depend on kalite...)

They already did when kalite was monkey-patching requests :)

benjaoming · 2016-06-06T22:19:17Z

Thanks for searching through fle_utils btw, adding that change as well.

jamalex · 2016-06-06T22:25:27Z

They already did when kalite was monkey-patching requests :)

Kind of soft dependency; implicit, and optional. They would still function fine without kalite. More like a "hook" style "dependency". :D

jamalex · 2016-06-06T22:27:32Z

Thanks for searching through fle_utils btw, adding that change as well.

Cool -- there were a bunch of other uses of requests in there, but they didn't seem so critical. The ones I listed are the places that hit the central server in ways we may want to examine the logs for.

benjaoming · 2016-06-06T22:39:40Z

They would still function fine without kalite.

If the intention was to have correct user agent stats, you would be failing but much more severely than a circular import, you'd have wrong stats and think they were right! So all this functionality really depends on the monkey patch firing.

benjaoming · 2016-06-06T22:39:51Z

@jamalex done in 5884508

jamalex · 2016-06-07T17:21:36Z

Ok, looks good to me! Merging now, but please make sure data syncing also still works (with a basic check -- registering and creating a local record and syncing it up, with no errors), in addition to the video downloading, as both depend on requests.

benjaoming · 2016-06-08T13:44:17Z

@jamalex

Tested:

One-click Device registration OK
Sync'ing OK
Video download OK

jamalex · 2016-06-08T17:22:34Z

💥 👍 ⚡

unbundle requests and use requests.get instead of urllib.urlretrieve

74faa2e

benjaoming added the bug label Jun 6, 2016

benjaoming added this to the 0.16.6 milestone Jun 6, 2016

benjaoming assigned jamalex Jun 6, 2016

benjaoming added the has PR label Jun 6, 2016

benjaoming reviewed Jun 6, 2016
View reviewed changes

Remove broken unused function make_request

da6a582

jamalex reviewed Jun 6, 2016
View reviewed changes

python-packages/fle_utils/internet/download.py

response = None

last_error = None

for retries in range(1, 1 + max_retries):

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

benjaoming reviewed Jun 6, 2016
View reviewed changes

Put our own User-Agent in all requests made to learningequality.org

5884508

benjaoming mentioned this pull request Jun 7, 2016

Videos not Updated after Download Process #5139

Closed

jamalex merged commit 1164dc6 into learningequality:0.16.x Jun 7, 2016

jamalex removed the has PR label Jun 7, 2016

benjaoming deleted the m2crypto-dl branch June 9, 2016 20:46

This was referenced Jun 13, 2016

Dependency management: Unbundle python-packages #3403

Closed

Unbundle requests #3718

Closed

This was referenced Jul 26, 2016

Fix Unclean shutdown in OS X installer. #5212

Closed

Disable unpack assessment zip command tests #5219

Merged

Conversation

benjaoming commented Jun 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

TODO

Reviewer guidance

Issues addressed

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

benjaoming commented Jun 6, 2016

Uh oh!

benjaoming commented Jun 6, 2016

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

jamalex commented Jun 6, 2016

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

benjaoming commented Jun 6, 2016

Uh oh!

benjaoming commented Jun 6, 2016

Uh oh!

jamalex commented Jun 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jamalex commented Jun 6, 2016

Uh oh!

benjaoming commented Jun 6, 2016

Uh oh!

benjaoming commented Jun 6, 2016

Uh oh!

jamalex commented Jun 7, 2016

Uh oh!

benjaoming commented Jun 8, 2016

Uh oh!

jamalex commented Jun 8, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

benjaoming commented Jun 6, 2016 •

edited

Loading

jamalex commented Jun 6, 2016 •

edited

Loading