Use python-requests instead of urllib/urllib2

[emijrp]

From nemow...@gmail.com on February 27, 2014 01:08:47

Smart folks nowadays only use Requests, we should probably adopt it everywhere. It may be one thing more to import for our users but let's track how many things it will probably help with.

Original issue: http://code.google.com/p/wikiteam/issues/detail?id=104

[emijrp]

From nemow...@gmail.com on February 26, 2014 16:11:06

Blocking: wikiteam:46

[emijrp]

From nemow...@gmail.com on February 26, 2014 16:11:06

Blocking: wikiteam:103

[emijrp]

From nemow...@gmail.com on February 26, 2014 16:11:16

Blocking: wikiteam:102

[emijrp]

From nemow...@gmail.com on February 26, 2014 16:11:16

Blocking: wikiteam:78

[emijrp]

From nemow...@gmail.com on June 04, 2014 06:23:03

Summary: Use python-requests instead of urllib/urllib2 (was: Reasons to use python-requests instead of urllib/urllib2)

[nemobis]

balrog is mostly done implementing this, hurray! He also has fixed some Unicode handling bugs while at it (perhaps some of those have been separately fixed in master my emijrp though). Watch your patches to avoid overlapping work in the next day or so.

[balr0g]

I'm also trying to rewrite this to use argparse to prevent other issues — currently the argument-parsing code is quite a mess.

[nemobis]

balr0g, 01/07/2014 00:39:

I'm also trying to rewrite this to use argparse to prevent other issues
— currently the argument-parsing code is quite a mess.

Great. That's #115 . So many
bugs at once. :-)

[nemobis]

@balr0g, I get:

$ python dumpgenerator.py --xml --images --api=http://www.wikihow.com/api.php
You didn't provide a path for index.php, using http://www.wikihow.com/index.php
Traceback (most recent call last):
File "dumpgenerator.py", line 1291, in
main()
File "dumpgenerator.py", line 1245, in main
config, other = getParameters(params=params)
File "dumpgenerator.py", line 947, in getParameters
if checkAPI(config['api'], config, session=other['session']):
File "dumpgenerator.py", line 970, in checkAPI
r = session.post(url=api, data={'action': 'query', 'meta': 'siteinfo', 'format': 'json'})
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 241, in post
return self.request('post', url, data=data, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 194, in request
self.cookies.update(r.response.cookies)
AttributeError: MozillaCookieJar instance has no attribute 'update'
$ python --version
Python 2.7.3

[emijrp]

Can you please stop the dump generation until we finish the major revamp we are doing, and publish a tested and stable version?

[nemobis]

Emilio J. Rodríguez-Posada, 04/07/2014 14:36:

Can you please stop the dump generation until we finish the major revamp
we are doing, and publish a tested and stable version?

That was only a test, I'm running the old code now for the dumps.

[balr0g]

@nemobis: I think you're running an old version of Requests: https://github.com/kennethreitz/requests/issues/1711

[nemobis]

balr0g, 04/07/2014 16:28:

@nemobis https://github.com/nemobis: I think you're running an old
version of Requests: kennethreitz/requests#1711
https://github.com/kennethreitz/requests/issues/1711

Probably; I didn't install it on that machine. Can we check for the
requests version and fallback in some way or give upgrade directions? That is "pip install --user --upgrade requests" :)

We have people running dumpgenerator on python 2.6 still...

[balr0g]

pip install --upgrade -r requirements.txt

[nemobis]

OK, saw 1391cf8 now; added to https://github.com/WikiTeam/wikiteam/blob/master/README.md#download-any-wiki.
I'm closing this, let's go through the dependencies (#46, #103, #102, #78) and file issues separately.