Use requests instead of urllib2

[jianli]

Using requests instead of urllib2 will prevent an exception from being raised in the case of a 404. For example, this fixes #16.

I also added some minor pep8 whitespace changes.

[gleitz]

Cool! Would you mind adding the requests library as a dependency in both setup.py and requirements.txt?

[jianli]

Cool! Would you mind adding the requests library as a dependency in both setup.py and requirements.txt?

Yep, done.

[gleitz]

Does this patch work for you? I receive the error "HTTP Error 400. The request has an invalid header name."

Please check. I believe you need to use "User-Agent" instead of "User-agent"

[jianli]

That's strange; I've tested it and it works for me.

Does changing it to "User-Agent" fix it for you?

Maybe we have different versions of requests? I have 0.13.3.

[hughdbrown]

I did this PR here: #23

Abandoned because I also added some gratuitous changes. Anyway.

Yes, use User-Agent.

[jianli]

Ok, I made the change from User-agent to User-Agent.

Just to be clear, are @gleitz and @hughdbrown getting Python exceptions with the text "HTTP Error 400" when using "User-agent"?

I can't reproduce the error, even if I purposely omit the header .

In that case, I merely get a version of the google results page for which html('.l') returns an empty list and the end result is "Sorry, couldn't find any help with that topic". Saving that html to a file and opening in my browser reveals a results page which is styled slightly differently.

[hughdbrown]

Allowing that I have a slightly broken/non-standard howdoi, try this with my repo:

cd ~/workspace
git clone git@github.com:hughdbrown/howdoi.git
cd howdoi
mkvirtualenv howdoi
pip install -r requirements.txt
cd howdoi

python howdoi.py format string bash
python howdoi.py print stack trace python
python howdoi.py -p=3 convert mp4 to animated gif
python howdoi.py -p=2 create tar archive

sed -i 's/User-Agent/User-agent/g' howdoi.py

python howdoi.py format string bash
python howdoi.py print stack trace python
python howdoi.py -p=3 convert mp4 to animated gif
python howdoi.py -p=2 create tar archive

You will find the first four work in a standard way and the last four all fail. The only difference is the change of User-Agent to User-agent.

Here is my pip freeze:

 ~/workspace/howdoi/howdoi@master✗✗✗ hughdbrown% pip freeze
Pygments==1.6rc1
argparse==1.2.1
cssselect==0.7.1
lxml==3.0.1
pyquery==1.2.2
requests==1.0.4
wsgiref==0.1.2

[gleitz]

I am also on requests==1.0.4. Seems 0.13.3 is very old.

On Wed, Jan 9, 2013 at 8:10 PM, Hugh Brown notifications@github.com wrote:

Allowing that I have a slightly broken/non-standard howdoi, try this with
my repo:

cd ~/workspace
git clone git@github.com:hughdbrown/howdoi.git
cd howdoi
mkvirtualenv howdoi
pip install -r requirements.txt
cd howdoi

python howdoi.py format string bash
python howdoi.py print stack trace python
python howdoi.py -p=3 convert mp4 to animated gif
python howdoi.py -p=2 create tar archive

sed -i 's/User-Agent/User-agent/g' howdoi.py

python howdoi.py format string bash
python howdoi.py print stack trace python
python howdoi.py -p=3 convert mp4 to animated gif
python howdoi.py -p=2 create tar archive

You will find the first four work in a standard way and the last four all
fail. The only difference is the change of User-Agent to User-agent.

Here is my pip freeze:

~/workspace/howdoi/howdoi@master✗✗✗ hughdbrown% pip freeze
Pygments==1.6rc1
argparse==1.2.1
cssselect==0.7.1
lxml==3.0.1
pyquery==1.2.2
requests==1.0.4
wsgiref==0.1.2

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/18#issuecomment-12079975.

[jianli]

@gleitz Ah, that solves the mystery. After updating requests, "User-agent" causes the StackOverflow query to always return 400.