python 3 issues with working-with-web-pages

[fredgibbs]

Hi! I'm enjoying learning to work with text files and web pages using Programming Historian.

I installed Python 3.6 and have run into a couple major differences from how the directions are written. Once I power through the lessons and hopefully do very well in a job interview, I'd be happy to help update the directions with more detail.

For now, this is what I've run into on http://programminghistorian.org/lessons/working-with-web-pages :

• import urllib2 does not work. StackOverflow kindly suggested the fix "from urllib.request import urlopen"
• f = open('obo-t17800628-33.html', 'w') no longer works. Again, StackOverflow pointed out that it needs to be opened in binary: "f = open('obo-t17800628-33.html', 'wb')." I'd love to understand that better

Thank you again for the site! I hope this is helpful info.

All the best,
Cathi

[ianmilligan1]

Here's a few workarounds. The lesson is indeed written for Python 2.7.

To import, use urllib.request as you've noted.

import urllib.request

url = 'http://www.oldbaileyonline.org/browse.jsp?id=t17800628-33&div=t17800628-33'

response = urllib.request.urlopen(url)
webContent = response.read()

print(webContent[0:300])

and

# save-webpage.py

import urllib.request

url = 'http://www.oldbaileyonline.org/browse.jsp?id=t17800628-33&div=t17800628-33'

response = urllib.request.urlopen(url)
webContent = response.read().decode('utf-8')

f = open('test.html', 'w')
f.write(webContent)
f.flush()
f.close()

Putting together:

import urllib.request

url = 'http://www.oldbaileyonline.org/browse.jsp?id=t17800628-33&div=t17800628-33'

response = urllib.request.urlopen(url)
HTML = response.read().decode('utf-8')

print(stripTags(HTML))

and to get the word list, I did

import urllib.request

url = 'http://www.oldbaileyonline.org/browse.jsp?id=t17800628-33&div=t17800628-33'

response = urllib.request.urlopen(url)
HTML = response.read().decode('utf-8')
clip = stripTags(HTML)
text = BeautifulSoup(clip,"html5lib").get_text().lower()

wordlist = text.split()

print(wordlist[0:120])

[mdlincoln]

Is this closeable now that we have the python warnings?

[acrymble]

We used to have comments back when we were on Wordpress. It's a shame we've lost that actually because this could be the type of thing solved through a comment (I'm not suggesting we implement that now!). But yes, I suppose we can close it.

[lukifer195]

from urllib.request import Request, urlopen
url = 'https://dict.laban.vn/find?type=1&query=ch%C3%A2n'
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
resource = urlopen(req)
print(resource.read())
content = resource.read().decode(resource.headers.get_content_charset())

any who can help me decode that link , i don't know factor when i tried with another link and it worked

[guanbuc]

doenst work

[svmelton]

@guanbuc could you provide a bit more information about what isn't working for you?