Using Python to Access Web Data_Coursera_Following Links in Python

[fairypp]

Following Links in Python
In this assignment you will write a Python program that expands on http://www.pythonlearn.com/code/urllinks.py
(http://www.pythonlearn.com/code/urllinks.py). The program will use urllib to read the HTML from the data files
below, extract the href= vaues from the anchor tags, scan for a tag that is in a particular position relative to the
first name in the list, follow that link and repeat the process a number of times and report the last name you find.
We provide two files for this assignment. One is a sample file where we give you the name for your testing and the
other is the actual data you need to process for the assignment
Sample problem: Start at http://python-data.dr-chuck.net/known_by_Fikret.html (http://python-data.drchuck.
net/known_by_Fikret.html)
Find the link at position 3 (the first name is 1). Follow that link. Repeat this process 4 times. The answer is
the last name that you retrieve.
Sequence of names: Fikret Montgomery Mhairade Butchi Anayah
Last name in sequence: Anayah
Actual problem: Start at: http://python-data.dr-chuck.net/known_by_Conar.html (http://python-data.drchuck.
net/known_by_Conar.html)
Find the link at position 18 (the first name is 1). Follow that link. Repeat this process 7 times. The answer is
the last name that you retrieve.
Hint: The first character of the name of the last page that you will load is: J
Strategy
The web pages tweak the height between the links and hide the page after a few seconds to make it difficult for
you to do the assignment without writing a Python program. But frankly with a little effort and patience you can
overcome these attempts to make it a little harder to complete the assignment without writing a Python program.
But that is not the point. The point is to write a clever Python program to solve the program.
Sample execution
Here is a sample execution of a solution:
$ python solution.py
Enter URL: http://python-data.dr-chuck.net/known_by_Fikret.html
Enter count: 4
Enter position: 3
Retrieving: http://python-data.dr-chuck.net/known_by_Fikret.html
Retrieving: http://python-data.dr-chuck.net/known_by_Montgomery.html
Retrieving: http://python-data.dr-chuck.net/known_by_Mhairade.html
Retrieving: http://python-data.dr-chuck.net/known_by_Butchi.html
Retrieving: http://python-data.dr-chuck.net/known_by_Anayah.html
The answer to the assignment for this execution is "Anayah".
✏ Exit
×
7/19/2016 PR4E
https://pr4e.dr-chuck.com/tsugi/mod/python-data/index.php?PHPSESSID=8ddd4cf133656c568693f78df92901d3 2/2
Turning in the Assignment
Enter the last name retrieved and your Python code below:
Name: (name starts with J) Submit Assignment

My Solution:

import urllib
from bs4 import BeautifulSoup

#url = 'http://python-data.dr-chuck.net/known_by_Conar.html'
url = raw_input('Enter URL:')
count = int(raw_input('Enter count:'))
position = int(raw_input('Enter position:'))-1
html = urllib.urlopen(url).read()

soup = BeautifulSoup(html,"html.parser")
href = soup('a')
#print href

for i in range(count):
    link = href[position].get('href', None)
    print href[position].contents[0]
    html = urllib.urlopen(link).read()
    soup = BeautifulSoup(html,"html.parser")
    href = soup('a')

[marwanlee]

Nice, here is mine:

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
position = 3
reps = 4
site = "http://dadadadadada link dadada lin"
count = 0
liste = list()
namelist= list()

url = site
while count <= reps-1:
liste.clear()
namelist.clear()
html = urllib.request.urlopen(url).read()
soup = BeautifulSoup(html, "html.parser")
tags = soup('a')
count = count+1

for tag in tags:
    link= tag.get('href', None)
    surname = tag.string
    namelist.append(surname)
    liste.append(link)
url=liste[3]

print(namelist[position-1])

[pavithravairavasundaram]

i am getting an error-> Module bs4 not found.
please help with this

[nidhi-acc]

I was getting same error so I ran "pip install beautifulsoup4" command on cmd. After that I am able to use this library.

[abhijithjadhav]

Nice here is mine too .. A little bit easy to understand

My Solution:

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter - ')
position = int(input("Enter position:"))-1
count = int(input("Enter count:"))
html = urllib.request.urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, 'html.parser')
Sequence = []
tags = soup('a')
for i in range(count):
    link = tags[position].get('href', None)
    print("Retrieving:",link)
    Sequence.append(tags[position].contents[0])
    html = urllib.request.urlopen(link, context=ctx).read()
    soup = BeautifulSoup(html, 'html.parser')
    tags = soup('a')
    link = tags[position].get('href', None)
print(Sequence[-1])

[solcross]

Hi, @abhijitjadhav1998 . Your solution also helped me, but I do not understand why you have "-1" at the end of "position" and the final "print". Referencing the order or what?

A little # on your code won't hurt. Thanks in advance.

[abhijithjadhav]

Hi, @abhijitjadhav1998 . Your solution also helped me, but I do not understand why you have "-1" at the end of "position" and the final "print". Referencing the order or what?

A little # on your code won't hurt. Thanks in advance.

@solcross the -1 is used to print the last name in the list named Sequence

[solcross]

@abhijitjadhav1998 Thanks a lot

[jasoriasaksham01]

import urllib.error, urllib.request, urllib.parse
from bs4 import BeautifulSoup
a = int(input("Position: "))-1
b = int(input("Times: "))
url = input("Enter URL: ")
for c in range(b):

x = urllib.request.urlopen(url).read()
y = BeautifulSoup(x,'html.parser')
z = y('a')
d = z[a].get('href',None)
url = d
e = z[a].contents[0]

print(e)

[ussefdesouky]

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup

input_url = input('enter URL - ')
open_url = urllib.request.urlopen(input_url).read()

soup = BeautifulSoup(open_url, 'html.parser')
tags = soup('a')

count = 7
for i in range (count):
pose = 18
for tag in tags:
pose -= 1
if pose == 0:
print(tag.get('href', None))
print(tag.contents[0])
new_link = tag.get('href', None)
open_url = urllib.request.urlopen(new_link).read()
soup = BeautifulSoup(open_url, 'html.parser')
tags = soup('a')

don't forget the indentation

[stevenparker-creator]

i am getting an error-> Module bs4 not found.
please help with this

i am getting the same problem from it

[vitalvalid]

revisited method here
import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup

url=input("Enter the url")
if len(url)<1:
url=' http://py4e-data.dr-chuck.net/known_by_Fikret.html'
try:
link_pos=int(input("Ener the link position:"))
repeat=int(input("Enter the number of repetaion:"))
except ValueError:
print("Enter the integer value and try again.")
quit()
def finding_links(link):
html=urllib.request.urlopen(link).read()
soup=BeautifulSoup(html,'html.parser')
tags=soup('a')
return tags
name_list=list()
for count in range(repeat):
tags=finding_links(url)
recursion_link=tags[link_pos-1].get('href',None)
name_list.append(tags[link_pos-1].contents[0])
url=recursion_link
print(name_list)
print(name_list[-1])

[jasoriasaksham01]

i am getting an error-> Module bs4 not found.
please help with this

i am getting the same problem from it

might be because your module is not placed at the same spot where your python file is

[sutkarsh-s]

i am getting an error-> Module bs4 not found.
please help with this

i am getting the same problem from it

or u can simply do all this stuff in pycharm(python's ide) without downloading this module if u are unable resolve this error

[vitalvalid]

i am getting an error-> Module bs4 not found.
please help with this

i am getting the same problem from it

might be because your module is not placed at the same spot where your python file is

i am getting an error-> Module bs4 not found.
please help with this

i am getting the same problem from it

you can simply install the beautiful soup for the whole environment ,
by running pip3 install beautifulsopu4 in your command line for python3

[KaziArman]

import urllib.error, urllib.request, urllib.parse
from bs4 import BeautifulSoup
a = int(input("Position: "))-1
b = int(input("Times: "))
url = input("Enter URL: ")
for c in range(b):

x = urllib.request.urlopen(url).read()
y = BeautifulSoup(x,'html.parser')
z = y('a')
d = z[a].get('href',None)
url = d
e = z[a].contents[0]

print(e)

Very simple & smooth code. Best one so far