In this project/tutorial we introduce some basics on web scraping. This technique allows us to retrieve data from websites. This can be especially very handy when there
are no datasets available or/and it is not possible to obtain such data via APIs.
In our example it is possible to see how to obtain data from different websites having a basic knowledge on HTML and using some Python, and Python packages.
The notebook explains the technique step-by-step using an entertainment example. However, this technique is applicable to any case where obtaining (extra) data is necessary.
A blog was also written with the same content in a more compact presentation.
-
requests: HTTP library for Python that allows us to send HTTP requests in a simple way. -
Beautiful Soup: Python library for pulling data out of HTML and XML files. It allows us to access easily the information we need to retrieve.
Some knowledge on lists, list comprehension, and string methods is also very helpful.
- Install requirements using
pip install -r requirements.txt.- Make sure you use Python 3.
- You may want to use a virtual environment for this.