Web scraping is the method of gathering data from the Internet. The term "web scraping" generally refers to an automated operation. Some websites dislike it when automated scrapers collect their data, while others do not.If you scrape a page responsibly for educational reasons, you should have no concerns. Still, before embarking on a large-scale project, conduct your own investigation and ensure that you are not in violation of any Terms of Service.
The tremendous amount of information available on the Internet is an invaluable resource for any subject of study or personal interest. Web scraping may be used to efficiently collect that data. The Python libraries like Requests and Beautiful Soup are excellent tools for this task.
5. Build a script that fetches comments of users from the Web and displays relevant information in your console
6. After testing the code on my local machine I deployed the code on Google cloud VM as large amount of pages needs to scraped for this project.
Working through this project gave me the knowledge of the process and tools you need to scrape any static website out there on the World Wide Web.
My next step would be to clean the collected data and run Sentiment Analysis and Text mining on it, which could be found here.