WEB SCRAPING

In this repository you can find functions for web scraping, based on a search query. The found content is saved in a csv format file. Note that not all sites allow web scraping, so you should check it first. This repository also contains functions for downloading tweets for text analysis.

Environment setup

pip install poetry In the current folder, in the terminal enter "poetry shell" to initialize the virtual environment (based on the .toml file).

To add libraries to the virtual env press "poetry add library_name"

Main File web_scraping_get_content.py is the main web scraping script