This Project scraps articles from archives of Telugu newspaper website Andhra Jyoti. A set of queries is created and corresponding ground truth answers were retrieved by a combination of 2 popular ranking functions namely BM25 and tf-idf.
Complete Dataset can be downloaded from here .
(OR) If you choose to create dataset by yourself using the code, here you go.
- Python3
- Pip3
- Telugu Language should be enabled in Language settings of your machine, to be able to see telugu text.
Open the terminal and change current working directory to the location where you want to clone the project.
git clone https://github.com/AnushaMotamarri/Telugu-Newspaper-Article-Dataset
cd Telugu-Newspaper-Article-Dataset
python makedirs.py
pip3 install bs4
pip3 install requests
python3 scrapeTelugu.py
You should now be seeing text files getting created in subfolders of the directory telugudata.
This Scraper is website specific. So, it does not work with other websites.
A similar work on Malayalam Dataset can be found here