webcrawler

a simple python web crawler.

##How to use Please make sure to install python3 & requirements before using the program:

pip install -r requirements.txt

###to run the program:

cd ~/webcrawler 
python3 app.py

##To do

• Get links from html to crawl them recursively
• Adding Algerian universities official websites to data/algerian_univ.txt
• Read websites from DB