Web Crawler for C

Pareses the webpage of a given URL, extracts any embedded URLs, then retrieves each page recursively, but limits its exploration to a given depth.

Directions:

Make sure to have libcurl library installed.

Create a folder that will store all the web pages extracted by the crawler

Makefile is already given. Therefore, type make to compile

To execute

 ./crawler url folder_name depth_number

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
Makefile		Makefile
README.md		README.md
crawler.c		crawler.c
crawler.h		crawler.h
curl.c		curl.c
curl.h		curl.h
hashtable.c		hashtable.c
hashtable.h		hashtable.h
pagedir.c		pagedir.c
pagedir.h		pagedir.h
set.c		set.c
set.h		set.h
url.c		url.c
url.h		url.h