web-crawler

Crawl site concurrently, extract page url, first title, first paragraph, image and link urls.
Save result to report.csv.

Contributing

Clone the repo

git clone https://github.com/wrelin/web-crawler
cd web-crawle

Build the project

go mod tidy
go build

Run the project

./web-crawler <site_url> <max_concurrent_thread> <max_pages_to_crawl>

Run the tests

go test ./...