crawler

A simple website crawler made with go

Usage

1. Clone the repo

git clone https://github.com/dagregi/crawler

2. Compile the package

go build -o crawler

3. Run the executable

./crawler <baseURL> <maxConcurrency> <maxPages>
# example
./crawler https://crawler-test.com 5 30

Warning: Unless you want your IP blocked don't send large amounts of requests