Web spider

Environment variables

URLLIST path to file with list of urls to crawl

CONCURRENCY number of concurrent requests

URLLIST=example_list.txt CONCURRENCY=10 go run cmd/main.go

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
cmd		cmd
pkg		pkg
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
example_list.txt		example_list.txt
go.mod		go.mod
go.sum		go.sum