Go Crawler

Website crawler written in Golang. Recursively searches internal links starting from base url provided. Returns report of number of links for each page.

Installation

Fork/Clone the repository:
Install dependencies:

Go

go mod download

Usage

go run . <website> <maxConcurrency> <maxPages>

maxConcurrency = number of workers to concurrently search

maxPages = set number of pages desired to process

Name		Name	Last commit message	Last commit date
.gitignore		.gitignore
README.md		README.md
crawl.go		crawl.go
get_html.go		get_html.go
get_urls_from_html.go		get_urls_from_html.go
get_urls_from_html_test.go		get_urls_from_html_test.go
go.mod		go.mod
go.sum		go.sum
main.go		main.go
normalize_url.go		normalize_url.go
normalize_url_test.go		normalize_url_test.go

Navigation Menu

Repository files navigation

Go Crawler

Installation

Usage

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Go Crawler

Installation

Usage

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages