Skip to content
Elegant Scraper and Crawler Framework for Golang
Branch: master
Clone or download
asciimoo Merge pull request #306 from jredl-va/gzip-sitemaps
Tweaks to how xml sitemaps that are decompressed and handled
Latest commit 9192be9 Mar 13, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Added .github/ (optional) Nov 22, 2017
cmd/colly [mod] add license header Mar 11, 2018
debug [mod] add license header Mar 11, 2018
extensions Fixes #229 Oct 1, 2018
proxy Proxy URL for request Sep 17, 2018
queue [fix] resolve race in queue II. Jun 5, 2018
.travis.yml [mod] update go versions in travis Feb 4, 2019 [enh] release v1.2.0 Feb 13, 2019 Update Dec 2, 2017
LICENSE.txt [enh] add request & response callbacks ++ cookie handling ++ readme Sep 29, 2017
VERSION [enh] release v1.2.0 Feb 13, 2019
colly.go Merge pull request #306 from jredl-va/gzip-sitemaps Mar 13, 2019
context.go [mod] add license header Mar 11, 2018
http_backend.go Decompress if content-type is set to gzip instead of content encoding. Mar 12, 2019
request.go Proxy URL for request Sep 17, 2018
response.go [fix] valid response filename handling - closes #119 Apr 2, 2018
unmarshal_test.go Add map unmarshal Feb 14, 2019


Lightning Fast and Elegant Scraping Framework for Gophers

Colly provides a clean interface to write any kind of crawler/scraper/spider.

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

GoDoc Backers on Open Collective Sponsors on Open Collective build status report card view examples Code Coverage FOSSA Status Twitter URL


  • Clean API
  • Fast (>1k request/sec on a single core)
  • Manages request delays and maximum concurrency per domain
  • Automatic cookie and session handling
  • Sync/async/parallel scraping
  • Caching
  • Automatic encoding of non-unicode responses
  • Robots.txt support
  • Distributed scraping
  • Configuration via environment variables
  • Extensions


func main() {
	c := colly.NewCollector()

	// Find and visit all links
	c.OnHTML("a[href]", func(e *colly.HTMLElement) {

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)


See examples folder for more detailed examples.


go get -u


Bugs or suggestions? Visit the issue tracker or join #colly on freenode

Other Projects Using Colly

Below is a list of public, open source projects that use Colly:

If you are using Colly in a project please send a pull request to add it to the list.


This project exists thanks to all the people who contribute. [Contribute].


Thank you to all our backers! 🙏 [Become a backer]


Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]


FOSSA Status

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.