Skip to content

Latest commit

 

History

History
24 lines (15 loc) · 637 Bytes

README.md

File metadata and controls

24 lines (15 loc) · 637 Bytes

go-simple-crawler-practice

Simple crawler for learning go

What can you learn from this simple codes is the simple go program structures and the little understanding about concurrency.


Description

  • crawler.go

    Crawls the following of users on github recursively.

  • crawler_pipeline.go

    Crawls the following of users using workers concurrently.

  • crawler_interface_pipeline.go

    Crawls the following and starred repositories of users using crwaler interface and workers concurrently.


Run

  1. Modify the startURL at top of file to fit your account
  2. go run <crawler-file>