Crawlers

Currently, we only support crawling repositories from GitHub. If you want to write your own crawler, there is a class BaseCrawler in core/crawler/basecrawler.py, please use it as your base class. Then you just need to implement the functions that are marked as NotImplementedError in the base class.

There is a Github crawler in core/crawler/githubcrawler.py, you can read its code for more reference. It uses a lot of help functions defined in the directory core/utils, which may be useful to you.