rovers is a service to retrieve repository URLs from multiple repository hosting providers and store the URLs and metadata to a PostgreSQL database while issuing messages to a RabbitMQ queue so you can connect other processes to it.
Quick start using docker images
Download docker images
Get the latest rovers image:
docker pull srcd/rovers
Start everything using docker-compose
Start Rabbit and Postgres
docker-compose -d up rovers-postgres rovers-rabbitmq
Export as environment variables the API keys (see section). Then, you can execute rovers:
docker-compose up --no-deps rovers
If you need to run just some of the available supported-providers, you can use this command replacing the flag
--provider with those providers you want to use:
docker-compose run --rm --no-deps --service-ports rovers /bin/sh -c "rovers initdb; rovers repos --provider=github --provider=bitbucket"
After that, rovers will generate a lot of 'mentions' (git repositories found on the internet), and sending them to the 'rovers' queue in Rabbit.
To stop and remove all the containers running
Finally, you can use Borges to fetch the content of these repositories.
All the supported providers are used by default. In case you need to run only some of them you must use the
rovers repos --provider=github --provider=bitbucket
Generate needed API keys for the providers
To be able to fetch github and cgit repositories, you should create several API keys:
- Get Github token: https://github.com/settings/tokens
- Get Bing token (Bing is the search engine used to fetch cgit repositories from internet): https://azure.microsoft.com/en-us/pricing/details/cognitive-services/search-api/web/
Uses the GitHub API to get new repositories. Requires a GitHub API token, which will only need repository read access. You can set the token through the environment variable:
$ export CONFIG_GITHUB_TOKEN=github-token
Uses the Bitbucket API to get new repositories as an anonymous user.
Note: the first time this provider runs it may take a while to find git repositories, as the vast majority of repositories at the beginning of the API results are mercurial repositories.
cgit is a popular service to create your own self-hosted Git repository provider. Rovers uses Bing search to discover cgit instances online and tracks them to get new repositories. Requires a Bing API key. You can set the key through the environment variable:
$ export CONFIG_BING_KEY=bing-api-key
go get -u github.com/src-d/rovers/...
rovers --help to get help about the supported commands and their options.
repos also have their own
for specific options related to them.
To initialize the database schemas. You need to run this command only once.
To start collecting repository URLs
rovers repos --provider=github
You can configure rovers using environment variables:
CONFIG_GITHUB_TOKENto set the github api key.
CONFIG_BING_KEYto set the cgit api key.
CONFIG_BROKER_URLto set the broker url, by default
CONFIG_DBUSER: database username, by default if not set
CONFIG_DBPASS: database user password, by default if not set
CONFIG_DBHOST: database host, by default if not set
CONFIG_DBPORT: database port, by default if not set
CONFIG_DBNAME: database name, by default if not set
CONFIG_DBSSLMODE: ssl mode to use, by default if not set
CONFIG_DBTIMEOUT: connection timeout, by default if not set
CONFIG_DBAPPNAME: application name
rm Makefile.main; rm -rf .cito make sure you will have the last Makefile changes.
make dependenciesto download vendor dependencies.
make packagesto generate binaries for several platforms.
You will find the built binaries under
This service uses PostgreSQL and RabbitMQ.
To execute tests locally you need to run RabbitMQ and PostgreSQL.
docker-compose up -d rovers-postgres rovers-rabbitmq make test