Rovers is a service to retrieve repository URLs from multiple repository hosting providers.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


Build Status codecov

rovers is a service to retrieve repository URLs from multiple repository hosting providers and store the URLs and metadata to a PostgreSQL database while issuing messages to a RabbitMQ queue so you can connect other processes to it.

Quick start using docker images

Download docker images

Get the latest rovers image:

docker pull srcd/rovers

Start everything using docker-compose

Install docker-compose.

Start Rabbit and Postgres

docker-compose -d up rovers-postgres rovers-rabbitmq

Export as environment variables the API keys (see section). Then, you can execute rovers:

docker-compose up --no-deps rovers

If you need to run just some of the available supported-providers, you can use this command replacing the flag --provider with those providers you want to use:

docker-compose run --rm --no-deps --service-ports rovers /bin/sh -c "rovers initdb; rovers repos --provider=github --provider=bitbucket"

After that, rovers will generate a lot of 'mentions' (git repositories found on the internet), and sending them to the 'rovers' queue in Rabbit.

To stop and remove all the containers running docker-compose down.

Finally, you can use Borges to fetch the content of these repositories.

Supported Providers

All the supported providers are used by default. In case you need to run only some of them you must use the --provider flag:

rovers repos --provider=github --provider=bitbucket

Generate needed API keys for the providers

To be able to fetch github and cgit repositories, you should create several API keys:


Uses the GitHub API to get new repositories. Requires a GitHub API token, which will only need repository read access. You can set the token through the environment variable:

$ export CONFIG_GITHUB_TOKEN=github-token


Uses the Bitbucket API to get new repositories as an anonymous user.

Note: the first time this provider runs it may take a while to find git repositories, as the vast majority of repositories at the beginning of the API results are mercurial repositories.


cgit is a popular service to create your own self-hosted Git repository provider. Rovers uses Bing search to discover cgit instances online and tracks them to get new repositories. Requires a Bing API key. You can set the key through the environment variable:

$ export CONFIG_BING_KEY=bing-api-key


go get -u


Run rovers --help to get help about the supported commands and their options. The commands initdb, replay and repos also have their own --help switch for specific options related to them.

To initialize the database schemas. You need to run this command only once.

rovers initdb

To start collecting repository URLs

rovers repos --provider=github

You can configure rovers using environment variables:


  • CONFIG_GITHUB_TOKEN to set the github api key.
  • CONFIG_BING_KEY to set the cgit api key.


  • CONFIG_BROKER_URL to set the broker url, by default amqp://guest:guest@localhost:5672


  • CONFIG_DBUSER: database username, by default if not set testing
  • CONFIG_DBPASS: database user password, by default if not set testing
  • CONFIG_DBHOST: database host, by default if not set
  • CONFIG_DBPORT: database port, by default if not set 5432
  • CONFIG_DBNAME: database name, by default if not set testing
  • CONFIG_DBSSLMODE: ssl mode to use, by default if not set disable
  • CONFIG_DBTIMEOUT: connection timeout, by default if not set 30s
  • CONFIG_DBAPPNAME: application name



  • rm Makefile.main; rm -rf .ci to make sure you will have the last Makefile changes.
  • make dependencies to download vendor dependencies.
  • make packages to generate binaries for several platforms.

You will find the built binaries under ./build.


This service uses PostgreSQL and RabbitMQ.

To execute tests locally you need to run RabbitMQ and PostgreSQL.

  docker-compose up -d rovers-postgres rovers-rabbitmq
  make test

Running Rovers in Kubernetes

You can use the official Helm chart to deploy Rovers in your Kubernetes cluster.