Our Microservices

We run a microservice architecture on kubernetes that hosts several different services. To give you a concept of all the moving parts, here is high level of those services.

• API
• DB
• Indexer
• Router
• Web

API

DB

We use a Cassandra db cluster for both prod and development. Our tables are as follows.

Keyspace Name
gophr

Column Name	Description
packages	Content Cell
github_api_key	Our internal github lib requires api keys to make requests so we store them in our db here
...

Storing packages

Indexer

The indexer is a core component of the Gophr infrastructure and is located in the main gophr repo in gophr/indexer. It's also currently set to run on a cron job at 3am everyday.

Responsibility

• Fetch and store internally any new packages from Godoc.org / awesome_go
• For each package stored internally, fetch and refresh metadata for that repo on github
• For each package if it no longer exists on Github remove it from our index

It's responsible for aggregating important and popular go packages from the community and storing them in our own internal database.

It's job is to make sure that we regularly index all (or the most important) go packages from a datasources. We need localized information about packages to provide fast a resilient up to date information for users, and to monitor statistical information related to packages that we generate via activity through Gophr.

Data sources

We currently only support packages from Github for now as 90% of go packages are hosted there. We collect the packages from a two different places in particular.

This first being from Google themselves at godoc.org/index. Godoc contains an automatically curated list of go packages from across the community.

The second being awesome-go, a manually curated list of the most important and pivotal packages hand picked from the community. A highly respected source in the community.

Sub components

Github Indexer

The Github Indexer is a sub component that runs as job inside the indexer. For go dependency in our DB we fetch important repo metadata such as star_gazers or star count of a repo to provide important ranking heuristics.

Router

Web