A database of information from AWS, GitHub, Synk, and other sources, Service Catalogue aims to provide a picture of the Guardian's estate, broken down by Product & Engineering (P&E) team.
In contrast with Prism, which collects data from a subset of AWS resources, Service Catalogue offers a more complete picture of production services, as we may provision a resource that Prism doesn't know about.
The Guardian has hundreds of EC2, lambda, and other services in AWS, each built from one of thousands of GitHub repositories, by one of many P&E teams.
Some of the questions Service Catalogue aims to answer include:
- For P&E teams:
- Which services do I own?
- Which services follow DevX best practice/use tooling?
- Which repo do services come from?
- What is my service reliability? (time since last incident)
- For the Developer Experience stream:
- What proportion of all services follow best practice/use tooling?
- What kinds of technologies are different streams using?
- Which teams are struggling with reliability and need more support?
- Which services belong to specific P&E product teams
Pricing information is not yet available in Service Catalogue, therefore, we're unable to answer questions such as:
- What does each service cost?
- What services are costing us the most money?
Service Catalogue has two parts:
- Data collection
- Data analysis
We use CloudQuery to collect data from AWS, GitHub, Snyk, and other sources.
We've implemented CloudQuery as a set of ECS tasks, writing to a Postgres database. For more details, see CloudQuery implementation.
Tip
To update CloudQuery, see Updating CloudQuery.
The data in Service Catalogue is analysed in two ways:
- Grafana, at https://metrics.gutools.co.uk
- AWS Lambda functions, for example RepoCop or data-audit
Follow the instruction in the dev-environment README to run cloudquery locally. Then follow the instructions in the repocop README to run repocop locally.
The diagram below outlines the architecture of the major components of the service catalogue.
